MELISSA BLOCK, HOST:
From NPR News, this is ALL THINGS CONSIDERED. I'm Melissa Block.
AUDIE CORNISH, HOST:
And I'm Audie Cornish. When a president taps someone to become a federal judge, the American Bar Association reviews and rates the nominee. That rating shapes whether the president's pick is confirmed by the Senate. Now, new analysis claims that the ABA ratings are biased. NPR social science correspondent Shankar Vedantam reports.
SHANKAR VEDANTAM, BYLINE: The American Bar Association's ratings are presented as advice to the White House and the Senate, but they carry real weight. The ratings produced by a senior ABA panel actually influence whether nominees become judges, says University of Rochester political scientist Maya Sen.
MAYA SEN: Receiving a low rating is devastating to the candidate. Candidates who receive a rating of not qualified, that's more or less the equivalent of an F, are something like 30 percent less likely to be confirmed.
VEDANTAM: Given that the ratings are so powerful, Sen recently decide to analyze ABA ratings of more than 1,700 nominees to U.S. district courts starting in 1960. She didn't find disparities in the ratings when it came to whether nominees were conservative or liberal, but she did find disparities when it came to race and gender.
SEN: What I find is that even taking into account potential discrepancies in educational backgrounds or in work experience that women and minorities do appear to receive lower ratings on average than do white and male candidates.
VEDANTAM: Sen's study is to be published soon in the Journal of Law and Courts, a peer reviewed law journal. In it, she also compares the performance of judges to see whether the ones who got the highest ratings when they were nominees turned out to be the most successful as judges. She used a commonly accepted measure of judicial competence, how often the judge's rulings were likely to stand the test of time. She found there was little connection between ratings and performance.
SEN: A judge who receives that F rating of not qualified, that judge is actually no more likely to be reversed than a judge who receives the A plus rating of well qualified.
VEDANTAM: James Silkenat, president of the American Bar Association, doesn't agree that the ratings are biased.
JAMES SILKENAT: If there was bias here, that would trouble me. I'm confident, though, there is not.
VEDANTAM: Silkenat says the ABA very carefully designs its process to eliminate bias. ABA presidents appoint the panel of legal experts who come up with the ratings. Silkenat say he picks experts with impeccable credentials and diverse backgrounds. The committee, he said, is charged with examining the competence, integrity and temperament of nominees.
To do that, the committee conducts detailed interviews with people who have worked with the nominee. They pour over the nominee's track record and experience. When negative information is found, it's double-checked and the nominee is given an opportunity to refute it. Silkenat says there might be innocent explanations for why disparities emerged in Sen's study.
SILKENAT: I'm tempted to say that you could prove almost anything with statistics. In this case, I think the author of the article has focused on the wrong issue.
VEDANTAM: Silkenat says Sen should have looked more closely at the trial experience of low and high rated nominees. That's because the committee tends to give higher ratings to candidates who have lots of trial experience. Nominees who lack such experience because they come from academia, for example, might suffer as a result.
SILKENAT: In my view, not enough attention was paid to trial experience as a factor. That could be a - I'm sure is a factor here and that bias is not.
VEDANTAM: Sen says she took trial experience into account and still found racial and gender disparities. She says that it isn't entirely clear how the ABA rates different kinds of legal experience, since the details of their evaluation process are not fully shared with the public. And some ABA measures, she says, are subjective, like judicial temperament.
SEN: It's true that there might be subjective criteria that are being applied in such a way that it's not favorable to women and minorities. Now, that's why I say that more transparency and the use of more objective criteria could be really important.
VEDANTAM: Subjectivity has widely been identified as a problem in decision-making. Many studies show subjective decisions are prone to biases, including many that are unconscious. The ABA's ratings depend on the testimony of large numbers of people who know the nominees. If that testimony unconsciously holds women and minorities to a tougher standard, for example, that could explain the bias in ABA ratings that Sen found. Shankar Vedantam, NPR News. Transcript provided by NPR, Copyright NPR.
