About twenty years ago I spent a good deal of my time testifying for the defense in criminal cases involving DNA evidence. These were trials in which the prosecution claimed that the defendant’s DNA profile had been found to match crime-scene samples (these involve blood or sperm analysis), and in which the prosecution presented “match probabilities: the supposed chance that a randomly-selected and innocent person would have had DNA that also matched the evidence. (If these probabilities are very low, say one in several million, juries tend to conclude that the suspect is guilty.)
While I favored the responsible use of DNA testing, the prosecution at that time was not being responsible, ergo my involvement. What I testified about, as an unpaid expert witness (I decided that saying I was paid for a case—and the prosecution always asks when you’re on the stand—might make the jury think that I was making money as a “professional witness”) were two issues: match probabilities and lab error rates. Here’s a brief synopsis:
Match probabilities. If the suspect’s own DNA matches that from the crime scene, you can then calculate the chance that a randomly-selected person would match the sample as well. This corresponds to the chance that an innocent person would have been implicated by the DNA evidence. Absent lab errors (see below), these calculations involve population genetics, which was my area of expertise. If, for example, the suspect matches the crime-scene sample at three tested genes, how do you calculate the probability of a random match?
That depends on who you consider to be a “random and innocent” person. Is it the population of Hispanics in America if the suspect is a Hispanic? Probably not, because we don’t know the ethnicity of the perpetrator. It’s thus best to use a series of databases and take the most conservative (highest) probability. Moreover, you can’t just multiply the probabilities for each gene together if gene forms are associated with each other in different groups, as they tend to be. One of my beefs was that the prosecution would use a database corresponding to the ethnicity of the defendant, and then just multiply the probabilities for each tested gene together; or they would use a variety of databases and select the lowest probability. Neither of these is kosher given the way human populations are genetically structured.
Nowadays, when we can do almost full DNA sequences rather than just matches at a few sites in three genes or so, this problem has been ameliorated. If a match is not perfect, then the suspect is exculpated. But one big problem remains, and it is one about which I testified at length:
Lab error rates. Labs aren’t perfect, and sometimes two samples whose DNA doesn’t match can be found to match if tubes get mixed up or if there is contamination. (Since genes are amplified thousands of times before sequencing, a small bit of contamination can be magnified.)
This happens more often than you think. When I was testifying (I stopped doing that after the Simpson trial), blind testing of labs gave an error rate of around 2%. That is, about one time in fifty, two non-matching samples sent to a lab to test its prowess would be seen to match based on lab error.
With error rates like this, match probabilities from popuation genetics become virtually useless. That’s because the chance of a random match becomes about equal to the lab error rate. If the rate of an innocent subject matching a crime sample includes both the random match probability based on the frequency of DNA profiles PLUS the chance that a match would occur from lab error, then the largest probability—lab error—dominates. If the former probability is, for example, one in a million (0.000001) and the latter one in fifty (0.02), then the total “random match” probability is the sum of these, or 0.020001. That’s about 2%. The default match probability is thus not one in a billion or one in a million, even if many genes are used, but simply the probability that a random person will match the crime sample because of lab error. The probability of lab error is invariably higher than the population-genetic probability.
This is all common sense, but prosecutors hated my testimony, because it made their evidence look a lot less incriminating than it was. So they tried to get around it, saying that the population-genetics calculation and the error rate calculations were “apples and oranges” and couldn’t be combined. (As I discovered from my courtroom experience, the prosecution is often less interested in presenting an honest case than in securing a conviction.) They also used irrelevant arguments that might appeal to non-scientists, often saying that my testimony was unreliable because it involved humans but my research was on fruit flies. (Both species, of course, have genes!)
One way I suggested to ameliorate the lab error rate was to label the samples blindly and to have the DNA tested in at least two or three labs independently. If all of them matched, the chances of error causing this would be reduced. (For three labs it would have been 0.02 X 0.02 X 0.02, or 8 in a million—comparable to some population-genetic calculations). But at that time the prosecution didn’t do this, and I don’t know if they do it now.
These considerations may seem simple to you, but juries are composed of a sample of voters, most of whom don’t even know what DNA is. To try to educate them about error rates, population genetics, and probabilities was a daunting task, and I often spent several days on the stand. Even then the jury was often baffled, as I suspect it was in the Simpson case.
While the population-genetic calculations have been improved by more extensive DNA analysis, the problem of lab errors remains, as shown in this new article from the New York Times (click on screenshot to read it). It’s by Greg Hampikan, a professor of biology at Boise State University, one of whose concerns is forensic DNA (he has a joint appointment in Criminal Justice).
Outside testing of forensic DNA labs have shown that there’s still a very large probability of lab error, and that error comes from two sources. (The article cites “an alarming new study of crime laboratories published this summer”, but I can’t find it and it isn’t cited.) I quote:
Researchers from the National Institute of Standards and Technology gave the same DNA mixture to about 105 American crime laboratories and three Canadian labs and asked them to compare it with DNA from three suspects from a mock bank robbery.
The first two suspects’ DNA was part of the mixture, and most labs correctly matched their DNA to the evidence. However, 74 labs wrongly said the sample included DNA evidence from the third suspect, an “innocent person” who should have been cleared of the hypothetical felony.
The test results are troubling, especially since errors also occur in actual casework.
In other words, an innocent person was deemed a match over 70% of the time due to lab error. This involves two types of mistakes: switching of tubes and the new possibility that the sensitivity of DNA tests allows the DNA of completely innocent people to be present in low concentration in crime-scene samples, but concentrations high enough to be detectable and thus judged “culpable”.
Tube swaps are easy to understand. But some laboratory errors are far more difficult to detect. For example, it’s hard to interpret DNA mixtures from three or more people. As DNA testing has become more sensitive, most laboratories are now able to produce profiles from anyone who may have lightly touched an object. The result is that DNA mixtures have become more common, making up about 15 percent of all evidence samples.
Moreover, there’s still the problem of different labs calculating different match probabilities, probably because they use different population-genetic calculations (my emphasis):
One shocking result from the new N.I.S.T. study is that labs analyzing the same evidence calculated vastly different statistics. Among the 108 crime labs in the study, the match statistics varied over 100 trillion-fold. That’s like the difference between soda change and the United States’ gross domestic product. These statistics are important because they are used by juries to consider whether a DNA match is just coincidence.
One would think that the data in the new paper (and again, I can’t find it) would make the prosecution think twice about how it presents data. But even the authors of that paper larded it with disclaimers, and the journal took four years to get the paper out, meaning that its results didn’t affect criminal cases over that period. Here’s Hampikan’s angry but justifiable complaint:
While this lapse in publication is troubling, more disturbing is that the authors try to mute the impact of their own excellent work. Neither the paper’s title nor the abstract mention the shocking findings. And the paper contains an amazing number of disclaimers.
In fact, the conclusion begins with a stark disclaimer apparently intended to block courtroom use:
The results described in this article provide only a brief snapshot of DNA mixture interpretation as practiced by participating laboratories in 2005 and 2013. Any overall performance assessment is limited to participating laboratories addressing specific questions with provided data based on their knowledge at the time. Given the adversarial nature of the legal system, and the possibility that some might attempt to misuse this article in legal arguments, we wish to emphasize that variation observed in DNA mixture interpretation cannot support any broad claims about “poor performance” across all laboratories involving all DNA mixtures examined in the past.
People serving time behind bars based on shoddy DNA methods may disagree. It is uncomfortable to read the study’s authors praising labs for their careful work when they get things right, but offering sophomoric excuses for them when they get things wrong. Scientists in crime labs need clear feedback to change entrenched, error-prone methods, and they should be strongly encouraged to re-examine old cases where such methods were used.
That disclaimer is absolutely unconscionable. ANY participating lab must be blind tested, and the results of that testing presented in the courtroom. There is no other way to ensure a fair presentation of evidence.
I’ve been out of this game for some time, so I wasn’t aware of this and had assumed that the lab error issue had been corrected. It hasn’t.
And those errors are important. When DNA testing exculpates a subject, it’s likely not due to lab error (though it could be). But when the testing implicates a suspect, one must be very scrupulous to ensure that the match isn’t an error and, if it isn’t, that the match statistics be presented fairly. I agree with the old dictum that it’s better (and, for DNA evidence, also less likely!) to let a hundred guilty people walk free than to jail one innocent person.
Hampikan suggests some fixes for correcting errors, but they aren’t perfect. Blind testing of samples and use of multiple labs remains two essential ways to ensure that the innocent don’t get jailed.