Back to Presentations

Transparency in DNA evidence

M.W. Perlin, "Transparency in DNA evidence", President’s Council of Advisors on Science and Technology (PCAST), Washington, DC, 18-Nov-2016.

Talk

Slide 1. Title slide

Thank you for inviting me to present at PCAST. Today I will be talking about "Transparency in DNA Evidence."

Slide 2. Pattern evidence

Forensic analysis of pattern evidence attempts to compare the data from one item to that of another item. This comparison is mathematically implausible. As DNA evidence has shown for the last two decades, valid comparisons must be made between the types that underlie observed features.

Slide 3. DNA mixtures

DNA mixtures arise when two or more people contribute their genetic material to the same item. Long ago the Justice Department's Federal Bureau of Investigation (FBI) developed mixture interpretation methods. These old approaches apply thresholds that simplify evidence data. The FBI's data analysis does not separate out contributor genotypes. Altering data introduces potential for bias. Comparing unseparated data with a person's genotype is not meaningful.

Slide 4. TrueAllele solution

The TrueAllele® solution is to unmix the mixture. Starting from the original data, Bayesian probability modeling separates out the genotypes of each contributor to the mixture. Genotype uncertainty is represented using probability. Afterwards, a separated genotype of one item is compared with a separated genotype from another item. This comparison gives a valid Bayesian likelihood ratio (LR) match statistic.

The TrueAllele philosophy dates from 1999.

• First, the computer uses all the data, without adding or removing peaks.
• Second, the objective approach based on separation does not consider a suspect's genotype.
• Third, TrueAllele's unmixing process works the same way for investigative and evidentiary applications.

The TrueAllele match statistic is neutral, providing either inclusionary or exclusionary results that are based on the evidence.

Slide 5. Validation studies

For over ten years, scientists have conducted TrueAllele validation studies based on identification information. Thirty-four studies have been completed so far. These show TrueAllele's accuracy by establishing specificity, sensitivity and reproducibility. Seven of these studies have been published in peer-reviewed journals. Yet most mixture interpretation methods have never been validated.

A 2015 TrueAllele validation published in the Journal of Forensic Sciences demonstrated TrueAllele reliability for interpreting DNA mixtures containing up to five unknown contributors, down to one percent mixture weights, on amounts as small as one human cell.

Slide 6. Linear relationship

Several TrueAllele validation studies have established a statistical relationship between a contributor's DNA quantity and the match statistic. On a logarithmic scale, the likelihood ratio is proportional to the amount of DNA.

Moreover, TrueAllele maintains the same linear slope under varying conditions. Analysis of covariance proves this occurs regardless of the number of contributors, the mixture weight, or the amount of a contributor's DNA. Such empirical validation shows there are no limits to TrueAllele's applicability. As mixture data become less informative, TrueAllele yields less identification information, whether inclusionary or exclusionary.

Slide 7. Inclusion probability

However, the inclusion methods promoted by the FBI never worked. Nor were they ever validated for match information. In the FBI Popstats software, Combined Probability of Inclusion (CPI) using a threshold produced unreliable results. The Commerce Department’s National Institute of Standards and Technology (NIST) showed this unreliability in a 2005 study, which they never published. The FBI introduced a second "stochastic threshold" (an oxymoron), hoping to resolve CPI's false positive flaws. However, that simplistic correction, unsupported by data, also failed.

Slide 8. Government failure

Government agencies have been floundering for almost 20 years with their DNA mixture analysis. They simply have no expertise in the modern statistical computing required to solve these problems. Some employ unworkable methods, while others adopt foreign commercial products developed by people they know.

But these interpretation approaches continue to ignore evidence data. The methods use simple models for familiar workflows. Commercial ties pose conflicts of interest. Government is accruing a huge liability from years of bad science causing bad justice in countless cases.

Much of the PCAST report is insightful. But some of the probabilistic genotyping findings similarly ignore good data and science.

Slide 9. Random counting

Scientific studies show that CPI doesn't work. Basically, the method just counts up the number of reported tests. For every two loci tested, on average CPI multiplies another factor of ten. There is little correlation between CPI and identification information.

Unfortunately some continue to promote CPI. A recent paper in BMC Genetics highlights a nexus between the New Zealand ESR company that develops STRmix, the NIST regulator, and FBI views. The paper contains no data, no results, and has no scientific support for its conclusions. Yet it was accepted by the journal in under two weeks, and published in under a month, which is not typical of independent peer review.

Slide 10. Peer review

Peer review can favor established interests, even when reported findings are not credible. This was seen in an International Society of Forensics and Genetics (ISFG) paper published this month. The nexus of NIST and ESR appeared again in the first authors. An ESR vendor helped write international recommendations for regulating his own commercial DNA mixture interpretation software. The ISFG paper was accepted and published in under a week, suggesting a rubber stamp, rather than rigorous peer review.

PCAST has raised concerns about manufacturers conducting peer-reviewed studies. However, independent peer review has addressed financial conflicts in science for a century. Scientists typically receive grant funding, which support their jobs and reputation. Removing all funded research papers from their resumes would erase their credentials. This situation applies to academic PCAST members; thinning their resumes this way would effectively disband the council.

Slide 11. Justice

DNA identification science is evidentiary, not prosecutorial. PCAST uses incorrect language here. Of the 400 DNA cases I have consulted on, 10% have been for the defense or innocence projects. Reliable match statistics can exclude defendants, or implicate others who are guilty. TrueAllele is neutral, and does not take sides.

Darryl Pinkins was wrongfully convicted for a rape he did not commit, and spent 24 years in an Indiana prison. DNA mixture evidence developed 15 years ago was not resolved by failed FBI methods. This year, exculpatory evidence developed by TrueAllele reanalysis of the same DNA data freed an innocent man. The unsupported PCAST findings on limiting TrueAllele's validated applicability would have ensured Mr. Pinkins stay in prison. PCAST would deny other incarcerated innocents their scientific DNA relief.

In the Nick Hillary case mentioned by PCAST's report, there was no conflict between TrueAllele and STRmix. At 30 RFU and most other threshold levels, STRmix excluded Hillary. However, using a threshold of 50 RFU, along with careful data selection, included him. STRmix conflicted with STRmix. This happened because limited statistical modeling required data choices that introduced human bias.

Slide 12. Transparency

Government secrecy in forensic science has failed the public for 20 years. Backroom forensics makes money and confers power, but obstructs justice. Crime labs hide their evidence data to avoid scientific scrutiny and lawsuits. This lack of data transparency is the root cause of forensic DNA failure.

The solution is opening forensic data for vigilant external review. PCAST could make the following helpful findings:

1. All crime lab data (past and future) must be open for outside scrutiny.
2. The FBI's CODIS database must be available to everyone.
3. The private sector solved the DNA mixture problem over ten years ago.
4. Failed government agencies do not need more funding for a solved problem.
5. Bad government should get out of the way of good science.

Thank you for your time and attention.

Slides

PCAST Handout