Forensic validation, error, and reporting: A unified approach

M. Perlin, "Forensic validation, error, and reporting: A unified approach", American Academy of Forensic Sciences 71th Annual Meeting, Baltimore, MD, 23-Feb-2019.

Talk

PowerPoint presentation of Dr. Perlin's talk.

Download Handout
Download PowerPoint

Abstract

After attending this presentation, attendees will understand how validation studies and match error are two sides of the same coin. Trial lawyers can apply these methods to ensure DNA evidence reliability in the courtroom.

This presentation will impact the forensic and legal communities by helping trial lawyers better understand and explain DNA limitations in criminal justice.

Federal Rule of Evidence (FRE) 702 requires reliable scientific evidence from expert witnesses. The Daubert standard specifically includes prongs for validation testing and error rate determination.

Simple DNA evidence comes equipped with a built-in false positive error rate, the random match probability (RMP) that states the chance of adventitious match to an uninvolved person. More complex evidence arises with small amounts of DNA, or mixtures of several people. To address uncertainty, likelihood ratios (LR) are used to quantify match strength.

Validation studies measure DNA match information on representative data sets. Match specificity is assessed by comparing many uncertain genotypes with many unrelated references, producing a large set of nonmatching LR values. A histogram of these numbers on a logarithmic scale shows the log(LR) distribution of nonmatching comparisons. These many-to-many evidence-to-reference comparisons provide a frequency framework for assessing a laboratory’s mixture data or interpretation method.

The match specificity histogram can be used to estimate the false positive error rate in an individual case. Comparing an evidence genotype with a suspect reference yields a match statistic. Positioning this log(LR) match value along the histogram shows how often a false match would occur. The validation fraction exceeding the reported log(LR) value is an error estimate of the probability of misleading evidence (PME). PME generalizes RMP to handle inexact DNA matches.

The specificity histogram can be directly constructed from a validation’s probabilistic genotypes, without reference sample comparisons. Information theory lets a computer assemble a composite log(LR) distribution from the distributions of many uncertain genotypes. This elegant many-to-none approach accelerates validation studies.

A log(LR) specificity histogram can also be developed for a single evidence item. Older statistical approximations construct a one-to-many comparison of the uncertain genotype against many unrelated references. But a more precise one-to-none direct construction immediately computes a numerically exact genotype specificity distribution solely from the evidence genotype, without making any reference comparisons. This direct approach is fast and accurate, and enables routine error reporting.

The log(LR) specificity distribution is inherent in a genotype’s probability description. Comparing evidence with suspect produces a log(LR) value. Positioning this reported match value within the frequency context of the specificity distribution yields an error rate for false match. The fraction of specificity matches exceeding the suspect’s statistic gives the PME – the chance this evidence would be as strong against someone who didn’t leave their DNA. This error rate is based on the actual DNA evidence in the case, not on a validation study done at another time and place by other people on unrelated DNA samples under idealized laboratory conditions.

For a reported match statistic, FRE 403 would suggest that a focused evidence-derived error rate is more relevant than a generic validation-derived error rate. Reporting on facts in evidence has greater probative value. A more germane evidence-based error rate can better assist a trier of fact.

Probability is readily communicated with whole numbers. For example, as “one in N people would match the evidence as strongly as does the suspect.”

With smaller match statistics (e.g., under a million), reporting PME error provides a frequency context for understanding what LR information means. Moreover, a numeric error is more precise than a verbal equivalent.

An exclusionary match statistic also has PME error. Defenders can use these false negative error rates to contextualize exculpatory DNA evidence.

The presentation gives a unified view of evidence and validation error rates. Case examples show how inclusionary and exclusionary match statistics, along with error rates, are presented in court. Direct error determination for evidence items may reduce the need for large-scale validation studies. Normative science reports relevant error for measured variables. Daubert encourages such error reporting in forensic science.

The author will present a unified approach to forensic validation and match error. The presentation will offer useful perspectives on DNA evidence reliability.