Bloomberg’s Toxics Law Reporter recently published a paper by Professor David L. Faigman titled “Bringing Scientific Peer Review to Scientific Evidence” that sets out an idea worth thinking about. Specifically, that the quality of scientific testimony presented to juries would be improved (and presumably the likelihood that justice is done would be increased) if the job of screening proposed testimony was shifted to, or at least augmented by, experts in the relevant field. Surely it would be better than the current system in which the task falls solely to someone who’s often sitting where he or she is precisely because science and math weren’t his or her thing. However, my objection goes to the premise lurking within his idea – that the courtroom is in the first place a proper venue for publishing, testing and ultimately deciding the scientific status of an hypothesis.
Name some established scientific theories that were first published in a courtroom. How about an analytical technique developed for and refined through the adversarial process at the courthouse that has made its way into widespread use beyond the bar? Cue the crickets. On the other hand, if asked for examples of theories and techniques deemed “scientific” at the courthouse yet found to be embarrassingly unscientific and the cause of widespread injustice you can start with the National Academy of Science’s 2009 scathing report on forensic science and update it with the weekend’s news that the FBI now admits that its hair analysis, admitted in hundreds of trials, was almost always “scientifically invalid”. Courtroom-produced science has a long and dismal record yet courts continue solemnly to admit into evidence opinions of experts that would be laughed at if presented in a venue geared toward the scientific enterprise.
Rather than speculate (again) about the cause of the problem I’d like to remind anyone who’s reading that there’s a simple solution and it doesn’t involve peer review (besides, let’s not forget there’s a reason why most peer reviewers do worse than coin flipping when it comes to separating the scientific wheat from the chaff). The solution is to admit only theories that have been borne out by observation of predictions made by the theory. That’s it. I propose treating scientific evidence like any other evidence.
Imagine a case involving a car wreck at an intersection with a traffic signal and a witness who will testify as to having seen Defendant enter the intersection against a red light. No difficult admissibility issue here. Now imagine the same facts except the witness didn’t actually observe the accident but nevertheless is prepared to testify that Defendant looks like the sort of person who’d run a red light and so probably did. Again, there’d be no angst over the (in)admissibility of such testimony. But when an expert converts “looks like the sort of person who’d run a red light” into the jargon of psychological science and converts “probably did” into something like “assuming independence among the variables I would estimate that there is less than a 1 in 2,000,000 chance that Defendant did not run the light” judges too often allow what would otherwise be seen as obvious baloney to be admitted as filet mignon. Yet all a judge need do to avoid the mistake is to first ask “have you or anyone else actually observed Defendant run a red light?”
We tend to lose sight of the fact that science and math are practical endeavors – the purpose of which is to make accurate predictions about questions like “where will this cannonball land if fired at an angle of elevation of 45 degrees and with a muzzle velocity of 1,054 ft/sec?” Practicality being paramount, conjectures about ballistics or any other aspect of nature are always tested the same way – subsequent observations are compared to predictions and whichever theory comes closest wins. There’s no trophy or ribbon for second place.
All “science” really is is a formalization of the way we’ve learned to understand the world: trial and error. It’s the way babies learn how to make Mom come running and it’s how toddlers learn to walk. Eventually we come to know how cars travel, how intersections are laid out, how things are perceived when at a distance and how a certain range of frequencies in the visible light spectrum means a traffic signal is red. Get it? We’re all scientists.
Now should the expert reply that the individual pieces that form the foundation of his theory have been tested and that ought to be enough remind him that that is no test at all of his theory that Defendant is a red light runner. Just as no judge would put her family on a type of airplane that has never been tested, irrespective of the evidence for the airworthiness of its component parts, the fact that the pieces of evidence that generate a hypothesis are sound doesn’t mean the hunch, however clever, will be borne out. One of the essential elements of the scientific revolution was the recognition that Nature is stranger than we can imagine; so much so that her secrets cannot yield even to the keenest mind precisely because we know so little about how the world works. Thus it was that originally through tragedy and investigation and later through extensive pre-maiden flight testing we learned that an airplane is more than the sum of its parts and that it has its own unique resonance vibration characteristics which if not recognized and controlled for can lead to disaster.
My argument then is as follows: if the prediction entailed by an expert’s opinion has never been observed outside the courtroom it is not evidence, because evidence is simply that which has been observed. If the response is that the harm alleged by plaintiff is in fact the observation that proves the predictive power of the expert’s hypothesis your reply should be that it is only evidence of our ability to string cherry picked pieces of the past together into a narrative to explain some other thing that lies in the past and thus beyond the reach of science. Real science means placing a bet, on your reputation if nothing else, on a forward looking experiment the results of which you do not know in advance. If on the other hand the opinion follows from some hypothesis that has indeed been tested (and I’ll save for some other day a discussion about why low power studies of small effects don’t constitute tests) then it ought to be admitted.
We’re seeing fewer and fewer cases tried and the enormous costs associated with expert witnesses and all the battles we have to fight over them just to get to the courthouse plays a very big part in it. Costs associated with experts increasingly force cases to be settled and others that might have merit never to be filed in the first place. Something needs to be done about it and it seems to me that the simple rule of requiring scientific evidence to be made out of the same thing as any other evidence, an observation that confirms the predictive power of the theory, would go along way toward untangling the current mess and would at the same time have the salutary effect of putting “evidence” back into “scientific evidence”.