By 1990 Paul E. Meehl had had enough. He’d had enough of lazy scientists polluting the literature with studies purporting to confirm fashionable theories that in fact couldn’t even be tested; enough of cynical scientists exploiting the tendency of low power statistical significance tests to produce false positive results just so they could churn out more of the same; and enough of too many PhD candidates, eager to get in on what Meehl called “the null hypothesis refutation racket”, who were unashamedly ignorant of the workings of the very mathematical tools they hoped to use to further muddy the waters with their own “intellectual pollution.” He called on them to give up their “scientifically feckless” enterprise and to take up honest work, something more suited to their talents – selling shoes perhaps. The shoe salesmen as we now know would not give up so easily.
Rather than a tiresome diatribe in a meaningless war of words among academics, what Meehl wrote in Why Summaries Of Research On Psychological Theories Are Often Uninterpretable is one of the best explanations you’ll ever read about what went wrong with science and why. And if you’re curious about whether he (posthumously) won the argument, the results are now coming in. Of the first 100 important discoveries in the field of psychology tested to see if they are in fact reproducible, all of which “discoveries” by the way were peer reviewed and published in prominent journals, only 39 passed the test. The obvious conclusion is that the literature has indeed been thoroughly polluted.
Meehl demonstrated that any time tests of statistical significance are used to test hypotheses involving complex systems where everything is correlated with everything, as in the psyche and the body, a weak hypothesis (which is to say one that is merely a suspicion not built upon other theories about underlying mechanisms that have been rigorously tested) carries an unacceptably high risk of producing a false positive result. This problem is not limited to psychology. It is estimated to arise in the biomedical sciences just as often.
Fortunately, those who in the past funded the “null hypothesis refutation racket” have begun to take notice, and action. The National Children’s Study which recently got the axe from the NIH (following a review by the NAS) is the most notable example thus far. Criticized for years as being short on robust hypotheses and long on collecting vast amounts of data on environmental exposures and physical, behavioral and intellectual outcomes it was finally determined that the study was “unlikely to achieve the goals of providing meaningful insight into the mechanisms through which environmental factors influence health and development.” That the study would have found all sorts of statistically significant correlations between environment and outcomes was a given. That none could reliably be said to be causal was the problem.
The shoe salesmen turned scientists had a good run of it. Uncounted billions in grant money went into research founded on nothing more than the ability of computers to find correlations among random numbers and of humans to weave those correlations into a plausible explanation. Scientists in the right fields and also blessed with earnestness or at least the skills of an advocate really got lucky. They became expert witnesses. But now, frustrated with research that never seems to go anywhere and alarmed that good research is being obscured by bad, funders are directing their money towards basic research. And it’s a target rich environment. Take for example the remarkable discovery that an otherwise harmless amoeba can for purposes known only to itself resuscitate a moribund Listeria monocytogenes, let it grow and multiply within itself and then release the bacteria into what was previously thought to be an L. monocytogenes-free environment.
Alas, such research is hard and its chances of success, unlike significance testing, wholly unpredictable. It looks like the shoe salesmen’s luck has run out. That one of their last redoubts has turned out to be the courthouse is perhaps the most remarkable development of all.