“… 62 of the plaintiffs … had statistically significantly higher rates of genitourinary and reproductive illness and procedures compared to the rest of the county.”

That’s from Whitlock v. Pepsi Americas, a hexavalent chromium case, and it was part of the reasoning that went into the court’s decision to grant plaintiff leave to supplement her expert report based on this “new scientific information.” I’ll explain just why the reasoning is deeply flawed shortly but first I’ll answer the question of why you should care. If the sort of risk factor epidemiology on which the court rests its opinion is really science, and if the sort of data dredging that that went into the study from which the “new scientific information” was inferred is really the scientific method, then anything can always be shown scientifically to cause everything and Daubert has been finally and thoroughly eviscerated.

Whitlock’s underlying facts are typical of those mass tort cases that follow the factory closing of a sparsely populated county’s largest employer. Toxins are identified and the lawyers file suit on behalf of dozens or hundreds of clients with conditions that might be associated with exposure. Here approximately 1,000 toxic tort cases blamed pollution from Remco Hydraulics, Inc.’s Willits, CA manufacturing plant for a host of ailments. Those cases have spawned numerous interesting orders and opinions, Whitlock being only the most recent.

The district court had previously found the proposed exposure and causation testimony of Plaintiff’s experts to be unreliable and accordingly granted summary judgment in favor of Defendants but the U.S. Court of Appeals, Ninth Circuit in an unpublished opinion held that the trial court had abused its discretion; and plaintiff was back in business. Meanwhile, a study to identify possible risk factors associated with living in Willits was being updated but the results arrived after Plaintiff’s deadline to amend her experts’ reports. This iteration of Whitlock then the court’s determination that the study update, what it deemed “newly discovered evidence in support of her claims”, constituted good cause for amending her experts’ reports.

The study, Longitudinal analysis of health outcomes after exposure to toxics, Willits California, 1991-2012: application of the cohort-period (cross-sequential) design  looks at the incidence of groups of ailments and/or procedures defined by “body system” noted at the time of any patient discharged between 1991 and 2012 sorted by the decade in which each patient was born (’40s, ’50s, ’60s, ’70s or ’80s). Then the rate of each grouping, for each decade of birth, for patients with a residential address containing the Willits ZIP-code, is compared to the rate of each grouping, by decade of birth, for patients who lived in the same county but didn’t have a Willits ZIP-code (a/k/a ROC, or “rest of the county”) generating thereby a relative risk. The authors also calculated the relative risk (Willits ZIP vs ROC) for hospital admissions, discharges and days spent in the hospital.

Willits men and women, sorted this way, were more likely to be hospitalized and to have spent more time in the hospital than non-Willits ZIP-code residents of the same county. And as for “body systems” Willits women were “at increased risk for all measures” whereas men “were at increased risk for all measures except genitourinary system diagnoses and procedures, and gender based procedures and cancer.” The authors conclude from their study that the people of Willits were at increased risk of “poor health”, that the burden on the community is “incalculable”, and that the cost of to the public is “enormous.” If you think those are reasonable inferences given this data you’re about 20 years late to the scientific community’s realization that risk factor epidemiology isn’t science, generates more false leads than promising hypotheses and is easily exploited.

In 1994 the late Petr Skrabanek wrote The Emptiness of the Black Box.  It wasn’t the first journal article to call BS on risk factor epidemiology but it was the best; and coming from a leading epidemiolgist and public health advocate it was also the most powerful of its time. Seven years later, reflecting on the fact that risk factor epidemiology had not only failed to uncover the cause of “a disease which showed an epidemic rise in industrialized countries” but had falsely indicted certain exposures thereby impeding attempts at prevention and cure, the new editors of The International Journal of Epidemiology wrote Epidemiology – is it time to call it a day? In it they discuss the failures, the lack of rigour in the discipline and the already obvious decline in the use of risk factor epidemiology to identify causes of health problems in groups of people. Over the last ten years (as we’ve chronicled repeatedly) the status of risk factor epidemiology has only fallen further. Imagine the money wasted, hopes dashed and time lost in the largely fruitless search for reliable markers of cancer prognosis despite the fact (or actually because of the fact) that Almost All Articles on Cancer Prognostic Markers Report Statistically Significant Results 

Now, the fact that most risk factors identified by risk factor epidemiology turn out to be false does not lead necessarily to the conclusion that having a Willits ZIP-code and having been born sometime between 1940 and 1989 doesn’t put you at greater risk of being hospitalized some time between 1991 and 2012 (though it ought to lead you to be intensely sceptical of such a claim). Furthermore, I’ve no reason to believe that the authors engaged in the sort of post hoc rationalizing, p-hacking, multiple comparison testing and selective publication responsible for much of the now widely recognized crisis of unreproducible “science”. But what I think I can demonstrate rather easily is that any inference about the cause of Whitlock’s ailment that is drawn from this data is fatally flawed.

Remember that business about sorting patients’ reason for hospitalization not by the ICD 9 disease codes but rather by “body systems”? A little rummaging around on the web turned up the “Level 1 of the Multi-level Clinical Classification Software” that does the sorting along with a handy appendix. It turns out that the purpose of such sorting hasn’t anything to do with discovering the cause of diseases but rather everything to do with analyzing and predicting healthcare costs. That doesn’t mean it can’t be (somehow) used to discover the causes of illness but it does make it an odd choice. You can find it at Healthcare Cost and Utilization Project – HCUP: A Federal-State-Industry Parthership in Health Data .

In any event go to appendix C1 and scroll down until you get to body system 10 – Diseases of the genitourinary system. Body system 11 is Complications of pregnancy; childbirth and the puerperium. The list of the procedures by body system can be found in Appendix D1. Operations on the urinary system are found in category 10 and operations on the female genital organs is category 12. These are the systems and categories of procedures to which the court was referring when it wrote that “… 62 of the plaintiffs had statistically significantly higher rates genitourinary and reproductive illness and procedures …”.

The plaintiff’s argument then goes like this: A peer reviewed and published study has shown a statistically significant increased risk of being hospitalized between 1991 and 2012 for treatment of a genitorurinary system problem among women with a Willits ZIP-code who were born between 1940 and 1989. I have a Willits ZIP-code, was born between 1940 and 1989 and had a genitorurinary system ailment. Therefore my ailment was caused by living within the Willits ZIP-code. Somehow from there must come “and living in the Willits ZIP-code meant I was exposed to hexavalent chromium so hexavalent chromium caused my genitourinary problem!” Since there was no data collected on any of the patients to determine whether they were actually exposed to hexavalent chromium, lived downstream, upstream, worked in or ever drove past the Remco factory the analytical gap between ZIP-code and hexavalent chromium exposure/dose would appear unbridgeable. But let’s assume it is because the argument is still demonstrably absurd.

If you’ve looked through the list of conditions and operations you know what I mean when I write that it’s full of cross-examination gold. However, given the highly personal and sensitive nature of these subsets of body systems and procedures I’ll use another category that was also statistically significantly elevated among patients with a Willits ZIP-code – Infectious and parasitic diseases (which is body system 1). In fact, Willits women had a slightly higher risk of infectious and parasitic diseases than of diseases of the genitorurinary system. And the last clue you need to figure out what’s going on here is the discovery that Willits women were at a statistically significantly increased risk for all of the categories of body systems and procedures for almost all years.

So let’s take, in honor of the 110th anniversary of Robert Koch’s Nobel Prize in Physiology or Medicine for his discovery of Mycobacterium tuberculosis, 1.1.1 – Tuberculosis from body system 1 and plug it into a hypothetical plaintiff’s argument.

  1. A peer reviewed and published study has shown a statistically significant risk of being hospitalized for infectious and parasitic diseases among women with Willits ZIP-code born any time between 1940 and 1989.
  2. A woman with a Willits ZIP-code born between 1940 and 1989 has been afflicted by tuberulosis, a member of the set of infectious and parasitic diseases.
  3. A Willits ZIP-code and exposure to hexavalent chromium are (somehow) the same thing
  4. Therefore, hexavalent chromium exposure caused plaintiff’s tuberculosis (Koch’s postulates, M. tuberculosis and the Nobel Prize notwithstanding)

Hopefully I’ve made my first point.

My second arises out of the sentence that launched this post. People don’t have rates of disease. They either get a disease or they don’t. Populations have rates of disease. And when you go from data about populations to inferences about individuals you commit a logical fallacy known as the ecological fallacy. The court’s reasoning is a perfect example of it.

Any finally my third point. If you’ve ever worked on one of these plant closure / toxic tort cases in a down and out county you know why the people who lived near the plant have more hospitalizations and procedures. They disproportionately had the best jobs in town meaning more money and more access to health care. In other words, as is so often the case in these risk factor studies, the authors have probably pointed the arrow of causation in the wrong direction. Living in Willits didn’t cause poor health and hospitalizations. Not living in Willits meant poor access to health care dispensed by hospitals.