“How are courts doing when it comes to interpreting the statistical data that goes into their decision-making?” That was a question posed by someone in the audience at a presentation I gave recently. I was discussing, among other things related to the perils of litigating statistical inferences, the recent paper “Robust Misinterpretation of Confidence Intervals.” It reports on the results of a study designed to determine how well researchers and students in a field that relies heavily on statistical inference actually understand their statistical tools. What it found was a widespread “gross misunderstanding” of those tools among both students and researchers. “[E]ven more surprisingly, researchers hardly outperformed the students, even though the students had not received any education on statistical inference whatsoever.” So, returning to the very good question, how are our courts doing?

To find out I ran the simple search “confidence interval” across Google Scholar’s Case Law database with the date range set to “Since 2014″. The query returned 56 hits. Below are eight representative quotes taken from those orders, reports and opinions. Can you tell which ones are correct and which constitute a “gross misunderstanding”?

Before I give you the answers (and thereafter some hopefully helpful insights into confidence intervals) I’ll give you the questionnaire given to the students and researchers in the study referenced above along with the answers. Thus armed you’ll be able to judge for yourself how our courts are doing.

Professor Bumbledorf conducts an experiment, analyzes the data, and reports:

The 95% confidence interval for the mean ranges from 0.1 to 0.4

Please mark each of the statements below as “true” or “false”. False means that the statement does not follow logically from Bumbledorf’s result.

(1) The probability that the true mean is greater than 0 is at least 95%.

(2) The probability that the true mean equals 0 is smaller than 5%.

(3) The “null hypothesis” that the true mean equals zero is likely to be incorrect.

(4) There is a 95% probability that the true mean lies between 0.1 and 0.4.

(5) We can be 95% confident that the true mean lies between 0.1 and 0.4.

(6) If we were to repeat the experiment over and over, then 95% of the time the true mean would fall between 0.1 and 0.4.