You may have seen the commercial on late night television. A glowing image of a human brain appears (along with a disclosure stating “dramatization”), with flashing lights pulsing through a crisscrossed mesh, depicting nerves. The voiceover intones, “Your brain is an amazing thing. But as you get older, it begins to change, causing a lack of sharpness or even trouble with recall.” So far, so good. Who, of a certain age, hasn’t experienced these symptoms?

The voiceover continues: “Thankfully, the breakthrough in Prevagen helps your brain and actually improves memory.” The flashing lights grow stronger and zoom more quickly across the neural net. “The secret is an ingredient originally discovered in jellyfish. In clinical trials, Prevagen has been shown to improve short term memory. Prevagen, the name to remember.”

A screen shot of the key frame, showing a graph of what appears to be recall improvement over time appears, along with a disclosure that states that “in a computer assessed, double-blinded, placebo controlled study, Prevagen improved recall tasks in subjects.”

Prevagen’s active ingredient was indeed isolated from bioluminescent hydrozoans (not technically jellyfish, as we know them, but close enough). It is called apoaequorin, a calcium-activated protein responsible for blue, phosphorescent light when activated. It is such an interesting protein that a trio of researchers received the 2008 Nobel Prize in Chemistry for studying how it all worked together in nature to produce bioluminescence in the absence of oxygen. Bioluminescence is a fascinating example of convergent, biochemical evolution that arose independently in unrelated genera over the eons, but I digress.

Needless to say, while Prevagen’s active ingredient has an interesting derivation, the question for federal and state regulators viewing Prevagen’s ads was this: does the substance actually work to improve memory and recall in humans? In January of this year, the Federal Trade Commission (FTC) and the New York Office of Attorney General (NYAG) brought suit against Quincy Bioaciences, the maker of Prevagen, for false and deceptive advertising. In addition to state law counts, their complaint alleged violations of Sections 5 and 12 of the FTC Act The FTC enforces Section 5(a) of the FTC Act, 15 U.S.C. § 45(a) (prohibits unfair or deceptive acts or practices in or affecting commerce); § 52 (prohibits false advertisements for food, drugs, devices, services, or cosmetics in or affecting commerce).

The complaint avers that the Prevagen memory improvement claims were unsubstantiated, despite the defendants’ reliance on a large, double blinded, placebo-controlled clinical study regarding Prevagen’s impact on cognitive functions such as memory and learning tasks. The FTC alleged that the Prevagen study, involving 218 human subjects, failed to demonstrate overall that its test subjects achieved statistically significant improvements in any of the nine, computer-assessed cognitive tasks that were studied over a period of 90 days. However, the FTC did acknowledge that the defendants had conducted a post hoc evaluation of the study results across 30 different views, a small number of which showed marginal benefits for the product. (Specifically, allegations before the court appeared to show that older treatment subgroups having low or mild initial impairment were later correlated with modest levels of improvement in three of the nine cognitive tests.) However, the FTC’s complaint rejected such post hoc analysis, which in its view “greatly increases the probability that the statistically significant improvements shown are by chance alone.” It went on to allege, “Given the sheer number of comparisons run and the fact that they were post hoc, the few positive findings on isolated tasks for small subgroups of the study population do not provide reliable evidence of a treatment effect.”

The complaint was thus unusual in some respects from the outset. The FTC and NYAG did not attack the overall set up and design of the study and also conceded that the defendants could indeed put forward some statistical evidence that the product may have benefited on some people, but they argued such evidence was made possible only by applying post hoc statistical methods that did not provide “reliable evidence of a treatment effect.” In this argument, the FTC was supported by consumer advocates and law professors, who claim that post hoc analyses of this sort are inadequate to support advertising claims, particularly where the overall study from which they were derived fails to show statistically significant results.

The key question, from a judicial, policy and statistical perspective, is stark: is post-hoc, subgroup analysis reliable substantiation for dietary supplement claims?

Surprisingly, the court resolved these questions on a motion to dismiss. In its opinion dismissing the case, the United States District Court for the Southern District of New York held that “the complaint fails to do more than point to possible sources of error but cannot allege that any actual errors occurred.” In other words, “the complaint does not allege facts from which it can be reasonably inferred that the representations at issue are false or unsubstantiated.” In a key passage, the court rejected the plaintiffs’ theory of liability, holding:

[P]laintiffs’ challenge never proceeds beyond the theoretical. They say that findings based on post hoc exploratory analyses have an increased risk of false positives, and increased probability of results altered by chance alone, but neither explain the nature of such risks nor show that they affected the subgroup performance in any way or registered any false positives. Nor do they give any reason to suspect that these risks are so large in the abstract that they prevent any use of the subgroup concept, which is widely used in the interpretation of data in the dietary supplement field. Thus, the complaint fails to show that reliance upon the subgroup data “is likely to mislead consumers acting reasonably under the circumstances,” as is necessary to state its claim. FTC v. LeadClick Media, LLC, 838 F.3d at 168

After dismissing the federal claims, the court went on to dismiss the pendent state claims for lack of jurisdiction, without further discussion of the merits. The FTC filed a notice of appeal on November 20, which will bring the issue before the Second Circuit.

On appeal, it is likely that the Second Circuit will address the core issue of sufficiency of the FTC allegations: did the FTC plead sufficient facts to show that the Prevagen claims were likely to mislead consumers acting reasonably under the circumstances? The answer is bound up in the sufficiency of statistical proof, which seems like the type of issue that ordinarily is not amenable to resolution on motion to dismiss. On this issue therefore, there is a significant chance that the Second Circuit may reverse. Of course, since the District Court dismissed without prejudice, the FTC also may replead, even if it does not prevail on appeal.

The appeal sets up a recurring and fundamental question regarding the relationship between statistical proof of substantiation and advertising claims. Does the FTC’s prior substantiation rule require that clinical studies include as part of their hypothesis at the outset the predetermined plan of subgroup analysis? Must the study protocol and methodology delineate exactly which groups will be analyzed to see if they benefit? More to the point: can post hoc subgroup analysis (less charitably, occasionally referred to as “data dredging”) ever support an advertising claim?

Subgroup analysis is simply the evaluation of the treatment effect on a small group of study participants defined by certain baseline characteristics. For example, if the study included 218 male subjects, ranging in ages from 40-90, a subgroup might include 75 men aged 65-90, with low or mild initial impairment as measured by baseline assessments.

Whether such a subgroup analysis is done can be specified in a predetermined manner, up front in the study protocol, or it can be carried out post hoc, after completing the study and observing the results. At first glance, the difference would seem to be trivial. However, the FDA has had a longstanding policy that permits the use of pre-determined subgroup analysis for clinical trials used to support drug applications. It also permits the use of post hoc subgroup analysis to evaluate safety outcomes. It prohibits, however, reliance on post hoc analysis to support claims of efficacy and specifies that any such analyses must be done by FDA scientists and not the drug sponsor. Why? The FDA is concerned both about conflicts of interest and that performing multiple subgroup analyses can increase the risk of false-positive findings. In other words, the more analyses one conducts, the more likely it is that one will find a false positive result because when groups get smaller, confidence intervals grow larger and statistical power is reduced.

How is the law to separate random occurrence from observed treatment effect and on whom should the risk of error fall if the court or regulator gets it wrong—the consumer or the advertiser)? Stated differently, do we care if many consumers see no benefit from a dietary supplement, so long as the advertiser has in fact obtained some statistical proof (thus holding their feet to the fire on prior substantiation)? As previously discussed, if the product is marketed as a cure or treatment for a serious medical condition, the calculus weighs against the advertiser, according to the FDA. But, Prevagen is a dietary supplement and not a drug, and the FTC has been repeatedly rebuffed in efforts to regulate supplements like drugs.

Quincy Biosciences represents the latest and perhaps most clearly defined controversy in which these issues will be tested. While the issues may at first blush seem technical and fact-specific, the policy implications are significant for industry and the FTC. We will be closely following the progress of this litigation.