The NAD recently determined that Euro-Pro could not support a claim that its Shark vacuum receives “more 5-star online reviews than any other vacuum brand.” To support the claim, Euro-Pro had looked at over 4,000 verified reviews on the websites of several major national retailers. Despite the number of reviews included in the analysis, the NAD determined that the data was still insufficient to support the broad claim.

Although Euro-Pro gathered review data from the top 85% of online retailers, the company actually based its claim on a much smaller subset of those reviews. For example, Euro-Pro did not include in its calculations reviews posted on Target, Best-Buy, or Costco websites because those sites did not indicate whether reviews could be verified as coming from actual purchasers. The company also excluded reviews from manufacturer websites because it was concerned about the reliability of those reviews. For example, Euro-Pro found that some reviews were incentivized by free products and was concerned that others could be manipulated or duplicated.

The NAD recognized that it may be difficult to parse through reviews and that advertisers may be left in a “Catch-22” situation. Including non-verified reviews could affect the reliability of the data, while excluding them could affect representativeness of the data. Nevertheless, this difficulty “does not relieve an advertiser from its obligation to provide appropriate and reliable substantiation for its advertising claims.” In this case, the NAD determined that by taking an all-or-nothing approach when deciding whether to include reviews from a certain website, Euro-Pro “materially undermined the reliability” of its calculations. Importantly, the NAD cautioned that “advertisers cannot base claims on tenuous evidence, simply because sufficiently reliable evidence is too difficult to collect.”

This isn’t the first case in which the NAD has considered the issue of crowdsourced reviews. As we wrote last year, the NAD had previously determined that reviews collected from retailer websites were insufficiently representative or reliable to support Euro-Pro’s “America’s Most Recommended” claim. In both decisions, the NAD stressed that it is not suggesting that no claim could be supported by crowdsourced data. Nevertheless, the decisions suggest that advertisers may face an uphill battle when trying to do so.