While recognizing the benefits of using new sources of information, the National Advertising Division (NAD) decided that an advertiser’s use of aggregated online reviews was insufficiently reliable and representative to support a claim that its vacuum was “America’s Most Recommended.”

Dyson Inc. challenged Euro-Pro’s advertising claims for its Shark brand vacuum cleaners in television commercials, infomercials, and on its Web site. The advertiser touted the Shark line as “America’s Most Recommended Vacuum Brand,” with a disclaimer that the claim was “Based on percentage of consumer recommendations for upright vacuums on major national retailer websites through August 2013, U.S. only.”

Euro-Pro based the claim on a quarterly survey of the Web sites of certain online retailers (including Amazon and Target) that sell upright vacuum cleaners and also solicit customer reviews. After collecting and amalgamating the data – specifically the answers given by consumers regarding whether they would recommend the product under review – Euro-Pro then tallied and compared the percentage of recommendations for various brands.

Dyson identified several problems with the claim support. Not only did the universe of consumers not represent American vacuum cleaner consumers (likely excluding purchasers of vacuums from retail locations, which constitute 84 percent of all buyers in the country), but the data was also unreliable, the challenger said. Euro-Pro could not provide any evidence that the largely anonymous population of online reviewers actually bought and used the product, were American customers, based their recommendations on actual experience with or use of the product, or represented a demographically representative data set by age, region, gender, or income.

In response, Euro-Pro pointed out that since online reviews have become an important influence in many consumers’ buying decisions, they provide a new, reliable way to discern consumer opinions. Consumers understand the limitations of online reviews, the advertiser added.

The NAD agreed that while consumers increasingly use and rely upon online reviews – and that the reviews provide “important new sources of information” for advertisers – “the standards of truthfulness, reliability, and representativeness to which advertisers’ substantiation is held remain the same.”

Euro-Pro’s substantiation failed to support its claim, the self-regulatory body concluded. A large majority of consumers continue to purchase their upright vacuum cleaners in brick-and-mortar stores and are far less likely to submit online reviews. The reviewer demographics were unverified, which presented a high possibility of fraud. The advertiser could not even verify whether the reviewers actually owned or used the vacuums. The Web sites from which Euro-Pro culled its data varied in how they worded their recommendation questions and had different policies for how long reviews remained available on the site.

“[T]here is a difference between the reliability of online reviews for consumers who directly read and analyze them when considering which product to purchase and their reliability as the basis of a broad advertising claim regarding the opinions of a wide swath of U.S. consumers,” the NAD wrote.

Finding the evidence insufficiently reliable or robust to provide a reasonable basis for the claim, the NAD recommended that the “most recommended” claim be discontinued.

In its advertiser’s statement, Euro-Pro said it intends to appeal to the National Advertising Review Board, so stay tuned.

To read the NAD’s press release about the decision, click here.

Why it matters: In light of this decision, advertisers using crowd-sourced or aggregated data collected across multiple platforms to support claims should take heed and consider whether such data would pass muster under the NAD’s strict and evolving standards of truthfulness, reliability, and representativeness. Advertisers considering the use of new types of claim support such as online reviews should also keep in mind that “[w]hile NAD will consider information sourced from new technology as claim support, NAD will hold that evidence to the same standards of reliability that it has with more traditional forms of substantiation.”