As covered in the April edition of this Update, Southern District of New York Magistrate Judge Andrew J. Peck issued an opinion and order earlier this year expressly approving the use of predictive coding in pre-trial discovery. Da Silva Moore v. Publicis Groupe, _ F. Supp. 2d _, 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012) (“Da Silva Moore”). This decision marked the first time that any court had addressed the issue of technology-assisted review, providing guidance to parties hesitant to use such technologies due to legal uncertainty. Since the time of our last Update, the district court has addressed — and affirmed — the Da Silva Moore decision on appeal, and other courts have provided additional guidance on when and how predictive coding technology may be used to expedite the document review process.

“Predictive Coding” Explained

Judge Peck’s February order provided a helpful explanation of predictive coding. It is a computerized process he described as using “sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer.” Id. at *2. Typically, a senior attorney will review and code a “seed set” of documents for responsiveness and the computer will identify properties of those documents. Id. It will then use those properties to code other documents until the “system’s predictions and the reviewer’s coding sufficiently coincide,” at which point “the system has learned enough to make confident predictions for the remaining documents.” Id. The result is that hundreds of thousands of documents (or more) are reviewed by the computer, while only a fraction of those documents, perhaps a few thousand, are reviewed by attorneys. This approach may drastically reduce the time and money spent on document review. But no court had addressed the issue prior to Da Silva Moore.

District Judge Approves Judge Peck’s Decision

Judge Peck’s opinion in Da Silva Moore was not only the first judicial guidance on predictive coding technologies, but offered an enthusiastic endorsement of the technology’s merits. In his February opinion, Judge Peck stated:  

What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Counsel no longer have to worry about being the “first” or “guinea pig” for judicial acceptance of computer-assisted review.

Id. at 15. District Judge Andrew Carter adopted Judge Peck’s rulings on April 26, 2012. Noting the “highly deferential” standard of review for a nondispositive order issued by a magistrate judge, Judge Carter stated the rulings “are well reasoned” and “consider the potential advantages and pitfalls of the predictive coding software.” Da Silva Moore v. Publicis Groupe, 2012 WL 1446534, at *2 (S.D.N.Y. Apr. 26, 2012).

The electronically stored information (ESI) protocol approved by Judge Peck is a cooperative process to be carried out and progressively evaluated in stages. The plaintiffs’ objections expressed concern regarding the reliability of the protocol approved by Judge Peck. See Pls. Rule 72(a) Objection to the Mag.’s Feb. 8, 2012 Disc. Ruling, at 13-18, Da Silva Moore, Case No. 11-cv-1279 (S.D.N.Y. Feb. 22, 2012), Dkt. 93. In response, Judge Carter reiterated Judge Peck’s rationale, pointing out that the protocol allows room for evaluation as litigation continues. Da Silva Moore, WL 1446534, at *2. If the software turns out to be flawed or if the plaintiffs are not satisfied with the documents they are getting, “the parties are allowed to reconsider their methods and raise their concerns with the Magistrate Judge.” Id. Thus, Judge Carter adhered to Judge Peck’s characterization of the “unreliability” argument as “premature” and “speculative.” Id. In so doing, Judge Carter provided an additional bolster to the use of predictive coding technologies.

Global Aerospace

Recently, a second court addressed predictive coding: the Virginia Circuit Court for Loudon County. Global Aerospace Inc. v. Landow Aviation, L.P., 2012 WL 1431215 (Va. Cir. Ct. Apr. 23, 2012) (“Global Aerospace”). In Global Aerospace, a suit over collapsed airplane hangars, the defendants moved for a protective order approving their use of predictive coding over the plaintiffs’ objection. This situation is distinct from Da Silva Moore, where the parties had initially both agreed to the technology’s use, but differed on its implementation.

The defendants stated, in support of their motion to use predictive coding software, that they had the approximate equivalent of two million documents in ESI. Mem. In Supp. of Mot. for Protective Order Approving the Use of Predictive Coding, Global Aerospace, 2012 WL 1419842, at *2 (Va. Cir. Ct. Apr. 9, 2012) (“Defs.’ Mot.”). They estimated that manual review of two million documents would take 20,000 man-hours, cost two million dollars, and yield only 60% of the relevant documents. Id. Keyword searching, the defendants suggested, would be cheaper but yield only 20%. Id. They argued that predictive coding, however, would be significantly more cost-effective, take much less time, and yield 75% or more of potentially relevant documents. Id.

In their opposition to the motion, the plaintiffs responded that a process that misses up to 25% of relevant documents is not acceptable. Opp. of [Certain] Pls. to Defs.’ Mot., Global Aerospace, 2012 WL 1419848, at **1-2 (Va. Cir. Ct. Apr. 16, 2012). They argued it was the defendants’ obligation to produce 100% of the relevant documents, and suggested that relying on “human review” in addition to some keyword searching would accomplish that. Id. at *2. Their brief stated, “[n]o computer program is an adequate substitute for having human beings review and sort the documents.” Id.

Circuit Judge James H. Chamblin apparently disagreed, approving defendants’ motion in a three-sentence opinion, and allowing the defendants to proceed with the use of predictive coding technologies. Global Aerospace, 2012 WL 1431215, at *1. He further gave the parties 60 days for processing and another 60 for production. Id. He did not address the issue of a protocol for implementing predictive coding, the issue in dispute in Da Silva Moore. He did, however, leave the door open for future objections, noting, “[t]his is without prejudice to a receiving party raising with the court an issue as to completeness or the contents of the production or the ongoing use of predictive coding.” Id.  

Kleen Products

Meanwhile, as of this writing, in an ongoing discovery dispute in the Northern District of Illinois, Magistrate Judge Nan Nolan is addressing the use of predictive coding in an antitrust case, Kleen Products, LLC v. Packaging Corporation of America, 775 F. Supp. 2d 1071 (N.D. Ill. 2011) (“Kleen Products”). Kleen Products presents a unique situation in that the plaintiffs have asked Judge Nolan to require the defendants to redo discovery using more advanced technology-assisted review, such as predictive coding. The defendants had already completed a substantial portion of their document review and produced over a million pages using Boolean keyword searches. To date, the number has surpassed three million pages produced. Joint Status Conference Report No. 3, at *3, Kleen Products, Case No. 10-cv-5711 (N.D. Ill. May 17, 2012), Dkt. 319 (“Status Conference Report”). The plaintiffs view Boolean searches as somewhat outdated and contend that more advanced technology, like predictive coding, would have been more accurate in retrieving the right documents. Id., Ex. B at 9.

Judge Nolan told the parties early on that she is “a believer of [P]rinciple 6 of Sedona,” referring to the Sedona Conference, a not-for-profit legal research and education organization committed to addressing current issues in antitrust law, complex litigation, and intellectual property rights. Principle 6 states:

Responding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronic data and documents.

The Sedona Principles: Best Practices Recommendations & Principles for Addressing Electronic Document Production, The Sedona Conference Working Group on Best Practices for Electronic Document Retention & Production (WG1), July 2005, at 12. This particular Working Group has conducted extensive research and writing on issues relating to e-discovery.

Transcripts from evidentiary hearings reveal that Judge Nolan is encouraging the parties to arrive at a mutually agreeable ESI search strategy based on all the work the defendants have already done using Boolean searches. She gave the parties a “homework assignment” to meet with one another and try to come up with a way to “tweak” the Boolean searches, with the goal of allaying the plaintiffs’ concerns regarding the responses the searches will likely yield. Status Conference Report, Ex. B at 300.

Although she has yet to issue an opinion, Judge Nolan’s admonitions to the parties to come to an agreement and her reference to Principle 6 suggest some reluctance about mandating the use of more advanced technology-assisted document review. It remains to be seen whether Kleen Products will provide any additional guidance on whether and when a party may be required to use predictive coding technologies.


Courts are increasingly confronting the proposed use of predictive coding and technology-assisted review approaches. As these recent cases illustrate, how and when these approaches are used, as well as who is requesting such use, may be critical factors in any judicial analysis, and no common pattern has yet emerged. If a litigant is considering the use of technology assisted review, it is advisable to consult with e-discovery counsel.

Summer Associates Daniel Lennard, David Mayo, Anna Schoenfelder, and Alexander Traum.