More On

Technology-assisted review: Judges, defensibility and efficiency

Intelligent Categorization is an effective, cost-saving measure.

Time is money, and linear document review is almost prohibitively expensive because of the surge in electronic data volumes.

During the past year, a spotlight has been placed on technology-assisted review (or Intelligent Review Technology) which delivers the discerning analytics of a human review team in an automated platform capable of increased processing speed, consistency and accuracy.

Although Intelligent Review Technology is composed of three components – Automated Workflow, Intelligent Prioritization and Intelligent Categorization – the third facet has received the most attention. Intelligent Categorization (iC) is referred to by many names including “automated coding,” “intelligent coding” and “predictive coding.”

This component analyzes and learns from category decisions made by human reviewers suggesting categories for documents not yet reviewed. Today we will discuss the major sticking point throughout the industry: defensibility.

First and foremost, Intelligent Categorization is defensible if for no other reason than it being the same search technologies we have employed for over a decade applied in new ways. One of the early qualms about iC was that until the technology became court-tested, it was too risky to use. However, we are seeing an increased acceptance by practitioners in the field and positive comments from the judiciary.

For example, a recent article authored by U.S. Magistrate Judge Andrew Peck titled, “Search, Forward: Time for Computer-Assisted Coding,” explores the use of computer-assisted coding while discussing the perceived “judicial endorsement” of keyword searching.[1] A common theme throughout his article, which echoes comments made during the Carmel Valley eDiscovery Retreat, is questioning why lawyers seem insistent on receiving a judicial blessing of this technology before using it.[2]

To address the issue of judicial endorsement, Judge Peck provided a thorough analysis regarding the problems inherent in keyword searching, citing several well-known opinions, including United States v. O’Keefe,[3] authored by U.S. Magistrate Judge John Facciola. In O’Keefe, Judge Facciola addressed the lawyers and judges’ design of keyword search terms, noting it was “truly to go where angels fear to tread.” Creating and following keyword search terms blindly was an issue with Judge Facciola, and has been echoed in the several opinions cited by Judge Peck.

With that analysis aside, Judge Peck then turned to computer-assisted document review, which he defined as “tools… that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer.”

After discussing how computer-assisted review tools work, Judge stated that it “will be a long wait” for lawyers waiting for a court to conclude: “It is the opinion of this court that the use of predictive coding is a proper and acceptable means of conducting searches under the Federal Rules of Civil Procedure, and furthermore that the software provided for this purpose by [insert name of your favorite vendor] is the software of choice in this court."

In addition, Judge Peck noted if the use of computer-assisted review technology was presented or challenged in a case before him, he would “want to know what was done and why that produced defensible results,” perhaps being less interested in the “science behind the ‘black box’…than whether it produced responsive documents with reasonably high recall and high precision.”

Further, proof of quality control would be important to defending use of the technology. This is an extremely important point where the combination of sound iC training, validation review and statistical sampling should meet even the strictest of scrutiny.

Judge Peck concluded, “Until there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval. In my opinion, computer-assisted coding should be used in those cases where it will help ‘secure the just, speedy, and inexpensive’ (Fed. R. Civ. P. 1) determination of cases in our e-discovery world.”

Although we do not have a judicial opinion on record at this time other than this article from Judge Peck, there is no reason not to take advantage of this revolutionary technology. Our studies indicate use of Intelligent Review Technology, including Intelligent Categorization, can save up to 50 percent on document review costs.

Indeed, these studies are consistent with findings published by non-profit agencies including the eDiscovery Institute which released a survey showing the use technology equivalent of Intelligent Categorization resulted in reduced review costs by 45 percent or more.[4]

In short, Intelligent Categorization is a defensible, effective, cost-saving measure that leverages existing search technology and the work of talented attorneys to decrease the time required to complete document review. It is designed to meet flexibility and repeatability needs of the client, and is proving to be the key differentiator in the ability to respond to electronic discovery demands quickly and proportionately.


[1] U.S. Magistrate Judge Andrew Peck, “Search, Forward: Time for Computer-Assisted Coding”, Law Technology News. Available at, http://www.law.com/jsp/lawtechnologynews/PubArticleLTN.jsp?id=1202516530534&Search_Forward_Time_for_ComputerAssisted_Coding. Last accessed Oct. 19, 2011.

[2] Chris Dale, “Judge Peck and Predictive Coding at the Carmel eDiscovery Retreat”, the eDisclosure Information Project. Available at, http://chrisdale.wordpress.com/2011/08/02/judge-peck-and-predictive-coding-at-the-carmel-ediscovery-retreat/. Last accessed Oct. 19, 2011.

[3] 2008 WL 449729 (D.D.C. Feb. 18, 2008).

[4] See "eDiscovery Institute Survey on Predictive Coding," available at http://www.lawinstitute.org/.

Join the Conversation

Advertisement. Closing in 15 seconds.