The next step for technology-assisted review: Unlocking the black box

A step-by-step approach to establishing a more defensible methodology

With technology-assisted review (TAR) gaining judicial support, practitioners now face the next hurdle: defending its accuracy and reliability.

TAR is something of a misnomer, however. Left to its own devices, the TAR algorithm would remain a black box that yields inexplicable results. The best way to achieve transparency is to leverage the know-how of experienced lawyers and other experts who can establish sound processes and work with opposing counsel to preempt disputes. As Judge Andrew Peck stated in Da Silva Moore v. Publicis Groupe, such transparency “allows the opposing side to be more comfortable with computer-assisted review and reduces fears about the so-called black box of the technology.”

A more appropriate name for TAR might be “human-assisted review.” Below, we touch on a number of ways human involvement can help avoid discovery disputes, enhance accuracy and lead to greater defensibility.

1. Collaborate with opposing counsel.

Judges have made clear that they expect cooperation of litigants in cases with electronically stored information (ESI), many citing The Sedona Conference’s “Cooperation Proclamation” as guidance. Chief among them was Judge Paul Grimm, who bemoaned the practice of serving broad, redundant and unnecessarily burdensome discovery requests in Mancia v. Mayflower Textile Services Co., noting: “The party propounding Rule 33 and 34 discovery does not know enough…to more narrowly tailor [the requests], but this would not be so if lawyers approached discovery responsibly, as the rule mandates, and met and conferred before initiating discovery.”

A first step is cooperation with opposing counsel. Counsel should ideally agree upon an appropriate protocol for review and production, addressing: the scope of discovery; technology, formatting and quality control; and inadvertent disclosure:

a. Scope of discovery. As a starting point, the scope of discovery should be negotiated by the concept of proportionality as set forth in Rules 1 and 26(b)(2)(C) of the Federal Rules of Civil Procedure.

One approach is a phased discovery schedule, as ordered in Tamburo v. Dworkin. There, Judge Nan Nolan required the parties to engage in “cooperative discussions to facilitate a logical discovery flow,” beginning with Rule 26(a) initial disclosures, narrowing it to the “claims most likely to go forward” and then prioritizing according to sources that are the least “expensive and burdensome.”

When addressing scope, opposing counsel should agree to ESI sources and key custodians. If necessary, parties can conduct limited depositions of custodial witnesses to determine the universe of documents at issue.                  

b. Technology, formatting and quality control. Opposing counsel should negotiate the selection of the TAR platform and production format. If not, the court may impose its own solution, as the Delaware Chancery Court did in EORHB, Inc. v. HOA Holdings. There, the judge ordered both parties to use TAR from the same e-discovery vendor. If they could not agree on the vendor, he threatened, “you can submit names to me and I will pick one for you.” As most counsel would prefer to control the course of pretrial discovery, it is advisable to reach consensus on these central issues.

In Da Silva Moore, Judge Peck made two recommendations to eliminate discovery disputes. First, he suggested that the reviewing party share the training set and its coding with opposing counsel to demonstrate that the computer is trained properly. Second, he encouraged parties to collaborate in designing a quality-control methodology at the outset to avoid later arguments that the TAR algorithm is inaccurate.

The parties used an alternative approach in In re Actos (Pioglitazone) Products Liability Litigation, where they agreed to select four custodians’ e-mails to create a sample set. The parties would then appoint three “experts to work collaboratively to train the TAR software.” Following the TAR process, the documents would be randomly sampled for quality control, and the defendants would have the opportunity to manually review them.

c. Protection against inadvertent disclosure. The parties have a mutual interest in protecting their documents against the inadvertent disclosure of privileged or confidential information. They may choose to enter a nondisclosure agreement or a protective order. If proposing a clawback agreement, it is best to submit it as a proposed court order to maximize protections afforded under Federal Rule of Evidence 502.

2. Establish a review methodology.

In any TAR undertaking, lawyers most familiar with the case first review a sample data set for responsiveness and privilege. The software then applies the coding from the sample across the entire population, applying statistical sampling and quality control techniques to refine its algorithm, which improves the responsiveness score given to each document. The document scores can then be used to prioritize review. Establishing sound processes and a defensible methodology requires a team of appropriate experts, including:

a. Senior-level review team. The attorneys who review the sample set must be well-versed in the issues relevant to the litigation and be trained to code the documents accurately. Because their assessments will be used to train the TAR software, they must code documents carefully documents; if there are debatable documents, this is the time to discuss and reconcile. This team will certify that the results of the TAR project are accurate.
b. Technologists. A technology expert can guide the day-to-day management of the TAR tool, increasing efficiency and transparency. The technologist should assist with document flow, ensuring that the right team members are reviewing the right documents at the right time, and carefully document the process. As Judge Peck remarked, clients need a technologist who can attend “bring your geek to court day” and explain the technology to the court and opposition—in layman’s terms.
c. Statisticians. Sampling and measurement are critical components of TAR. A statistician validates the reliability of sampling techniques as well as the quality of results. During the set-up phase, the statistician will apply sampling techniques to determine the rate of responsiveness in the population. Statistical sampling quantifies the reliability of a system’s assessments through testing and comparing them to the coding calls the legal team has made. It also determines a confidence rating for assessing whether the software has returned consistent, reliable results. These metrics are used to buttress quality assurance, showing that the software yielded responsive documents with a reasonably high recall and high precision
d. Linguists. TAR relies upon the successful detection of linguistic patterns through statistical algorithms. A linguist can evaluate the substance of the entire data set and ensure keywords and related terms are included within seed sets so that responsive documents are not overlooked. There is another important responsibility: excluding terms that are overly broad to the point that the objective is lost. The linguist can enhance the responsiveness rate results by generating sophisticated search terms to maintain context.

3. Use an adaptive approach.

The “one and done” approach does not work in TAR. The technology should be trained iteratively, allowing it to learn and adapt to new data it gathers over the course of the modeling process. Rigorous testing and analysis of preliminary outputs should accompany each training cycle, improving quality and providing reviewers with opportunities to course correct. 

4. Create a thorough audit trail.

Transparency involves a thoughtful explanation of the underlying rationale of the methodology and detailed documentation of every step of the process. Furthermore, it delineates the sampling parameters and the acceptance/rejection rates for the data set to confirm its accuracy.

It is best to invest time, care and attention at the outset of a TAR project to get it right. A thorough approach pays off through increased efficiency, reduced costs and demonstrable performance metrics that should hold up in front of opposing parties and the court.

Contributing Author

author image

Randall Burrows

Randy Burrows is the chief executive responsible for the operational performance of Xerox Litigation Services, including the overall management of more than 400 employees and...

Bio and more articles

Join the Conversation

Advertisement. Closing in 15 seconds.