A recent slew of articles have stoked the age old “man versus machine” debate, often suggesting that man is on the losing end of yet another automation battle. This Homeric contest has recently been waged publically by Watson (IBM’s artificial intelligence computer), which competed on the quiz show Jeopardy and beat two human challengers—the biggest all-time money winner and the record-holder for the longest win streak. This battle has now transitioned to a larger stage, the U.S. economy, where some have gone so far as to suggest that computers are already winning a much more important skirmish.
In an article entitled “Technology Is Eliminating Higher-Skill Jobs”, National Public Radio touted another Watson win, over teams from MIT and Harvard in a “Race Against the Machine” contest. In the NPR piece Andrew McAfee, the MIT researcher who helped organize the conference, cited electronic discovery as the prime battlefield upon which machines are winning. "We see already that the work of legal discovery — in other words, sitting around and reading huge volumes of documents at the early stage of a lawsuit ... is being very quickly and very heavily automated. And, by one estimate, it lets one lawyer do the work of 500."
As the economy continues to bump along the bottom of a recession, quotes like these don’t help many in the legal profession (particularly contract reviewers) sleep well at night. While sensational headlines help generate readership, the real question is: how is technology impacting (for good or for ill) the future practice of law?
The best place to begin is by examining the electronic discovery review process, which has historically been handled by attorneys and paralegals who’ve had to diligently read documents and make a variety of privilege and relevancy judgments. The first hurdle in the man versus machine debate is the notion that eyes-on review is the gold standard to which all other machine approaches must be measured.
In the 2007 Sedona Conference “Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery”, the influential think tank cautions against believing in the manual review myth: “Even assuming that the profession had the time and resources to continue to conduct manual review of massive sets of electronic data sets (which it does not), the relative efficacy of that approach versus utilizing newly developed automated methods of review remains very much open to debate.”
Sedona puts it politely. But, the truth is that the human review process is just not very good from a precision and recall perspective. In one of the seminal pieces on the topic (“Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review”), authors Maura Grossman and Gordon Cormack note that “manual review is far from perfect — [i]t is well established that human assessors will disagree in a substantial number of cases as to whether a document is relevant, regardless of the information need or the assessors’ expertise and diligence.” Despite the flaws, the authors note that the legal field tends to nevertheless give the human review process the “gold standard” imprimatur for a few nuanced reasons:
1. “The information need (responsiveness or privilege) is more precisely defined for e-discovery than for classical information retrieval.
2. Lawyers are better able to assess relevance and privilege than the non-lawyers typically employed for information retrieval tasks.
3. The most defensible way to ensure that a production is accurate is to have a lawyer examine each and every document.”
The challenge is how to improve upon the human review process without throwing the baby out with the bath water. This is why many experts in the e-discovery field are talking about “technology-assisted review,” which is a broad umbrella concept that involves the use of many approaches including keyword search, Boolean search, conceptual search, clustering, relevance ranking, sampling and predictive (aka computer-assisted) coding.
Judge Andrew Peck, U.S. Magistrate Judge for the Southern District of New York, recently wrote an article entitled “Predictive Coding: Reading the Judicial Tea Leaves.” He voices support for this computer-aided approach and provides a basic overview:
“By computer-assisted coding, I mean tools (different vendors use different names) that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer. Unlike manual review, where the review is done by the most junior staff, computer-assisted coding involves a senior partner (or team) who review and code a ‘seed set’ of documents. The computer identifies properties of those documents that it uses to code other documents. As the senior reviewer continues to code more sample documents, the computer predicts the reviewer’s coding.”
Unfortunately, the role of senior attorneys has often been left out of the story since we’re talking about technology assisting human reviewers, not replacing them outright. Noted electronic discovery attorney Ralph Losey claims that many have “fallen for what economists call the Luddite fallacy, the erroneous assumption that new technologies increasing productivity necessarily reduce employment.” He goes on to say:
“The new technologies forces the so-called ‘expensive lawyers’ to grow, to become more tech savvy, but it does not put them out of business… In fact, the only impact it will have on the legal jobs market is on attorneys in the lowest paid spectrum, the so-called contract lawyers.”
In the past, armies of contract reviewers could be massed to slog through millions of documents, all at outlandish costs and unknown quality. But now, the scale of electronic information is growing so quickly that manual review isn’t practical or economical, since e-discovery collections routinely number in the billions of items, versus millions in years past. Even if human review was able to live up to the mythical gold standard, there’d be a call from inside counsel to manage the escalating costs. The fact that technology assisted review is both more accurate and less expensive sets the stage for a new gold standard, which will surely gain momentum in the upcoming year—particularly as the underlying technologies and their use continues to mature.
