A recent slew of articles have stoked the age old “man versus machine” debate, often suggesting that man is on the losing end of yet another automation battle. This Homeric contest has recently been waged publically by Watson (IBM’s artificial intelligence computer), which competed on the quiz show Jeopardy and beat two human challengers—the biggest all-time money winner and the record-holder for the longest win streak. This battle has now transitioned to a larger stage, the U.S. economy, where some have gone so far as to suggest that computers are already winning a much more important skirmish.
In an article entitled “Technology Is Eliminating Higher-Skill Jobs”, National Public Radio touted another Watson win, over teams from MIT and Harvard in a “Race Against the Machine” contest. In the NPR piece Andrew McAfee, the MIT researcher who helped organize the conference, cited electronic discovery as the prime battlefield upon which machines are winning. "We see already that the work of legal discovery — in other words, sitting around and reading huge volumes of documents at the early stage of a lawsuit ... is being very quickly and very heavily automated. And, by one estimate, it lets one lawyer do the work of 500."
In the 2007 Sedona Conference “Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery”, the influential think tank cautions against believing in the manual review myth: “Even assuming that the profession had the time and resources to continue to conduct manual review of massive sets of electronic data sets (which it does not), the relative efficacy of that approach versus utilizing newly developed automated methods of review remains very much open to debate.”
Sedona puts it politely. But, the truth is that the human review process is just not very good from a precision and recall perspective. In one of the seminal pieces on the topic (“Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review”), authors Maura Grossman and Gordon Cormack note that “manual review is far from perfect — [i]t is well established that human assessors will disagree in a substantial number of cases as to whether a document is relevant, regardless of the information need or the assessors’ expertise and diligence.” Despite the flaws, the authors note that the legal field tends to nevertheless give the human review process the “gold standard” imprimatur for a few nuanced reasons:
Judge Andrew Peck, U.S. Magistrate Judge for the Southern District of New York, recently wrote an article entitled “Predictive Coding: Reading the Judicial Tea Leaves.” He voices support for this computer-aided approach and provides a basic overview:
“By computer-assisted coding, I mean tools (different vendors use different names) that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer. Unlike manual review, where the review is done by the most junior staff, computer-assisted coding involves a senior partner (or team) who review and code a ‘seed set’ of documents. The computer identifies properties of those documents that it uses to code other documents. As the senior reviewer continues to code more sample documents, the computer predicts the reviewer’s coding.”