An Empirical Study of the Application of Machine Learning and Keyword Terms Methodologies to Privilege-Document Review Projects in Legal Matters

TL;DR

This paper compares keyword search and predictive modeling methods for identifying privileged documents in legal review, highlighting their respective strengths and weaknesses based on a real-world data study.

Contribution

It provides an empirical evaluation of the effectiveness of keyword searching versus predictive modeling in legal privilege document review.

Findings

Predictive modeling outperforms keyword search in identifying privileged documents.

Keyword searches tend to produce more over-inclusive results, increasing review costs.

Predictive models can reduce false positives and improve review efficiency.

Abstract

Protecting privileged communications and data from disclosure is paramount for legal teams. Unrestricted legal advice, such as attorney-client communications or litigation strategy. are vital to the legal process and are exempt from disclosure in litigations or regulatory events. To protect this information from being disclosed, companies and outside counsel must review vast amounts of documents to determine those that contain privileged material. This process is extremely costly and time consuming. As data volumes increase, legal counsel employ methods to reduce the number of documents requiring review while balancing the need to ensure the protection of privileged information. Keyword searching is relied upon as a method to target privileged information and reduce document review populations. Keyword searches are effective at casting a wide net but return over inclusive results -- most of which do not contain privileged information -- and without detailed knowledge of the data, keyword lists cannot be crafted to find all privilege material. Overly-inclusive keyword searching can also be problematic, because even while it drives up costs, it also can cast `too far of a net' and thus produce unreliable results.To overcome these weaknesses of keyword searching, legal teams are using a new method to target privileged information called predictive modeling. Predictive modeling can successfully identify privileged material but little research has been published to confirm its effectiveness when compared to keyword searching. This paper summarizes a study of the effectiveness of keyword searching and predictive modeling when applied to real-world data. With this study, this group of collaborators wanted to examine and understand the benefits and weaknesses of both approaches to legal teams with identifying privilege material in document populations.