Is there something I'm missing? Topic Modeling in eDiscovery

TL;DR

This study demonstrates that in legal eDiscovery, even with incomplete document retrieval, all relevant topics can still be identified, supporting the idea that search can be both efficient and complete in terms of topics.

Contribution

The paper shows that partial document retrieval in eDiscovery still captures all relevant topics, using topic modeling and machine learning classifiers, challenging the need for 100% recall.

Findings

Less than full document recall still captures all topics.

Naive Bayes and SVM classifiers both find all topics in the hit set.

Topic coverage remains complete despite missing relevant documents.

Abstract

In legal eDiscovery, the parties are required to search through their electronically stored information to find documents that are relevant to a specific case. Negotiations over the scope of these searches are often based on a fear that something will be missed. This paper continues an argument that discovery should be based on identifying the facts of a case. If a search process is less than complete (if it has Recall less than 100%), it may still be complete in presenting all of the relevant available topics. In this study, Latent Dirichlet Allocation was used to identify 100 topics from all of the known relevant documents. The documents were then categorized to about 80% Recall (i.e., 80% of the relevant documents were found by the categorizer, designated the hit set and 20% were missed, designated the missed set). Despite the fact that less than all of the relevant documents were identified by the categorizer, the documents that were identified contained all of the topics derived from the full set of documents. This same pattern held whether the categorizer was a na\"ive Bayes categorizer trained on a random selection of documents or a Support Vector Machine trained with Continuous Active Learning (which focuses evaluation on the most-likely-to-be-relevant documents). No topics were identified in either categorizer's missed set that were not already seen in the hit set. Not only is a computer-assisted search process reasonable (as required by the Federal Rules of Civil Procedure), it is also complete when measured by topics.