Two Kinds of Recall

TL;DR

This paper distinguishes two types of recall—diversity and exhaustiveness—and shows that pattern-based models excel at e-recall while neural models excel at d-recall, suggesting a need for balanced evaluation.

Contribution

It introduces the concept of two kinds of recall, providing experimental evidence that pattern-based and neural models excel at different types, challenging existing assumptions.

Findings

Neural methods outperform pattern-based models in d-recall.

Pattern-based methods outperform neural models in e-recall.

Evaluation should consider both types of recall for a complete assessment.

Abstract

It is an established assumption that pattern-based models are good at precision, while learning based models are better at recall. But is that really the case? I argue that there are two kinds of recall: d-recall, reflecting diversity, and e-recall, reflecting exhaustiveness. I demonstrate through experiments that while neural methods are indeed significantly better at d-recall, it is sometimes the case that pattern-based methods are still substantially better at e-recall. Ideal methods should aim for both kinds, and this ideal should in turn be reflected in our evaluations.