Finding Inverse Document Frequency Information in BERT

TL;DR

This paper investigates whether BERT-based neural ranking models implicitly encode Inverse Document Frequency (IDF) information, traditionally crucial in lexical retrieval, by analyzing embeddings and attention weights across multiple models and datasets.

Contribution

The study demonstrates that BERT-based neural ranking models inherently contain IDF information, bridging the gap between traditional lexical methods and neural models.

Findings

BERT models encode IDF information in embeddings and attention weights.

IDF information is detectable across different BERT-based models.

Neural models retain traditional lexical features despite semantic focus.

Abstract

For many decades, BM25 and its variants have been the dominant document retrieval approach, where their two underlying features are Term Frequency (TF) and Inverse Document Frequency (IDF). The traditional approach, however, is being rapidly replaced by Neural Ranking Models (NRMs) that can exploit semantic features. In this work, we consider BERT-based NRMs and study if IDF information is present in the NRMs. This simple question is interesting because IDF has been indispensable for the traditional lexical matching, but global features like IDF are not explicitly learned by neural language models including BERT. We adopt linear probing as the main analysis tool because typical BERT based NRMs utilize linear or inner-product based score aggregators. We analyze input embeddings, representations of all BERT layers, and the self-attention weights of CLS. By studying MS-MARCO dataset with three BERT-based models, we show that all of them contain information that is strongly dependent on IDF.