Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference

TL;DR

This paper introduces SDR, a self-supervised model capable of ranking document similarity regardless of document length, outperforming existing methods especially on very long documents, with publicly available datasets and code.

Contribution

The paper presents SDR, a novel self-supervised approach for document similarity ranking that handles arbitrarily long documents and surpasses existing methods in performance.

Findings

SDR significantly outperforms alternatives across all metrics.

SDR effectively applies to documents exceeding 4,096 tokens.

Published human-annotated test sets facilitate future research.

Abstract

We present a novel model for the problem of ranking a collection of documents according to their semantic similarity to a source (query) document. While the problem of document-to-document similarity ranking has been studied, most modern methods are limited to relatively short documents or rely on the existence of "ground-truth" similarity labels. Yet, in most common real-world cases, similarity ranking is an unsupervised problem as similarity labels are unavailable. Moreover, an ideal model should not be restricted by documents' length. Hence, we introduce SDR, a self-supervised method for document similarity that can be applied to documents of arbitrary length. Importantly, SDR can be effectively applied to extremely long documents, exceeding the 4,096 maximal token limits of Longformer. Extensive evaluations on large document datasets show that SDR significantly outperforms its alternatives across all metrics. To accelerate future research on unlabeled long document similarity ranking, and as an additional contribution to the community, we herein publish two human-annotated test sets of long documents similarity evaluation. The SDR code and datasets are publicly available.