Implementation Notes for the Soft Cosine Measure

TL;DR

This paper improves the computational efficiency of the Soft Cosine Measure, enabling its practical deployment in real-world information retrieval systems by providing tighter complexity bounds and implementation strategies.

Contribution

It proves a lower bound of O(n^3) for the orthonormalization algorithm and presents an efficient algorithm with near-constant worst-case complexity under realistic conditions.

Findings

Tighter lower bound of O(n^3) for the orthonormalization algorithm

An efficient similarity computation algorithm with O(1) worst-case complexity under realistic conditions

Implementation strategies for vector databases and search engines

Abstract

The standard bag-of-words vector space model (VSM) is efficient, and ubiquitous in information retrieval, but it underestimates the similarity of documents with the same meaning, but different terminology. To overcome this limitation, Sidorov et al. proposed the Soft Cosine Measure (SCM) that incorporates term similarity relations. Charlet and Damnati showed that the SCM is highly effective in question answering (QA) systems. However, the orthonormalization algorithm proposed by Sidorov et al. has an impractical time complexity of $\mathcal O(n^4)$, where n is the size of the vocabulary. In this paper, we prove a tighter lower worst-case time complexity bound of $\mathcal O(n^3)$. We also present an algorithm for computing the similarity between documents and we show that its worst-case time complexity is $\mathcal O(1)$ given realistic conditions. Lastly, we describe implementation in general-purpose vector databases such as Annoy, and Faiss and in the inverted indices of text search engines such as Apache Lucene, and ElasticSearch. Our results enable the deployment of the SCM in real-world information retrieval systems.