The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

TL;DR

This paper critically evaluates scientific language models' effectiveness in scholarly retrieval, revealing significant limitations in handling short queries and textual neighbors, with retrieval heavily influenced by surface form rather than semantics.

Contribution

It provides an empirical analysis of language models' retrieval performance on short queries and textual neighbors, highlighting their surface form bias and limitations in scientific IR.

Findings

Language models struggle to retrieve relevant documents for short queries.

Textual neighbors generated by small perturbations often do not remain close in embedding space.

Retrieval performance is more affected by surface form than by semantic similarity.

Abstract

Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related, and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.