On the Effects of Regional Spelling Conventions in Retrieval Models

TL;DR

This paper investigates how regional spelling differences, like color versus colour, affect neural retrieval models, revealing that models generally generalize well despite spelling biases, but normalization impacts performance variably.

Contribution

It provides a systematic analysis of the impact of regional spelling conventions on neural retrieval models and the effects of spelling normalization on their performance.

Findings

American spelling conventions are more prevalent in datasets.

Models generally generalize well despite spelling biases.

Normalization affects models differently, with lexical models improving and dense retrievers unaffected.

Abstract

One advantage of neural ranking models is that they are meant to generalise well in situations of synonymity i.e. where two words have similar or identical meanings. In this paper, we investigate and quantify how well various ranking models perform in a clear-cut case of synonymity: when words are simply expressed in different surface forms due to regional differences in spelling conventions (e.g., color vs colour). We first explore the prevalence of American and British English spelling conventions in datasets used for the pre-training, training and evaluation of neural retrieval methods, and find that American spelling conventions are far more prevalent. Despite these biases in the training data, we find that retrieval models often generalise well in this case of synonymity. We explore the effect of document spelling normalisation in retrieval and observe that all models are affected by normalising the document's spelling. While they all experience a drop in performance when normalised to a different spelling convention than that of the query, we observe varied behaviour when the document is normalised to share the query spelling convention: lexical models show improvements, dense retrievers remain unaffected, and re-rankers exhibit contradictory behaviour.