Position bias in features

TL;DR

This paper investigates position bias in search engine relevance modeling, highlighting the importance of accurate bias estimation and proposing the combined use of biased and unbiased features to improve ranking accuracy.

Contribution

It introduces a method to extend inverse propensity weighting for unbiased relevance estimation and advocates for using both biased and unbiased features simultaneously.

Findings

Unbiased relevance estimates can approximate true relevance in ideal conditions.

High variance in unbiased features increases with position bias degree.

Inaccurate bias estimation can degrade ranking performance.

Abstract

The purpose of modeling document relevance for search engines is to rank better in subsequent searches. Document-specific historical click-through rates can be important features in a dynamic ranking system which updates as we accumulate more sample. This paper describes the properties of several such features, and tests them in controlled experiments. Extending the inverse propensity weighting method to documents creates an unbiased estimate of document relevance. This feature can approximate relevance accurately, leading to near-optimal ranking in ideal circumstances. However, it has high variance that is increasing with respect to the degree of position bias. Furthermore, inaccurate position bias estimation leads to poor performance. Under several scenarios this feature can perform worse than biased click-through rates. This paper underscores the need for accurate position bias estimation, and is unique in suggesting simultaneous use of biased and unbiased position bias features.