Unsupervised Contrast-Consistent Ranking with Language Models

TL;DR

This paper introduces Contrast-Consistent Ranking (CCR), an unsupervised probing method that enforces logical consistency in language model rankings, outperforming or matching prompting techniques across various models and datasets.

Contribution

It proposes CCR, an unsupervised, contrastive probing approach that improves ranking consistency in language models by extending CCS with ranking-specific objectives.

Findings

CCR outperforms prompting in ranking tasks

CCR maintains logical consistency across comparisons

Method is effective across models and datasets

Abstract

Language models contain ranking-based knowledge and are powerful solvers of in-context ranking tasks. For instance, they may have parametric knowledge about the ordering of countries by size or may be able to rank product reviews by sentiment. We compare pairwise, pointwise and listwise prompting techniques to elicit a language model's ranking knowledge. However, we find that even with careful calibration and constrained decoding, prompting-based techniques may not always be self-consistent in the rankings they produce. This motivates us to explore an alternative approach that is inspired by an unsupervised probing method called Contrast-Consistent Search (CCS). The idea is to train a probe guided by a logical constraint: a language model's representation of a statement and its negation must be mapped to contrastive true-false poles consistently across multiple statements. We hypothesize that similar constraints apply to ranking tasks where all items are related via consistent, pairwise or listwise comparisons. To this end, we extend the binary CCS method to Contrast-Consistent Ranking (CCR) by adapting existing ranking methods such as the Max-Margin Loss, Triplet Loss and an Ordinal Regression objective. Across different models and datasets, our results confirm that CCR probing performs better or, at least, on a par with prompting.