Measuring IIA Violations in Similarity Choices with Bayesian Models

TL;DR

This paper introduces Bayesian methods to detect and quantify violations of the independence of irrelevant alternatives (IIA) in similarity choice data, revealing context effects influence choices more than classical models assume.

Contribution

The main contribution is the development of Bayesian posterior predictive check methods for testing IIA violations and assessing population homogeneity in similarity choice data.

Findings

Significant IIA violations detected in curated datasets.

Comparable IIA violations between designed and random choice sets.

Population is homogeneous, indicating context effects drive violations.

Abstract

Similarity choice data occur when humans make choices among alternatives based on their similarity to a target, e.g., in the context of information retrieval and in embedding learning settings. Classical metric-based models of similarity choice assume independence of irrelevant alternatives (IIA), a property that allows for a simpler formulation. While IIA violations have been detected in many discrete choice settings, the similarity choice setting has received scant attention. This is because the target-dependent nature of the choice complicates IIA testing. We propose two statistical methods to test for IIA: a classical goodness-of-fit test and a Bayesian counterpart based on the framework of Posterior Predictive Checks (PPC). This Bayesian approach, our main technical contribution, quantifies the degree of IIA violation beyond its mere significance. We curate two datasets: one with choice sets designed to elicit IIA violations, and another with randomly generated choice sets from the same item universe. Our tests confirmed significant IIA violations on both datasets, and notably, we find a comparable degree of violation between them. Further, we devise a new PPC test for population homogeneity. Results show that the population is indeed homogenous, suggesting that the IIA violations are driven by context effects -- specifically, interactions within the choice sets. These results highlight the need for new similarity choice models that account for such context effects.