On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures

TL;DR

This paper investigates how large language models handle different measurement systems across cultures, revealing they often default to dominant systems and require more computation to accurately reason about underrepresented ones, raising accessibility concerns.

Contribution

The study provides a comprehensive analysis of LLMs' performance across diverse measurement systems and highlights the increased computational cost when reasoning about underrepresented cultures.

Findings

LLMs default to the most common measurement system in training data.

Performance varies significantly across different measurement systems.

Reasoning methods like chain-of-thought increase test-time compute, especially for underrepresented systems.

Abstract

Measurement systems (e.g., currencies) differ across cultures, but the conversions between them are well defined so that humans can state facts using any measurement system of their choice. Being available to users from diverse cultural backgrounds, large language models (LLMs) should also be able to provide accurate information irrespective of the measurement system at hand. Using newly compiled datasets we test if this is the case for seven open-source LLMs, addressing three key research questions: (RQ1) What is the default system used by LLMs for each type of measurement? (RQ2) Do LLMs' answers and their accuracy vary across different measurement systems? (RQ3) Can LLMs mitigate potential challenges w.r.t. underrepresented systems via reasoning? Our findings show that LLMs default to the measurement system predominantly used in the data. Additionally, we observe considerable instability and variance in performance across different measurement systems. While this instability can in part be mitigated by employing reasoning methods such as chain-of-thought (CoT), this implies longer responses and thereby significantly increases test-time compute (and inference costs), marginalizing users from cultural backgrounds that use underrepresented measurement systems.