Medal Matters: Probing LLMs' Failure Cases Through Olympic Rankings

TL;DR

This paper investigates how large language models understand and organize historical Olympic medal data, revealing strengths in fact recall but weaknesses in ranking tasks, which reflect their internal knowledge limitations.

Contribution

It introduces a novel evaluation framework using Olympic data to analyze LLMs' internal knowledge structures and highlights their difficulty in ranking tasks compared to fact retrieval.

Findings

LLMs excel at recalling medal counts

LLMs struggle with ranking teams

Reveals limitations in LLMs' internal knowledge organization

Abstract

Large language models (LLMs) have achieved remarkable success in natural language processing tasks, yet their internal knowledge structures remain poorly understood. This study examines these structures through the lens of historical Olympic medal tallies, evaluating LLMs on two tasks: (1) retrieving medal counts for specific teams and (2) identifying rankings of each team. While state-of-the-art LLMs excel in recalling medal counts, they struggle with providing rankings, highlighting a key difference between their knowledge organization and human reasoning. These findings shed light on the limitations of LLMs' internal knowledge integration and suggest directions for improvement. To facilitate further research, we release our code, dataset, and model outputs.