CPopQA: Ranking Cultural Concept Popularity by LLMs

TL;DR

This paper introduces CPopQA, a new few-shot question-answering task to evaluate large language models' ability to rank long-tail cultural concepts by popularity, revealing their potential in understanding geo-cultural trends.

Contribution

The study presents a novel dataset and task for assessing LLMs' statistical ranking of cultural concepts, highlighting their capacity to recognize geo-cultural similarities.

Findings

GPT-3.5 outperforms other models in ranking accuracy.

Large models can effectively rank long-tail cultural concepts.

Models show potential to identify geo-cultural proximity.

Abstract

Prior work has demonstrated large language models' (LLMs) potential to discern statistical tendencies within their pre-training corpora. Despite that, many examinations of LLMs' knowledge capacity focus on knowledge explicitly appearing in the training data or implicitly inferable from similar contexts. How well an LLM captures the corpus-level statistical trends of concepts for reasoning, especially long-tail ones, is still underexplored. In this study, we introduce a novel few-shot question-answering task (CPopQA) that examines LLMs' statistical ranking abilities for long-tail cultural concepts (e.g., holidays), with a specific focus on these concepts' popularity in the United States and the United Kingdom, respectively. We curate a dataset containing 459 holidays across 58 countries, generating a total of 6,000 QA testing pairs. Experiments on four strong LLMs show that large models are capable of ranking long-tail cultural concepts regarding their statistical tendency. Notably, GPT-3.5 displayed superior performance and exhibited its potential to identify geo-cultural proximity across continents.