A Culturally-Rich Romanian NLP Dataset from "Who Wants to Be a Millionaire?" Videos

TL;DR

This paper presents a new multilingual, culturally-rich Romanian NLP dataset from game show videos, revealing how cultural context affects LLM performance and aiding development of culturally-aware NLP systems.

Contribution

Introduces a novel Romanian dataset from game show videos, combining OCR, manual verification, and metadata, and benchmarks LLMs highlighting cultural performance gaps.

Findings

LLMs perform better on international questions (80-95%) than Romanian-specific ones (50-75%)

Cultural context significantly impacts LLM accuracy and robustness

Cross-lingual tests reveal challenges in cultural understanding by LLMs

Abstract

Large Language Models (LLMs) demonstrate varying performance across languages and cultural contexts. This study introduces a novel, culturally-rich, multilingual dataset derived from video recordings of the Romanian game show "Who Wants to Be a Millionaire?" (Vrei s\u{a} fii Milionar?). We employed an innovative process combining optical character recognition (OCR), automated text extraction, and manual verification to collect question-answer pairs, enriching them with metadata including question domain (e.g., biology, history), cultural relevance (Romanian-specific vs. international), and difficulty. Benchmarking state-of-the-art LLMs, including Romanian-adapted models, on this dataset revealed significant performance disparities: models consistently achieve higher accuracy (80-95%) on international questions compared to Romanian-specific cultural questions (50-75%). We further investigate these differences through experiments involving machine translation of Romanian questions into English and cross-lingual tests using a comparable dataset in French. Our findings underscore the impact of cultural context and data source on LLM performance and offer practical insights for building robust, culturally-aware multilingual NLP systems, especially in educational domains. The dataset is publicly available at Hugging Face.