Extending Beacon to Hindi: Cultural Adaptation Drives Cross-Lingual Sycophancy

TL;DR

This study investigates how cultural adaptation of prompts influences sycophantic responses in language models across English and Hindi, revealing that cultural context significantly impacts alignment behaviors.

Contribution

The paper extends the Beacon sycophancy diagnostic to Hindi, demonstrating that cultural adaptation increases sycophantic responses more than language translation alone.

Findings

Culturally adapted Hindi prompts increase sycophancy rates by 12-16% across models.

Cultural adaptation accounts for most of the cross-lingual sycophancy gap.

Advice prompts show the largest cross-lingual differences in sycophantic responses.

Abstract

Sycophancy, the tendency of language models to prioritize agreement with user preferences over principled reasoning, has been identified as a persistent alignment failure in English-language evaluations. However, it remains unclear whether such diagnostics generalize across languages and cultural contexts. We extend the Beacon single-turn forced-choice sycophancy diagnostic to Hindi through a controlled three-condition design: English original, Hindi literal translation, and Hindi culturally adapted prompts. We evaluate four open-weight instruction-tuned models on 50 prompts per condition, enabling separation of language encoding effects from cultural adaptation effects. Across all models, sycophancy rates are consistently higher for culturally adapted Hindi prompts than for English, with absolute differences ranging from 12.0 to 16.0 percentage points. A decomposition on Qwen 2.5-Coder-7B shows that cultural adaptation (delta = 14.0%, 95% CI: [4.0%, 26.0%]) accounts for the majority of this gap, while language encoding contributes minimally (delta = 2.0%, 95% CI: [0.0%, 6.0%]). Category-level analysis reveals that advice prompts exhibit the largest cross-lingual differences (20-25 percentage points), achieving statistical significance in two of four models. These findings indicate that alignment behaviors measured in English may not transfer uniformly across languages and that culturally grounded prompt framing plays a substantial role. We release all datasets and evaluation code to support replication and extension.