Steering LLMs for Culturally Localized Generation

TL;DR

This paper introduces Cultural Embeddings (CuE), a mechanistic interpretability approach to analyze and steer large language models towards culturally faithful responses, revealing and manipulating cultural biases more transparently.

Contribution

It presents a novel method using sparse autoencoders to identify interpretable cultural features in LLMs and develop white-box steering techniques for cultural localization.

Findings

CuE-based steering improves cultural faithfulness in responses.

CuE captures long-tail cultural concepts better than prompting alone.

Combining CuE with prompt-augmentation yields the best localization results.

Abstract

LLMs are deployed globally, yet produce responses biased towards cultures with abundant training data. Existing cultural localization approaches such as prompting or post-training alignment are black-box, hard to control, and do not reveal whether failures reflect missing knowledge or poor elicitation. In this paper, we address these gaps using mechanistic interpretability to uncover and manipulate cultural representations in LLMs. Leveraging sparse autoencoders, we identify interpretable features that encode culturally salient information and aggregate them into Cultural Embeddings (CuE). We use CuE both to analyze implicit cultural biases under underspecified prompts and to construct white-box steering interventions. Across multiple models, we show that CuE-based steering increases cultural faithfulness and elicits significantly rarer, long-tail cultural concepts than prompting alone. Notably, CuE-based steering is complementary to black-box localization methods, offering gains when applied on top of prompt-augmented inputs. This also suggests that models do benefit from better elicitation strategies, and don't necessarily lack long-tail knowledge representation, though this varies across cultures. Our results provide both diagnostic insight into cultural representations in LLMs and a controllable method to steer towards desired cultures.