Towards Geo-Culturally Grounded LLM Generations

TL;DR

This paper explores how retrieval-augmented generation techniques, especially search grounding, can enhance large language models' cultural knowledge, while also discussing associated risks like stereotypes and limitations of knowledge bases.

Contribution

It provides a comparative analysis of KB and search grounding methods, highlighting the effectiveness and challenges of search grounding in improving cultural awareness in LLMs.

Findings

Search grounding improves propositional cultural knowledge performance.

KB grounding is limited by knowledge base coverage.

Search grounding increases stereotypical judgments.

Abstract

Generative large language models (LLMs) have demonstrated gaps in diverse cultural awareness across the globe. We investigate the effect of retrieval augmented generation and search-grounding techniques on LLMs' ability to display familiarity with various national cultures. Specifically, we compare the performance of standard LLMs, LLMs augmented with retrievals from a bespoke knowledge base (i.e., KB grounding), and LLMs augmented with retrievals from a web search (i.e., search grounding) on multiple cultural awareness benchmarks. We find that search grounding significantly improves the LLM performance on multiple-choice benchmarks that test propositional knowledge (e.g., cultural norms, artifacts, and institutions), while KB grounding's effectiveness is limited by inadequate knowledge base coverage and a suboptimal retriever. However, search grounding also increases the risk of stereotypical judgments by language models and fails to improve evaluators' judgments of cultural familiarity in a human evaluation with adequate statistical power. These results highlight the distinction between propositional cultural knowledge and open-ended cultural fluency when it comes to evaluating LLMs' cultural awareness.