Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense

TL;DR

This paper evaluates large language models' ability to understand cultural commonsense, revealing significant biases and performance variations across different cultures and query languages.

Contribution

It provides a comprehensive analysis of LLMs' cultural commonsense understanding, highlighting biases and influencing factors, and offers insights for developing culturally aware models.

Findings

LLMs show performance disparities across cultures

Cultural context affects general commonsense capabilities

Query language impacts LLMs' cultural task performance

Abstract

Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks. Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally aware language models.