Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

TL;DR

This study examines how cultural differences affect the understanding of book reviews across cultures, highlighting the prevalence of culture-specific elements and evaluating GPT-4o's ability to identify them, revealing room for improvement.

Contribution

It introduces a dataset of culturally-specific review elements and assesses GPT-4o's effectiveness in detecting these, addressing cross-cultural comprehension gaps.

Findings

83% of reviews contained culture-specific difficult-to-understand elements

GPT-4o's detection performance was mixed, indicating need for improvement

The dataset is publicly available for further research

Abstract

In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83\% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: https://github.com/sougata-ub/reading_between_lines