Safer in Translation? Presupposition Robustness in Indic Languages

TL;DR

This paper introduces Cancer-Myth-Indic, a multilingual benchmark in five Indic languages to evaluate LLMs' handling of presupposition robustness in healthcare-related queries, addressing a significant gap in multilingual medical AI evaluation.

Contribution

It creates and validates a new multilingual benchmark for assessing LLMs' ability to handle presuppositions in healthcare contexts across Indic languages.

Findings

LLMs show varied robustness to presuppositions across languages.

Translation style guides help preserve presuppositional nuances.

Benchmark reveals gaps in current LLM capabilities in multilingual healthcare scenarios.

Abstract

Increasingly, more and more people are turning to large language models (LLMs) for healthcare advice and consultation, making it important to gauge the efficacy and accuracy of the responses of LLMs to such queries. While there are pre-existing medical benchmarks literature which seeks to accomplish this very task, these benchmarks are almost universally in English, which has led to a notable gap in existing literature pertaining to multilingual LLM evaluation. Within this work, we seek to aid in addressing this gap with Cancer-Myth-Indic, an Indic language benchmark built by translating a 500-item subset of Cancer-Myth, sampled evenly across its original categories, into five under-served but widely used languages from the subcontinent (500 per language; 2,500 translated items total). Native-speaker translators followed a style guide for preserving implicit presuppositions in translation; items feature false presuppositions relating to cancer. We evaluate several popular LLMs under this presupposition stress.