CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare

TL;DR

CLINIC introduces a comprehensive multilingual benchmark to evaluate healthcare language models on trustworthiness aspects like truthfulness, fairness, safety, robustness, and privacy across diverse languages and healthcare topics.

Contribution

This work presents CLINIC, the first extensive multilingual benchmark for assessing trustworthiness of healthcare language models across five key dimensions and 18 tasks in 15 languages.

Findings

LMS struggle with factual accuracy

Bias exists across demographic and linguistic groups

Models are vulnerable to privacy breaches and adversarial attacks

Abstract

Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their real-world adoption is the lack of reliable evaluation of their trustworthiness, especially in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, posing significant challenges for deploying them in global healthcare contexts where linguistic diversity is key. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks, spanning 15 languages (covering all the major continents), and encompassing a wide array of critical healthcare topics like disease conditions, preventive actions, diagnostic tests, treatments, surgeries, and medications. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and are susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.