Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust

TL;DR

This paper emphasizes the importance of context-specific empirical research to understand and improve user trust in machine translation systems, especially in high-stakes scenarios like healthcare.

Contribution

It highlights the need for empirical studies on how users interact with MT in real-world settings to inform trust calibration and system design.

Findings

Clinicians use MT to communicate across language barriers.

Users often have varying levels of trust based on context and system performance.

Empirical research is crucial for understanding practical MT usage and trust issues.

Abstract

Machine Translation (MT) has the potential to help people overcome language barriers and is widely used in high-stakes scenarios, such as in hospitals. However, in order to use MT reliably and safely, users need to understand when to trust MT outputs and how to assess the quality of often imperfect translation results. In this paper, we discuss research directions to support users to calibrate trust in MT systems. We share findings from an empirical study in which we conducted semi-structured interviews with 20 clinicians to understand how they communicate with patients across language barriers, and if and how they use MT systems. Based on our findings, we advocate for empirical research on how MT systems are used in practice as an important first step to addressing the challenges in building appropriate trust between users and MT tools.