WMT24 Test Suite: Gender Resolution in Speaker-Listener Dialogue Roles

TL;DR

This paper evaluates gender resolution challenges in speaker-listener dialogues, highlighting how external stereotypes influence gender agreement, which is crucial for improving dialogue understanding systems.

Contribution

It introduces a test suite for gender resolution in dialogue and analyzes the impact of stereotypes on gender agreement.

Findings

External stereotypes significantly affect gender resolution.

Dialogue context alone is insufficient for accurate gender prediction.

Stereotype influence varies with character and manner descriptions.

Abstract

We assess the difficulty of gender resolution in literary-style dialogue settings and the influence of gender stereotypes. Instances of the test suite contain spoken dialogue interleaved with external meta-context about the characters and the manner of speaking. We find that character and manner stereotypes outside of the dialogue significantly impact the gender agreement of referents within the dialogue.