An Annotated Corpus of Reference Resolution for Interpreting Common Grounding

TL;DR

This paper introduces a new annotated corpus of referring expressions to better understand and analyze the process of common grounding in dialogue systems, addressing challenges of ambiguity and partial understanding.

Contribution

It provides a large, reliable, and detailed resource for studying reference resolution as a key component of common grounding in natural language dialogue.

Findings

High annotation reliability and natural disagreement levels

Enables detailed analysis of common grounding strategies

Improves interpretation and performance of dialogue systems

Abstract

Common grounding is the process of creating, repairing and updating mutual understandings, which is a fundamental aspect of natural language conversation. However, interpreting the process of common grounding is a challenging task, especially under continuous and partially-observable context where complex ambiguity, uncertainty, partial understandings and misunderstandings are introduced. Interpretation becomes even more challenging when we deal with dialogue systems which still have limited capability of natural language understanding and generation. To address this problem, we consider reference resolution as the central subtask of common grounding and propose a new resource to study its intermediate process. Based on a simple and general annotation schema, we collected a total of 40,172 referring expressions in 5,191 dialogues curated from an existing corpus, along with multiple judgements of referent interpretations. We show that our annotation is highly reliable, captures the complexity of common grounding through a natural degree of reasonable disagreements, and allows for more detailed and quantitative analyses of common grounding strategies. Finally, we demonstrate the advantages of our annotation for interpreting, analyzing and improving common grounding in baseline dialogue systems.