VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution

TL;DR

This paper introduces VD-PCR, a framework that enhances visual dialog understanding by jointly resolving pronouns and pruning irrelevant dialog history, leading to state-of-the-art results on the VisDial dataset.

Contribution

The paper presents a novel joint training approach for pronoun coreference resolution and visual dialog tasks, along with explicit dialog history pruning, to improve understanding.

Findings

Achieved state-of-the-art results on VisDial dataset.

Joint training of coreference resolution and dialog tasks improves performance.

Explicit pruning of irrelevant dialog history enhances model focus and accuracy.

Abstract

The visual dialog task requires an AI agent to interact with humans in multi-round dialogs based on a visual environment. As a common linguistic phenomenon, pronouns are often used in dialogs to improve the communication efficiency. As a result, resolving pronouns (i.e., grounding pronouns to the noun phrases they refer to) is an essential step towards understanding dialogs. In this paper, we propose VD-PCR, a novel framework to improve Visual Dialog understanding with Pronoun Coreference Resolution in both implicit and explicit ways. First, to implicitly help models understand pronouns, we design novel methods to perform the joint training of the pronoun coreference resolution and visual dialog tasks. Second, after observing that the coreference relationship of pronouns and their referents indicates the relevance between dialog rounds, we propose to explicitly prune the irrelevant history rounds in visual dialog models' input. With pruned input, the models can focus on relevant dialog history and ignore the distraction in the irrelevant one. With the proposed implicit and explicit methods, VD-PCR achieves state-of-the-art experimental results on the VisDial dataset.