Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding

TL;DR

This paper introduces a two-stage, discourse-aware method for generating referring expressions in visually grounded dialogue, improving discriminative quality and retrieval accuracy.

Contribution

It presents a novel generate-and-rerank approach that incorporates discourse context to produce more effective referring expressions in visual dialogue.

Findings

Higher text-image retrieval accuracy with reranked REs.

Discourse-aware reranking improves discriminative power.

Human evaluation confirms effectiveness of the approach.

Abstract

We propose an approach to referring expression generation (REG) in visually grounded dialogue that is meant to produce referring expressions (REs) that are both discriminative and discourse-appropriate. Our method constitutes a two-stage process. First, we model REG as a text- and image-conditioned next-token prediction task. REs are autoregressively generated based on their preceding linguistic context and a visual representation of the referent. Second, we propose the use of discourse-aware comprehension guiding as part of a generate-and-rerank strategy through which candidate REs generated with our REG model are reranked based on their discourse-dependent discriminatory power. Results from our human evaluation indicate that our proposed two-stage approach is effective in producing discriminative REs, with higher performance in terms of text-image retrieval accuracy for reranked REs compared to those generated using greedy decoding.