Generating Easy-to-Understand Referring Expressions for Target Identifications

TL;DR

This paper proposes a method for generating referring expressions that are not only accurate but also easy for humans to understand and locate quickly, especially for less salient targets, using a new GTA V dataset.

Contribution

The study introduces a novel model that considers human comprehension speed and accuracy, optimizing referring expressions for better understandability and creating a new dataset for evaluation.

Findings

The proposed method improves human comprehension speed and accuracy.

The new dataset from GTA V enables better evaluation of referential expression quality.

Experimental results confirm the effectiveness of the approach.

Abstract

This paper addresses the generation of referring expressions that not only refer to objects correctly but also let humans find them quickly. As a target becomes relatively less salient, identifying referred objects itself becomes more difficult. However, the existing studies regarded all sentences that refer to objects correctly as equally good, ignoring whether they are easily understood by humans. If the target is not salient, humans utilize relationships with the salient contexts around it to help listeners to comprehend it better. To derive this information from human annotations, our model is designed to extract information from the target and from the environment. Moreover, we regard that sentences that are easily understood are those that are comprehended correctly and quickly by humans. We optimized this by using the time required to locate the referred objects by humans and their accuracies. To evaluate our system, we created a new referring expression dataset whose images were acquired from Grand Theft Auto V (GTA V), limiting targets to persons. Experimental results show the effectiveness of our approach. Our code and dataset are available at https://github.com/mikittt/easy-to-understand-REG.