Differentiated Relevances Embedding for Group-based Referring Expression Comprehension

TL;DR

This paper introduces a novel group-based referring expression comprehension approach that models cross-modal relevance more accurately by considering multiple attributes and their values, improving comprehension accuracy.

Contribution

The paper proposes a multi-group self-paced relevance learning schema and across-group relevance constraint to better capture cross-modal relevance in REC.

Findings

Outperforms existing methods on three standard benchmarks.

Effectively models multiple attributes and their values for improved accuracy.

Demonstrates the importance of balancing group relevance biases.

Abstract

The key of referring expression comprehension lies in capturing the cross-modal visual-linguistic relevance. Existing works typically model the cross-modal relevance in each image, where the anchor object/expression and their positive expression/object have the same attribute as the negative expression/object, but with different attribute values. These objects/expressions are exclusively utilized to learn the implicit representation of the attribute by a pair of different values, which however impedes the accuracies of the attribute representations, expression/object representations, and their cross-modal relevances since each anchor object/expression usually has multiple attributes while each attribute usually has multiple potential values. To this end, we investigate a novel REC problem named Group-based REC, where each object/expression is simultaneously employed to construct the multiple triplets among the semantically similar images. To tackle the explosion of the negatives and the differentiation of the anchor-negative relevance scores, we propose the multi-group self-paced relevance learning schema to adaptively assign within-group object-expression pairs with different priorities based on their cross-modal relevances. Since the average cross-modal relevance varies a lot across different groups, we further design an across-group relevance constraint to balance the bias of the group priority. Experiments on three standard REC benchmarks demonstrate the effectiveness and superiority of our method.