Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues

TL;DR

This paper introduces a comprehensive framework for phrase localization and visual relationship detection in images, leveraging diverse linguistic and visual cues for improved accuracy and joint inference.

Contribution

It models multiple visual and linguistic cues, learns their combination weights automatically, and performs joint inference for enhanced phrase grounding and relationship detection.

Findings

Achieves state-of-the-art results on Flickr30k Entities dataset

Outperforms previous methods on Stanford VRD dataset

Demonstrates effective integration of multiple cues for image-language tasks

Abstract

This paper presents a framework for localization or grounding of phrases in images using a large collection of linguistic and visual cues. We model the appearance, size, and position of entity bounding boxes, adjectives that contain attribute information, and spatial relationships between pairs of entities connected by verbs or prepositions. Special attention is given to relationships between people and clothing or body part mentions, as they are useful for distinguishing individuals. We automatically learn weights for combining these cues and at test time, perform joint inference over all phrases in a caption. The resulting system produces state of the art performance on phrase localization on the Flickr30k Entities dataset and visual relationship detection on the Stanford VRD dataset.