# Referring Relationships

## TL;DR

This paper introduces an iterative attention-based model for localizing entities in images based on their relationships, improving disambiguation and interpretability, and enabling localization of unseen categories.

## Contribution

The paper presents a novel model that uses attention shifts to represent relationships, outperforming existing methods and enabling localization without category labels.

## Key findings

- Outperforms existing methods on CLEVR, VRD, and Visual Genome datasets.
- Produces interpretable predicate shifts as attention transitions.
- Enables localization of entities from unseen categories.

## Abstract

Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets --- CLEVR, VRD and Visual Genome --- but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.