RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognition

TL;DR

RelTransformer introduces a message-passing attention mechanism with a learnable memory to effectively recognize long-tail visual relationships in images, significantly improving performance on large-scale VRR benchmarks.

Contribution

It proposes a novel attention-based scene graph model with a learnable memory to address long-tail distribution challenges in visual relationship recognition.

Findings

Outperforms state-of-the-art on VG8K-LT with +2.0% accuracy

Achieves +26.0% accuracy on GQA-LT

Shows strong results on VG200 relation detection

Abstract

The visual relationship recognition (VRR) task aims at understanding the pairwise visual relationships between interacting objects in an image. These relationships typically have a long-tail distribution due to their compositional nature. This problem gets more severe when the vocabulary becomes large, rendering this task very challenging. This paper shows that modeling an effective message-passing flow through an attention mechanism can be critical to tackling the compositionality and long-tail challenges in VRR. The method, called RelTransformer, represents each image as a fully-connected scene graph and restructures the whole scene into the relation-triplet and global-scene contexts. It directly passes the message from each element in the relation-triplet and global-scene contexts to the target relation via self-attention. We also design a learnable memory to augment the long-tail relation representation learning. Through extensive experiments, we find that our model generalizes well on many VRR benchmarks. Our model outperforms the best-performing models on two large-scale long-tail VRR benchmarks, VG8K-LT (+2.0% overall acc) and GQA-LT (+26.0% overall acc), both having a highly skewed distribution towards the tail. It also achieves strong results on the VG200 relation detection task. Our code is available at https://github.com/Vision-CAIR/RelTransformer.