Learning of Visual Relations: The Devil is in the Tails

TL;DR

This paper proposes a simple yet effective training approach for visual relation models that addresses the long-tailed distribution challenge, outperforming complex models in scene graph generation.

Contribution

It introduces a novel decoupled training scheme with a new sampling method, demonstrating that simplicity combined with targeted sampling improves long-tailed visual relation learning.

Findings

DT2-ACBS outperforms complex state-of-the-art methods

Simple models can be highly effective with proper training strategies

Addressing long-tailed distributions is crucial for scene graph tasks

Abstract

Significant effort has been recently devoted to modeling visual relations. This has mostly addressed the design of architectures, typically by adding parameters and increasing model complexity. However, visual relation learning is a long-tailed problem, due to the combinatorial nature of joint reasoning about groups of objects. Increasing model complexity is, in general, ill-suited for long-tailed problems due to their tendency to overfit. In this paper, we explore an alternative hypothesis, denoted the Devil is in the Tails. Under this hypothesis, better performance is achieved by keeping the model simple but improving its ability to cope with long-tailed distributions. To test this hypothesis, we devise a new approach for training visual relationships models, which is inspired by state-of-the-art long-tailed recognition literature. This is based on an iterative decoupled training scheme, denoted Decoupled Training for Devil in the Tails (DT2). DT2 employs a novel sampling approach, Alternating Class-Balanced Sampling (ACBS), to capture the interplay between the long-tailed entity and predicate distributions of visual relations. Results show that, with an extremely simple architecture, DT2-ACBS significantly outperforms much more complex state-of-the-art methods on scene graph generation tasks. This suggests that the development of sophisticated models must be considered in tandem with the long-tailed nature of the problem.