SAViR-T: Spatially Attentive Visual Reasoning with Transformers

TL;DR

SAViR-T introduces a transformer-based model that explicitly encodes spatial semantics in visual reasoning tasks, achieving state-of-the-art results on multiple RPM benchmarks and natural image datasets.

Contribution

The paper presents SAViR-T, a novel transformer architecture that models intra- and inter-image dependencies using spatially attentive tokens for improved visual reasoning.

Findings

Sets new state-of-the-art on RPM benchmarks

Outperforms prior models significantly

Effective on both synthetic and natural images

Abstract

We present a novel computational model, "SAViR-T", for the family of visual reasoning problems embodied in the Raven's Progressive Matrices (RPM). Our model considers explicit spatial semantics of visual elements within each image in the puzzle, encoded as spatio-visual tokens, and learns the intra-image as well as the inter-image token dependencies, highly relevant for the visual reasoning task. Token-wise relationship, modeled through a transformer-based SAViR-T architecture, extract group (row or column) driven representations by leveraging the group-rule coherence and use this as the inductive bias to extract the underlying rule representations in the top two row (or column) per token in the RPM. We use this relation representations to locate the correct choice image that completes the last row or column for the RPM. Extensive experiments across both synthetic RPM benchmarks, including RAVEN, I-RAVEN, RAVEN-FAIR, and PGM, and the natural image-based "V-PROM" demonstrate that SAViR-T sets a new state-of-the-art for visual reasoning, exceeding prior models' performance by a considerable margin.