YORO -- Lightweight End to End Visual Grounding

TL;DR

YORO introduces a fast, single-stage transformer-based model for visual grounding that balances speed and accuracy, outperforming existing methods and enabling real-time applications.

Contribution

YORO is the first single-stage, transformer-only architecture for visual grounding that achieves state-of-the-art speed and accuracy trade-offs.

Findings

Supports real-time inference.

Outperforms all single-stage methods in accuracy.

Achieves the best speed/accuracy trade-off.

Abstract

We present YORO - a multi-modal transformer encoder-only architecture for the Visual Grounding (VG) task. This task involves localizing, in an image, an object referred via natural language. Unlike the recent trend in the literature of using multi-stage approaches that sacrifice speed for accuracy, YORO seeks a better trade-off between speed an accuracy by embracing a single-stage design, without CNN backbone. YORO consumes natural language queries, image patches, and learnable detection tokens and predicts coordinates of the referred object, using a single transformer encoder. To assist the alignment between text and visual objects, a novel patch-text alignment loss is proposed. Extensive experiments are conducted on 5 different datasets with ablations on architecture design choices. YORO is shown to support real-time inference and outperform all approaches in this class (single-stage methods) by large margins. It is also the fastest VG model and achieves the best speed/accuracy trade-off in the literature.