FindIt: Generalized Localization with Natural Language Queries

TL;DR

FindIt is a unified, end-to-end framework that effectively addresses various visual grounding and localization tasks using a multi-scale fusion module and a standard object detector, outperforming state-of-the-art methods.

Contribution

The paper introduces a versatile framework that unifies multiple localization tasks with a single model, eliminating the need for task-specific components or pre-computed detections.

Findings

Outperforms state-of-the-art on referring expression and text-based localization.

Shows competitive performance on object detection.

Generalizes better to out-of-distribution data and new categories.

Abstract

We propose FindIt, a simple and versatile framework that unifies a variety of visual grounding and localization tasks including referring expression comprehension, text-based localization, and object detection. Key to our architecture is an efficient multi-scale fusion module that unifies the disparate localization requirements across the tasks. In addition, we discover that a standard object detector is surprisingly effective in unifying these tasks without a need for task-specific design, losses, or pre-computed detections. Our end-to-end trainable framework responds flexibly and accurately to a wide range of referring expression, localization or detection queries for zero, one, or multiple objects. Jointly trained on these tasks, FindIt outperforms the state of the art on both referring expression and text-based localization, and shows competitive performance on object detection. Finally, FindIt generalizes better to out-of-distribution data and novel categories compared to strong single-task baselines. All of these are accomplished by a single, unified and efficient model. The code will be released.