Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videos

TL;DR

Planner-Refiner is a novel framework that iteratively refines space-time visual representations guided by language to improve video-language alignment, especially for complex prompts, demonstrated on new and existing benchmarks.

Contribution

It introduces a Planner-Refiner framework that decomposes complex language prompts and refines visual features iteratively for better alignment in videos.

Findings

Outperforms state-of-the-art on Referring Video Object Segmentation.

Achieves superior results on Temporal Grounding with complex prompts.

Introduces the MeViS-X benchmark for long query evaluation.

Abstract

Vision-language alignment in video must address the complexity of language, evolving interacting entities, their action chains, and semantic gaps between language and vision. This work introduces Planner-Refiner, a framework to overcome these challenges. Planner-Refiner bridges the semantic gap by iteratively refining visual elements' space-time representation, guided by language until semantic gaps are minimal. A Planner module schedules language guidance by decomposing complex linguistic prompts into short sentence chains. The Refiner processes each short sentence, a noun-phrase and verb-phrase pair, to direct visual tokens' self-attention across space then time, achieving efficient single-step refinement. A recurrent system chains these steps, maintaining refined visual token representations. The final representation feeds into task-specific heads for alignment generation. We demonstrate Planner-Refiner's effectiveness on two video-language alignment tasks: Referring Video Object Segmentation and Temporal Grounding with varying language complexity. We further introduce a new MeViS-X benchmark to assess models' capability with long queries. Superior performance versus state-of-the-art methods on these benchmarks shows the approach's potential, especially for complex prompts.