Context-Aware Integration of Language and Visual References for Natural Language Tracking

TL;DR

This paper introduces a joint multi-modal tracking framework that combines language and visual cues through prompt modulation and unified decoding, improving accuracy and reducing drift in natural language video tracking.

Contribution

The paper presents a novel end-to-end multi-modal tracking method that effectively integrates language and visual information to enhance tracking accuracy and robustness.

Findings

Achieves competitive results on TNL2K, OTB-Lang, LaSOT, and RefCOCOg datasets.

Outperforms existing methods in tracking and grounding tasks.

Demonstrates robustness to language-visual misalignments.

Abstract

Tracking by natural language specification (TNL) aims to consistently localize a target in a video sequence given a linguistic description in the initial frame. Existing methodologies perform language-based and template-based matching for target reasoning separately and merge the matching results from two sources, which suffer from tracking drift when language and visual templates miss-align with the dynamic target state and ambiguity in the later merging stage. To tackle the issues, we propose a joint multi-modal tracking framework with 1) a prompt modulation module to leverage the complementarity between temporal visual templates and language expressions, enabling precise and context-aware appearance and linguistic cues, and 2) a unified target decoding module to integrate the multi-modal reference cues and executes the integrated queries on the search image to predict the target location in an end-to-end manner directly. This design ensures spatio-temporal consistency by leveraging historical visual information and introduces an integrated solution, generating predictions in a single step. Extensive experiments conducted on TNL2K, OTB-Lang, LaSOT, and RefCOCOg validate the efficacy of our proposed approach. The results demonstrate competitive performance against state-of-the-art methods for both tracking and grounding.