CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models

TL;DR

CoMaTrack introduces a multi-agent reinforcement learning framework for embodied visual tracking, enhancing robustness and adaptability through competitive scenarios and providing a new benchmark for evaluation.

Contribution

It presents a novel game-theoretic multi-agent training framework and a comprehensive open-source benchmark for language-conditioned embodied visual tracking.

Findings

Achieved state-of-the-art results on standard benchmarks and CoMaTrack-Bench.

A 3B VLM trained with CoMaTrack surpasses previous models on EVT-Bench.

The benchmark enables standardized robustness evaluation under adversarial interactions.

Abstract

Embodied Visual Tracking (EVT), a core dynamic task in embodied intelligence, requires an agent to precisely follow a language-specified target. Yet most existing methods rely on single-agent imitation learning, suffering from costly expert data and limited generalization due to static training environments. Inspired by competition-driven capability evolution, we propose CoMaTrack, a competitive game-theoretic multi-agent reinforcement learning framework that trains agents in a dynamic adversarial setting with competitive subtasks, yielding stronger adaptive planning and interference-resilient strategies. We further introduce CoMaTrack-Bench, the first open-source Habitat-based benchmark protocol and episode set for language-conditioned competitive EVT featuring dynamic dueling, featuring game scenarios between a tracker and adaptive opponents across diverse environments and instructions, enabling standardized robustness evaluation under active adversarial interactions. Experiments show that CoMaTrack achieves state-of-the-art results on both standard benchmarks and CoMaTrack-Bench. Notably, a 3B VLM trained with our framework surpasses previous single-agent imitation learning methods based on 7B models on the challenging EVT-Bench, achieving 92.1% in STT, 74.2% in DT, and 57.5% in AT. The benchmark code will be available at https://github.com/wlqcode/CoMaTrack-Bench.