PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines

TL;DR

PAT-VCM introduces a flexible, plug-and-play auxiliary token framework for video coding for machines, enabling multi-task adaptability without retraining separate codecs.

Contribution

It proposes a scalable auxiliary-token approach that decouples video compression from specific downstream tasks, supporting multiple auxiliary information types.

Findings

Supports segmentation, depth estimation, and semantic recognition tasks.

Achieves high recognition performance with minimal bitrate overhead.

Enables task-specific improvements without retraining the entire codec.

Abstract

Existing video coding for machines is often trained for a specific downstream task and model. As a result, the compressed representation becomes tightly coupled to the end task, making it difficult to scale across multiple tasks or adapt to model updates. We propose PAT-VCM, a plug-and-play auxiliary-token framework for video coding for machines. PAT-VCM keeps a shared baseline compressed stream and augments it with lightweight task-aware auxiliary tokens, allowing different downstream tasks to recover the information they need without retraining a separate codec for each task. The framework supports three forms of auxiliary information: visual residual tokens, prompt/control tokens, and semantic tokens. We evaluate PAT-VCM on segmentation, depth estimation, and semantic recognition. A shared detection-oriented auxiliary branch provides a reusable first refinement, task-specific visual branches improve segmentation and depth, prompt tokens provide further segmentation gains at negligible bitrate, and semantic tokens achieve strong recognition performance with extremely low overhead. These results suggest that a shared compressed representation, combined with lightweight task-aware auxiliary tokens, is a practical and scalable alternative to tightly task-coupled VCM design.