Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing

TL;DR

This paper introduces ColorTrigger, a real-time, online method that reduces color sensing in streaming video by selectively activating color capture only when necessary, maintaining high accuracy with significantly fewer RGB frames.

Contribution

The paper proposes a novel grayscale-guided trigger mechanism for efficient, resource-aware video sensing that operates without training, enabling practical always-on video understanding on edge devices.

Findings

Achieves 91.6% of full-color performance with only 8.1% RGB frames

Color redundancy is substantial in natural videos, allowing significant sensing reduction

ColorTrigger operates in real-time with lightweight computation

Abstract

Always-on sensing is essential for next-generation edge/wearable AI systems, yet continuous high-fidelity RGB video capture remains prohibitively expensive for resource-constrained mobile and edge platforms. We present a new paradigm for efficient streaming video understanding: grayscale-always, color-on-demand. Through preliminary studies, we discover that color is not always necessary. Sparse RGB frames suffice for comparable performance when temporal structure is preserved via continuous grayscale streams. Building on this insight, we propose ColorTrigger, an online training-free trigger that selectively activates color capture based on windowed grayscale affinity analysis. Designed for real-time edge deployment, ColorTrigger uses lightweight quadratic programming to detect chromatic redundancy causally, coupled with credit-budgeted control and dynamic token routing to jointly reduce sensing and inference costs. On streaming video understanding benchmarks, ColorTrigger achieves 91.6% of full-color baseline performance while using only 8.1% RGB frames, demonstrating substantial color redundancy in natural videos and enabling practical always-on video sensing on resource-constrained devices.