CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning

TL;DR

CAKE is a real-time online action detection framework that distills motion information into RGB models, using a novel dynamic motion adapter and contrastive learning to improve accuracy and efficiency.

Contribution

The paper introduces CAKE, a novel flow-based distillation framework with a dynamic motion adapter and contrastive learning for efficient, accurate online action detection without explicit optical flow computation.

Findings

Achieves higher mAP than state-of-the-art methods.

Operates at over 72 FPS on a single CPU.

Effectively models motion without explicit optical flow.

Abstract

Online Action Detection (OAD) systems face two primary challenges: high computational cost and insufficient modeling of discriminative temporal dynamics against background motion. Adding optical flow could provides strong motion cues but it incurs significant computational overhead. We propose CAKE, a OAD Flow-based distillation framework to transfer motion knowledge into RGB models. We propose Dynamic Motion Adapter (DMA) to suppress static background noise and emphasize pixel changes, effectively approximating optical flow without explicit computation. The framework also integrates a Floating Contrastive Learning strategy to distinguish informative motion dynamics from temporal background. Various experiments conducted on the TVSeries, THUMOS'14, Kinetics-400 datasets show effectiveness of our model. CAKE achieves a standout mAP compared with SOTA while using the same backbone. Our model operates at over 72 FPS on a single CPU, making it highly suitable for resource-constrained systems.