PRISM: Video Dataset Condensation with Progressive Refinement and Insertion for Sparse Motion

TL;DR

PRISM introduces a unified, adaptive video condensation method that efficiently captures complex motion by progressively inserting key frames based on gradient misalignments, reducing storage while maintaining performance.

Contribution

It proposes a holistic, coupled spatiotemporal approach with progressive refinement and insertion, improving over static/dynamic disentanglement methods for video dataset condensation.

Findings

Achieves state-of-the-art storage efficiency.

Maintains competitive performance on standard benchmarks.

Effectively captures non-linear motion dynamics.

Abstract

Video dataset condensation aims to reduce the immense computational cost of video processing. However, it faces a fundamental challenge regarding the inseparable interdependence between spatial appearance and temporal dynamics. Prior work follows a static/dynamic disentanglement paradigm where videos are decomposed into static content and auxiliary motion signals. This multi-stage approach often misrepresents the intrinsic coupling of real-world actions. We introduce Progressive Refinement and Insertion for Sparse Motion (PRISM), a holistic approach that treats the video as a unified and fully coupled spatiotemporal structure from the outset. To maximize representational efficiency, PRISM addresses the inherent temporal redundancy of video by avoiding fixed-frame optimization. It begins with minimal temporal anchors and progressively inserts key-frames only where linear interpolation fails to capture non-linear dynamics. These critical moments are identified through gradient misalignments. Such an adaptive process ensures that representational capacity is allocated precisely where needed, minimizing storage requirements while preserving complex motion. Extensive experiments demonstrate that PRISM achieves competitive performance across standard benchmarks while providing state-of-the-art storage efficiency through its sparse and holistically learned representation.