MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows

TL;DR

MOOSE is a novel video encoder that efficiently models temporal dynamics by integrating optical flow with spatial embeddings, improving interpretability and achieving state-of-the-art results across various video understanding tasks.

Contribution

Introducing MOOSE, a temporally-centric architecture that leverages pre-trained optical flow and visual encoders for efficient and interpretable video analysis.

Findings

State-of-the-art performance on multiple benchmarks

Reduced computational complexity compared to prior models

Enhanced interpretability of temporal dynamics

Abstract

Many motion-centric video analysis tasks, such as atomic actions, detecting atypical motor behavior in individuals with autism, or analyzing articulatory motion in real-time MRI of human speech, require efficient and interpretable temporal modeling. Capturing temporal dynamics is a central challenge in video analysis, often requiring significant computational resources and fine-grained annotations that are not widely available. This paper presents MOOSE (Motion Flow Over Spatial Space), a novel temporally-centric video encoder explicitly integrating optical flow with spatial embeddings to model temporal information efficiently, inspired by human perception of motion. Unlike prior models, MOOSE takes advantage of rich, widely available pre-trained visual and optical flow encoders instead of training video models from scratch. This significantly reduces computational complexity while enhancing temporal interpretability. Our primary contributions includes (1) proposing a computationally efficient temporally-centric architecture for video understanding (2) demonstrating enhanced interpretability in modeling temporal dynamics; and (3) achieving state-of-the-art performance on diverse benchmarks, including clinical, medical, and standard action recognition datasets, confirming the broad applicability and effectiveness of our approach.