Learning Time in Static Classifiers

TL;DR

This paper introduces a simple framework that adds temporal reasoning to static classifiers using a novel training paradigm, improving performance in image classification and video anomaly detection without altering model architecture.

Contribution

It proposes the SEQ learning paradigm that structures training data into trajectories, enabling temporal learning with a differentiable loss, without modifying existing classifier architectures.

Findings

Enhances image classification accuracy, especially in fine-grained tasks.

Achieves temporally consistent predictions in video anomaly detection.

Requires only pre-extracted features and no architectural changes.

Abstract

Real-world visual data rarely presents as isolated, static instances. Instead, it often evolves gradually over time through variations in pose, lighting, object state, or scene context. However, conventional classifiers are typically trained under the assumption of temporal independence, limiting their ability to capture such dynamics. We propose a simple yet effective framework that equips standard feedforward classifiers with temporal reasoning, all without modifying model architectures or introducing recurrent modules. At the heart of our approach is a novel Support-Exemplar-Query (SEQ) learning paradigm, which structures training data into temporally coherent trajectories. These trajectories enable the model to learn class-specific temporal prototypes and align prediction sequences via a differentiable soft-DTW loss. A multi-term objective further promotes semantic consistency and temporal smoothness. By interpreting input sequences as evolving feature trajectories, our method introduces a strong temporal inductive bias through loss design alone. This proves highly effective in both static and temporal tasks: it enhances performance on fine-grained and ultra-fine-grained image classification, and delivers precise, temporally consistent predictions in video anomaly detection. Despite its simplicity, our approach bridges static and temporal learning in a modular and data-efficient manner, requiring only a simple classifier on top of pre-extracted features.