Video Analysis and Generation via a Semantic Progress Function

TL;DR

This paper introduces a Semantic Progress Function to analyze and linearize the semantic evolution in videos, enabling smoother transitions and better control over semantic pacing.

Contribution

It proposes a novel semantic linearization method that reparameterizes video sequences for consistent semantic change, improving analysis and editing capabilities.

Findings

Semantic progress can be modeled as a smooth curve of semantic embeddings.

The linearization procedure produces videos with uniform semantic pacing.

Framework allows comparison and steering of semantic evolution in videos.

Abstract

Transformations produced by image and video generation models often evolve in a highly non-linear manner: long stretches where the content barely changes are followed by sudden, abrupt semantic jumps. To analyze and correct this behavior, we introduce a Semantic Progress Function, a one-dimensional representation that captures how the meaning of a given sequence evolves over time. For each frame, we compute distances between semantic embeddings and fit a smooth curve that reflects the cumulative semantic shift across the sequence. Departures of this curve from a straight line reveal uneven semantic pacing. Building on this insight, we propose a semantic linearization procedure that reparameterizes (or retimes) the sequence so that semantic change unfolds at a constant rate, yielding smoother and more coherent transitions. Beyond linearization, our framework provides a model-agnostic foundation for identifying temporal irregularities, comparing semantic pacing across different generators, and steering both generated and real-world video sequences toward arbitrary target pacing.