On the Space-Time Statistics of Motion Pictures

TL;DR

This paper investigates the statistical properties of motion pictures, revealing how space-time regularities depend on local motion trajectories and can be used to estimate optical flow.

Contribution

It introduces a model of space-time natural image statistics based on temporal bandpass filtering and divisive normalization, linking statistical regularities to motion estimation.

Findings

Displaced frame differences along motion directions show stronger regularities.

Statistical regularities vary with local motion trajectories.

These regularities can be used to estimate optical flow.

Abstract

It is well-known that natural images possess statistical regularities that can be captured by bandpass decomposition and divisive normalization processes that approximate early neural processing in the human visual system. We expand on these studies and present new findings on the properties of space-time natural statistics that are inherent in motion pictures. Our model relies on the concept of temporal bandpass (e.g. lag) filtering in LGN and area V1, which is similar to smoothed frame differencing of video frames. Specifically, we model the statistics of the differences between adjacent or neighboring video frames that have been slightly spatially displaced relative to one another. We find that when these space-time differences are further subjected to locally pooled divisive normalization, statistical regularities (or lack thereof) arise that depend on the local motion trajectory. We find that bandpass and divisively normalized frame-differences that are displaced along the motion direction exhibit stronger statistical regularities than for other displacements. Conversely, the direction-dependent regularities of displaced frame differences can be used to estimate the image motion (optical flow) by finding the space-time displacement paths that best preserve statistical regularity.