Hamiltonian latent operators for content and motion disentanglement in image sequences

TL;DR

HALO is a deep generative model that uses Hamiltonian latent operators to effectively disentangle content and motion in image sequences, enabling reversible, continuous, and controllable sequence generation.

Contribution

The paper introduces HALO, a novel Hamiltonian-based model that ensures reversible and volume-preserving dynamics for disentangling content and motion in sequences.

Findings

Effective disentanglement of content and motion.

Long-term sequence generation with controlled motion.

Successful sequence swapping and rotation tasks.

Abstract

We introduce \textit{HALO} -- a deep generative model utilising HAmiltonian Latent Operators to reliably disentangle content and motion information in image sequences. The \textit{content} represents summary statistics of a sequence, and \textit{motion} is a dynamic process that determines how information is expressed in any part of the sequence. By modelling the dynamics as a Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-preserving structure in phase space means paths are continuous and are not divergent in the latent space. Consequently, the nearness of sequence frames is realised by the nearness of their coordinates in the phase space, which proves valuable for disentanglement and long-term sequence generation. The sequence space is generally comprised of different types of dynamical motions. To ensure long-term separability and allow controlled generation, we associate every motion with a unique Hamiltonian that acts in its respective subspace. We demonstrate the utility of \textit{HALO} by swapping the motion of a pair of sequences, controlled generation, and image rotations.