Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer

TL;DR

This paper introduces a novel inference-time stochastic refinement method for GRU-Normalizing Flows, enhancing diversity and realism in real-time video motion transfer predictions without retraining.

Contribution

It proposes integrating MCMC steps during inference in GRU-NF models, inspired by Stochastic Normalizing Flows, to better explore output space and improve diversity.

Findings

Outperforms GRU-NF in generating diverse, realistic trajectories

Maintains accuracy over longer prediction horizons

Enhances multimodal behavior capture during inference

Abstract

Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision making under uncertainty. To improve the diversity of such sequential forecasts we propose a novel inference-time refinement technique that combines Gated Recurrent Unit-Normalizing Flows (GRU-NF) with stochastic sampling methods. While GRU-NF can capture multimodal distributions through its integration of normalizing flows within a temporal forecasting framework, its deterministic transformation structure can limit expressivity. To address this, inspired by Stochastic Normalizing Flows (SNF), we introduce Markov Chain Monte Carlo (MCMC) steps during GRU-NF inference, enabling the model to explore a richer output space and better approximate the true data distribution without retraining. We validate our approach in a keypoint-based video motion transfer pipeline, where capturing temporally coherent and perceptually diverse future trajectories is essential for realistic samples and low bandwidth communication. Experiments show that our inference framework, Gated Recurrent Unit- Stochastic Normalizing Flows (GRU-SNF) outperforms GRU-NF in generating diverse outputs without sacrificing accuracy, even under longer prediction horizons. By injecting stochasticity during inference, our approach captures multimodal behavior more effectively. These results highlight the potential of integrating stochastic dynamics with flow-based sequence models for generative time series forecasting. The code is available at: https://github.com/Tasmiah1408028/Inference-Time-Stochastic-Refinement-Of-GRU-NF-For-Real-Time-Video-Motion-Transfer