DyNCA: Real-time Dynamic Texture Synthesis Using Neural Cellular Automata

TL;DR

DyNCA introduces a real-time, controllable neural cellular automata framework for dynamic texture synthesis, producing more realistic videos at significantly faster speeds and enabling interactive control features.

Contribution

The paper presents DyNCA, a novel neural cellular automata-based approach for real-time, scalable, and controllable dynamic texture synthesis, outperforming previous methods by 2-4 orders of magnitude.

Findings

Achieves real-time synthesis of arbitrarily sized videos

Outperforms state-of-the-art methods by 2-4 orders of magnitude

Provides interactive controls like motion speed and direction

Abstract

Current Dynamic Texture Synthesis (DyTS) models can synthesize realistic videos. However, they require a slow iterative optimization process to synthesize a single fixed-size short video, and they do not offer any post-training control over the synthesis process. We propose Dynamic Neural Cellular Automata (DyNCA), a framework for real-time and controllable dynamic texture synthesis. Our method is built upon the recently introduced NCA models and can synthesize infinitely long and arbitrary-sized realistic video textures in real time. We quantitatively and qualitatively evaluate our model and show that our synthesized videos appear more realistic than the existing results. We improve the SOTA DyTS performance by $2\sim 4$ orders of magnitude. Moreover, our model offers several real-time video controls including motion speed, motion direction, and an editing brush tool. We exhibit our trained models in an online interactive demo that runs on local hardware and is accessible on personal computers and smartphones.