Neural Neural Textures Make Sim2Real Consistent

TL;DR

TRITON is an unsupervised, end-to-end sim2real translation method that uses neural textures and differentiable rendering to achieve temporal consistency across frames, including object movements.

Contribution

It introduces neural neural textures and a surface consistency loss to ensure temporally consistent image translation in sim2real tasks, regardless of camera or object movement.

Findings

Achieves temporal consistency in unpaired image translation.

Handles both camera and object movements.

Outperforms previous methods in sim2real tasks.

Abstract

Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and \emph{neural neural textures}. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. Unlike previous algorithms, TRITON is not limited to camera movements -- it can handle the movement of objects as well, making it useful for downstream tasks such as robotic manipulation.