ATATA: One Algorithm to Align Them All

TL;DR

This paper introduces ATATA, a multi-modal joint inference algorithm using Rectified Flow models that achieves faster, high-quality, structurally aligned sample generation across image, video, and 3D domains.

Contribution

It presents a novel joint transport approach that improves speed and quality in multi-modal generation, outperforming existing methods like Score Distillation Sampling.

Findings

Achieves high structural alignment in sample pairs.

Improves state-of-the-art in image and video generation.

Provides comparable 3D generation quality with significantly faster inference.

Abstract

We suggest a new multi-modal algorithm for joint inference of paired structurally aligned samples with Rectified Flow models. While some existing methods propose a codependent generation process, they do not view the problem of joint generation from a structural alignment perspective. Recent work uses Score Distillation Sampling to generate aligned 3D models, but SDS is known to be time-consuming, prone to mode collapse, and often provides cartoonish results. By contrast, our suggested approach relies on the joint transport of a segment in the sample space, yielding faster computation at inference time. Our approach can be built on top of an arbitrary Rectified Flow model operating on the structured latent space. We show the applicability of our method to the domains of image, video, and 3D shape generation using state-of-the-art baselines and evaluate it against both editing-based and joint inference-based competing approaches. We demonstrate a high degree of structural alignment for the sample pairs obtained with our method and a high visual quality of the samples. Our method improves the state-of-the-art for image and video generation pipelines. For 3D generation, it is able to show comparable quality while working orders of magnitude faster.