InTraGen: Trajectory-controlled Video Generation for Object Interactions

TL;DR

InTraGen is a new pipeline that enhances trajectory-controlled video generation for realistic object interactions, introducing datasets and a metric to evaluate interaction quality.

Contribution

The paper presents InTraGen, a novel method with datasets and a metric for improved trajectory-based object interaction video generation.

Findings

Improved visual fidelity in generated videos.

Introduction of four new datasets for evaluation.

Enhanced quantitative performance in object interaction scenarios.

Abstract

Advances in video generation have significantly improved the realism and quality of created scenes. This has fueled interest in developing intuitive tools that let users leverage video generation as world simulators. Text-to-video (T2V) generation is one such approach, enabling video creation from text descriptions only. Yet, due to the inherent ambiguity in texts and the limited temporal information offered by text prompts, researchers have explored additional control signals like trajectory-guided systems, for more accurate T2V generation. Nonetheless, methods to evaluate whether T2V models can generate realistic interactions between multiple objects are lacking. We introduce InTraGen, a pipeline for improved trajectory-based generation of object interaction scenarios. We propose 4 new datasets and a novel trajectory quality metric to evaluate the performance of the proposed InTraGen. To achieve object interaction, we introduce a multi-modal interaction encoding pipeline with an object ID injection mechanism that enriches object-environment interactions. Our results demonstrate improvements in both visual fidelity and quantitative performance. Code and datasets are available at https://github.com/insait-institute/InTraGen