Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes

TL;DR

Talk2Move is a reinforcement learning framework that enables precise, natural language-guided object transformations in scenes without requiring paired training data, advancing the capabilities of multimodal scene editing.

Contribution

It introduces a novel RL-based diffusion method with spatial rewards and active learning for text-guided object-level geometric transformations without paired supervision.

Findings

Outperforms existing methods in spatial accuracy

Achieves coherent and semantically faithful transformations

Demonstrates effectiveness on curated benchmarks

Abstract

We introduce Talk2Move, a reinforcement learning (RL) based diffusion framework for text-instructed spatial transformation of objects within scenes. Spatially manipulating objects in a scene through natural language poses a challenge for multimodal generation systems. While existing text-based manipulation methods can adjust appearance or style, they struggle to perform object-level geometric transformations-such as translating, rotating, or resizing objects-due to scarce paired supervision and pixel-level optimization limits. Talk2Move employs Group Relative Policy Optimization (GRPO) to explore geometric actions through diverse rollouts generated from input images and lightweight textual variations, removing the need for costly paired data. A spatial reward guided model aligns geometric transformations with linguistic description, while off-policy step evaluation and active step sampling improve learning efficiency by focusing on informative transformation stages. Furthermore, we design object-centric spatial rewards that evaluate displacement, rotation, and scaling behaviors directly, enabling interpretable and coherent transformations. Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations, outperforming existing text-guided editing approaches in both spatial accuracy and scene coherence.