Latent Action Control for Reasoning-Guided Unified Image Generation

TL;DR

This paper introduces Latent Action Control (LAC), a novel method that makes reasoning actionable within unified image generation models by learning latent action trajectories to improve control and fidelity.

Contribution

LAC is a new approach that encodes reasoning as hidden continuous actions, enabling better control over image generation based on reasoning cues.

Findings

LAC improves compositional and knowledge-grounded generation across multiple benchmarks.

Significant gains in spatial relations, attribute binding, and world-knowledge prompts.

Latent interventions show the learned action trajectory influences the generator.

Abstract

Unified multimodal models can encode visual understanding and image generation within a shared backbone, yet understanding does not automatically translate into control: models may infer objects, relations, or knowledge cues but fail to instantiate them in the generated image. We propose Latent Action Control (LAC), which makes reasoning actionable by representing it as hidden continuous actions inside a unified generator. Given a prompt, LAC rolls out a role-structured latent trajectory for planning, internal visual drafting, diagnosis, and refinement, and injects these actions into the hidden stream that conditions flow-based generation, without producing reasoning tokens or intermediate images. Since such action trajectories are unobserved, LAC learns them through prior-guided variational latent action alignment from training-only rendered semantic priors, draft image features, and supervised halting signals, followed by Latent-Flow GRPO to align the latent-to-image rollout with terminal visual feedback. This provides a control path from inferred relations, bindings, and knowledge cues to the generation process. Instantiated on BAGEL-7B-MoT, LAC consistently improves compositional and knowledge-grounded generation across GenEval, WISE, and T2I-CompBench, with the largest gains on spatial relations, attribute binding, and world-knowledge-sensitive prompts. Ablations and latent interventions show that the learned action trajectory is consumed by the generator, suggesting that unified generation benefits when understanding is not only encoded, but made actionable during generation.