LILAC: Language-Conditioned Object-Centric Optical Flow for Open-Loop Trajectory Generation

TL;DR

LILAC introduces a flow-based model that generates object-centric optical flow from images and language instructions, enabling open-loop robotic manipulation with minimal embodiment-specific data and outperforming existing methods.

Contribution

The paper presents LILAC, a novel flow-based vision-language model that aligns instructions with object trajectories for robotic manipulation, incorporating semantic and prompt-based alignment techniques.

Findings

Outperformed existing methods in flow quality benchmarks.

Achieved higher task success rates in physical manipulation experiments.

Effective language-conditioned trajectory generation from minimal data.

Abstract

We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at https://lilac-75srg.kinsta.page/.