FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation

TL;DR

FOCUS introduces a unified vision-language model that integrates segmentation-aware perception with controllable object-centric image editing, enabling accurate understanding and high-quality, guided visual synthesis within an end-to-end framework.

Contribution

It presents FOCUS, a novel end-to-end LVLM that combines segmentation and generation modules, bridging perception and editing for improved multimodal understanding and controllable image synthesis.

Findings

Achieves state-of-the-art referring segmentation accuracy.

Demonstrates superior controllable image editing quality.

Effectively unifies perception and generation tasks.

Abstract

Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling, enabling both accurate content understanding and flexible editing. However, current approaches treat "what to see" and "how to edit" separately: they either perform isolated object segmentation or utilize segmentation masks merely as conditional prompts for local edit generation tasks, often relying on multiple disjointed models. To bridge these gaps, we introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework. FOCUS employs a dual-branch visual encoder to simultaneously capture global semantic context and fine-grained spatial details. In addition, we leverage a MoVQGAN-based visual tokenizer to produce discrete visual tokens that enhance generation quality. To enable accurate and controllable image editing, we propose a progressive multi-stage training pipeline, where segmentation masks are jointly optimized and used as spatial condition prompts to guide the diffusion decoder. This strategy aligns visual encoding, segmentation, and generation modules, effectively bridging segmentation-aware perception with fine-grained visual synthesis. Extensive experiments across three core tasks, including multimodal understanding, referring segmentation accuracy, and controllable image generation, demonstrate that FOCUS achieves strong performance by jointly optimizing visual perception and generative capabilities.