LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance

TL;DR

LIRA enhances large multi-modal models' segmentation and comprehension by integrating semantic features and local supervision, significantly reducing errors and hallucinations in visual understanding.

Contribution

The paper introduces LIRA, a novel framework combining semantic-enhanced feature extraction and local visual coupling to improve segmentation accuracy and reduce hallucinations in multi-modal models.

Findings

LIRA achieves state-of-the-art segmentation performance.

LIRA significantly reduces hallucinated comprehension errors.

The Attributes Evaluation dataset quantifies semantic inference ability.

Abstract

While large multi-modal models (LMMs) demonstrate promising capabilities in segmentation and comprehension, they still struggle with two limitations: inaccurate segmentation and hallucinated comprehension. These challenges stem primarily from constraints in weak visual comprehension and a lack of fine-grained perception. To alleviate these limitations, we propose LIRA, a framework that capitalizes on the complementary relationship between visual comprehension and segmentation via two key components: (1) Semantic-Enhanced Feature Extractor (SEFE) improves object attribute inference by fusing semantic and pixel-level features, leading to more accurate segmentation; (2) Interleaved Local Visual Coupling (ILVC) autoregressively generates local descriptions after extracting local features based on segmentation masks, offering fine-grained supervision to mitigate hallucinations. Furthermore, we find that the precision of object segmentation is positively correlated with the latent related semantics of the <seg> token. To quantify this relationship and the model's potential semantic inferring ability, we introduce the Attributes Evaluation (AttrEval) dataset. Our experiments show that LIRA achieves state-of-the-art performance in both segmentation and comprehension tasks. Code will be available at https://github.com/echo840/LIRA.