LLMBind: A Unified Modality-Task Integration Framework
Bin Zhu, Munan Ning, Peng Jin, Bin Lin, Jinfa Huang, Qi Song, Junwu Zhang, Zhenyu Tang, Mingjun Pan, Li Yuan

TL;DR
LLMBind is a versatile framework that unifies diverse multimodal tasks using dual-pathway mechanisms and MoE architecture, enabling high performance and extensibility across perception and generation tasks.
Contribution
It introduces LLMBind, a novel, extensible multimodal integration framework with dual-pathway design and MoE routing, addressing task flexibility and modality interference issues.
Findings
Achieves state-of-the-art results on multiple benchmarks.
Demonstrates superior task extensibility and modality disentanglement.
Curates a large multi-turn interactive dataset for visual refinement.
Abstract
Despite recent progress in Multi-Modal Large Language Models (MLLMs), it remains challenging to integrate diverse tasks ranging from pixel-level perception to high-fidelity generation. Existing approaches often suffer from either restricted task extensibility or severe performance degradation due to modality interference. n this paper, we present LLMBind, an extensible framework that unifies multimodal tasks through a dual-pathway mechanism: In-Situ semantic embeddings for localization-sensitive tasks like semantic segmentation and Ex-Situ task-prompts for generation across image, video, and audio modalities. Additionally, we employ a Mixture-of-Experts (MoE) architecture to route task-specific tokens, thereby achieving modality disentanglement and mitigating negative transfer. We also curate a 400k multi-turn interactive dataset focused on iterative visual refinement to enable…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSemantic Web and Ontologies · Advanced Database Systems and Queries · Natural Language Processing Techniques
