Connector-S: A Survey of Connectors in Multi-modal Large Language Models

TL;DR

This survey comprehensively reviews the design, taxonomy, and future challenges of connectors in multi-modal large language models, aiming to guide future research and development in this critical component.

Contribution

It provides a structured taxonomy of connectors, analyzes their technical progress, and discusses future research challenges in multi-modal large language models.

Findings

Taxonomy categorizes connectors into atomic and holistic types.

Highlights recent advancements and technical contributions.

Identifies key challenges and future research directions.

Abstract

With the rapid advancements in multi-modal large language models (MLLMs), connectors play a pivotal role in bridging diverse modalities and enhancing model performance. However, the design and evolution of connectors have not been comprehensively analyzed, leaving gaps in understanding how these components function and hindering the development of more powerful connectors. In this survey, we systematically review the current progress of connectors in MLLMs and present a structured taxonomy that categorizes connectors into atomic operations (mapping, compression, mixture of experts) and holistic designs (multi-layer, multi-encoder, multi-modal scenarios), highlighting their technical contributions and advancements. Furthermore, we discuss several promising research frontiers and challenges, including high-resolution input, dynamic compression, guide information selection, combination strategy, and interpretability. This survey is intended to serve as a foundational reference and a clear roadmap for researchers, providing valuable insights into the design and optimization of next-generation connectors to enhance the performance and adaptability of MLLMs.