Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning

TL;DR

This paper introduces a lightweight Bottleneck Adapter for multimodal models, enabling efficient joint optimization of vision and language components, resulting in superior performance on vision-language tasks.

Contribution

It proposes a novel Bottleneck Adapter and Multimodal Model Tuning approach for end-to-end optimization of vision-language models with fewer parameters.

Findings

Achieved 90.12% accuracy, surpassing human performance and LaVIN-7B.

Demonstrated effective joint optimization with lightweight adapters.

Outperformed existing models in vision-language tasks.

Abstract

The integration of large language models (LLMs) with vision-language (VL) tasks has been a transformative development in the realm of artificial intelligence, highlighting the potential of LLMs as a versatile general-purpose chatbot. However, the current trend in this evolution focuses on the integration of vision and language to create models that can operate in more diverse and real-world contexts. We present a novel approach, termed Bottleneck Adapter, specifically crafted for enhancing the multimodal functionalities of these complex models, enabling joint optimization of the entire multimodal LLM framework through a process known as Multimodal Model Tuning (MMT). Our approach utilizes lightweight adapters to connect the image encoder and LLM without the need for large, complex neural networks. Unlike the conventional modular training schemes, our approach adopts an end-to-end optimization regime, which, when combined with the adapters, facilitates the joint optimization using a significantly smaller parameter set. Our method exhibits robust performance with 90.12\% accuracy, outperforming both human-level performance (88.4\%) and LaVIN-7B (89.41\%).