InfMLLM: A Unified Framework for Visual-Language Tasks

TL;DR

InfMLLM is a unified multimodal language model that effectively handles vision-language tasks like captioning, VQA, and grounding through a three-stage training process and a novel visual adapter, achieving state-of-the-art results.

Contribution

This work introduces InfMLLM, a new multimodal language model with a simple visual adapter and a three-stage training scheme for improved vision-language task performance.

Findings

Achieves state-of-the-art or comparable results on benchmark datasets.

The pool-adapter effectively preserves positional information of visual embeddings.

Three-stage training enhances instruction-following capabilities.

Abstract

Large language models (LLMs) have proven their remarkable versatility in handling a comprehensive range of language-centric applications. To expand LLMs' capabilities to a broader spectrum of modal inputs, multimodal large language models (MLLMs) have attracted growing interest. This work delves into enabling LLMs to tackle more vision-language-related tasks, particularly image captioning, visual question answering (VQA,) and visual grounding. To this end, we implemented a three-stage training scheme: starting with lightweight alignment pretraining, then moderate-weight multitask hybrid training, and finally, LLM fine-tuning to improve instruction following capability. Throughout the training process, the requirements on GPU memory gradually increase. To effectively manage the number of visual embeddings passed to the LLM while preserving their positional information, we introduce a straightforward visual adapter module dubbed pool-adapter. Our experiments demonstrate that preserving the positional information of visual embeddings through the pool-adapter is particularly beneficial for tasks like visual grounding. We name our proposed approach InfMLLM and have evaluated it extensively on various benchmark datasets. Our results demonstrate that InfMLLM achieves either state-of-the-art (SOTA) performance or performance comparable to recent MLLMs. The code and model will be made open-source at: \url{https://github.com/mightyzau/InfMLLM}.