Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

TL;DR

This survey explores how integrating external tools like APIs and knowledge bases can significantly enhance the performance, data quality, evaluation, and future development of Multimodal Large Language Models (MLLMs).

Contribution

It provides a comprehensive overview of leveraging external tools to improve MLLMs across data acquisition, task performance, evaluation, and future research directions.

Findings

External tools can improve multimodal data quality and annotation.

Tool integration enhances MLLM performance on complex tasks.

The survey identifies current limitations and future opportunities for tool-augmented MLLMs.

Abstract

By integrating the perception capabilities of multimodal encoders with the generative power of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs), exemplified by GPT-4V, have achieved great success in various multimodal tasks, pointing toward a promising pathway to artificial general intelligence. Despite this progress, the limited quality of multimodal data, poor performance on many complex downstream tasks, and inadequate evaluation protocols continue to hinder the reliability and broader applicability of MLLMs across diverse domains. Inspired by the human ability to leverage external tools for enhanced reasoning and problem-solving, augmenting MLLMs with external tools (e.g., APIs, expert models, and knowledge bases) offers a promising strategy to overcome these challenges. In this paper, we present a comprehensive survey on leveraging external tools to enhance MLLM performance. Our discussion is structured along four key dimensions about external tools: (1) how they can facilitate the acquisition and annotation of high-quality multimodal data; (2) how they can assist in improving MLLM performance on challenging downstream tasks; (3) how they enable comprehensive and accurate evaluation of MLLMs; (4) the current limitations and future directions of tool-augmented MLLMs. Through this survey, we aim to underscore the transformative potential of external tools in advancing MLLM capabilities, offering a forward-looking perspective on their development and applications. The project page of this paper is publicly available athttps://github.com/Lackel/Awesome-Tools-for-MLLMs.