Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

TL;DR

This paper introduces a multi-agent framework using large language models for knowledge-based visual question answering, enabling autonomous tool use and team collaboration to improve answer accuracy.

Contribution

It presents a novel multi-agent voting framework that simulates team collaboration and tool utilization in LLM-based VQA, outperforming existing methods.

Findings

Outperforms baselines on OK-VQA and A-OKVQA datasets

Enables autonomous external tool usage in LLMs

Simulates human-like team collaboration in VQA

Abstract

Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.