ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools

TL;DR

ToolVQA introduces a large, real-world multimodal dataset with multi-step reasoning tasks and diverse tools, enabling better evaluation and training of foundation models for practical tool-augmented VQA.

Contribution

We present ToolVQA, a new dataset with real-world contexts and multi-step reasoning, along with ToolEngine, a data generation pipeline for simulating human-like tool use in multimodal tasks.

Findings

Fine-tuned 7B LFMs perform well on ToolVQA test set.

Models surpass GPT-3.5-turbo on out-of-distribution datasets.

ToolVQA bridges the gap between synthetic datasets and real-world applications.

Abstract

Integrating external tools into Large Foundation Models (LFMs) has emerged as a promising approach to enhance their problem-solving capabilities. While existing studies have demonstrated strong performance in tool-augmented Visual Question Answering (VQA), recent benchmarks reveal significant gaps in real-world tool-use proficiency, particularly in functionally diverse multimodal settings requiring multi-step reasoning. In this work, we introduce ToolVQA, a large-scale multimodal dataset comprising 23K instances, designed to bridge this gap. Unlike previous datasets that rely on synthetic scenarios and simplified queries, ToolVQA features real-world visual contexts and challenging implicit multi-step reasoning tasks, better aligning with real user interactions. To construct this dataset, we propose ToolEngine, a novel data generation pipeline that employs Depth-First Search (DFS) with a dynamic in-context example matching mechanism to simulate human-like tool-use reasoning. ToolVQA encompasses 10 multimodal tools across 7 diverse task domains, with an average inference length of 2.78 reasoning steps per instance. The fine-tuned 7B LFMs on ToolVQA not only achieve impressive performance on our test set but also surpass the large close-sourced model GPT-3.5-turbo on various out-of-distribution (OOD) datasets, demonstrating strong generalizability to real-world tool-use scenarios.