VQA Training Sets are Self-play Environments for Generating Few-shot Pools

TL;DR

This paper introduces a method to transform existing VQA training sets into self-play environments, enabling models to autonomously improve their reasoning skills by iteratively selecting training examples that maximize task metrics.

Contribution

The authors propose a novel technique that leverages existing datasets for self-play training, reducing dataset construction costs and enhancing model performance in visual-question answering tasks.

Findings

Models can learn to use themselves or other models as tools.

The approach improves zero-shot performance on various VQA datasets.

Self-play training enhances reasoning capabilities in visual question answering.

Abstract

Large-language models and large-vision models are increasingly capable of solving compositional reasoning tasks, as measured by breakthroughs in visual-question answering benchmarks. However, state-of-the-art solutions often involve careful construction of large pre-training and fine-tuning datasets, which can be expensive. The use of external tools, whether other ML models, search engines, or APIs, can significantly improve performance by breaking down high-level reasoning questions into sub-questions that are answerable by individual tools, but this approach has similar dataset construction costs to teach fine-tuned models how to use the available tools. We propose a technique in which existing training sets can be directly used for constructing computational environments with task metrics as rewards. This enables a model to autonomously teach itself to use itself or another model as a tool. By doing so, we augment training sets by integrating external signals. The proposed method starts with zero-shot prompts and iteratively refines them by selecting few-shot examples that maximize the task metric on the training set. Our experiments showcase how Gemini learns how to use itself, or another smaller and specialized model such as ScreenAI, to iteratively improve performance on training sets. Our approach successfully generalizes and improves upon zeroshot performance on charts, infographics, and document visual question-answering datasets