Gamified crowd-sourcing of high-quality data for visual fine-tuning

TL;DR

This paper presents a gamified crowdsourcing framework called GAP that collects high-quality visual instruction data, significantly improving the performance of small multimodal models and demonstrating cross-model benefits.

Contribution

The paper introduces a scalable, engaging platform for crowd-sourcing targeted visual question-answer pairs to enhance multimodal model training, addressing model weaknesses effectively.

Findings

Improved model accuracy from 0.147 to 0.477 GPT score.

Collected data from over 50,000 participants in weeks.

Enhanced performance across multiple models and benchmarks.

Abstract

This paper introduces Gamified Adversarial Prompting (GAP), a framework that crowd-sources high-quality data for visual instruction tuning of large multimodal models. GAP transforms the data collection process into an engaging game, incentivizing players to provide fine-grained, challenging questions and answers that target gaps in the model's knowledge. Our contributions include (1) an approach to capture question-answer pairs from humans that directly address weaknesses in a model's knowledge, (2) a method for evaluating and rewarding players that successfully incentivizes them to provide high-quality submissions, and (3) a scalable, gamified platform that succeeds in collecting this data from over 50,000 participants in just a few weeks. Our implementation of GAP has significantly improved the accuracy of a small multimodal model, namely MiniCPM-Llama3-V-2.5-8B, increasing its GPT score from 0.147 to 0.477 on our dataset, approaching the benchmark set by the much larger GPT-4V. Moreover, we demonstrate that the data generated using MiniCPM-Llama3-V-2.5-8B also enhances its performance across other benchmarks, and exhibits cross-model benefits. Specifically, the same data improves the performance of QWEN2-VL-2B and QWEN2-VL-7B on the same multiple benchmarks.