FlipVQA: Scaling Multi-modal Instruction Tuning via Textbook-to-Knowledge Synthesis

TL;DR

FlipVQA introduces an automated pipeline to extract and curate large-scale question-answering datasets from textbooks, significantly reducing costs while maintaining high data quality for multi-modal reasoning tasks.

Contribution

The paper presents FlipVQA-Miner, a novel method for extracting structured QA and VQA pairs from complex textbook layouts, enabling scalable, high-fidelity data generation.

Findings

Constructed FlipVQA-83K dataset with 83,000 QA pairs across 11 disciplines

Achieved 50x cost savings compared to manual annotation

Models trained on FlipVQA-83K show improved reasoning and generalization

Abstract

Textbooks are among the richest repositories of human-verified reasoning knowledge, yet their complex layouts contain multi-column typesetting, cross-page question answer separation, and interleaved figures, make automated extraction of structured QA and VQA pairs extremely challenging. Existing alternatives either synthesize data from scratch, which lacks authentic problem contexts, or rely on costly expert annotation that cannot scale. We propose $\textbf{FlipVQA-Miner}$, an automated pipeline that resolves long-range logical dependencies and cross-page discontinuities in OCR-parsed documents, recovering coherent question--answer--figure associations even when answers reside in separate companion volumes. A subsequent multi-stage curation pipeline transforms these raw extractions into AI-ready supervision signals. Using FlipVQA-Miner, we construct $\textbf{FlipVQA-83K}$, comprising 83K QA and VQA pairs spanning 11 academic disciplines, at a $\textbf{50$\times$}$ cost saving compared to manual annotation while maintaining high structural fidelity ($F_1 > 0.96$). Models fine-tuned on FlipVQA-83K demonstrate significantly improved reasoning ability and cross-domain generalization, establishing a scalable paradigm for human-knowledge-grounded data curation. Our dataset and the complete data generating and curating methods can be found in https://github.com/OpenDCAI/DataFlow-VQA .