ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering

TL;DR

ORCA introduces a multi-agent system for document VQA that decomposes complex questions, coordinates specialized agents, and employs iterative refinement to improve reasoning accuracy and reliability.

Contribution

The paper presents a novel multi-agent framework with strategic coordination and iterative refinement for improved document visual question answering.

Findings

Significant performance improvements over state-of-the-art methods

Effective decomposition of complex questions into manageable sub-tasks

Robust answer validation through debate and sanity checks

Abstract

Document Visual Question Answering (DocVQA) remains challenging for existing Vision-Language Models (VLMs), especially under complex reasoning and multi-step workflows. Current approaches struggle to decompose intricate questions into manageable sub-tasks and often fail to leverage specialized processing paths for different document elements. We present ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering, a novel multi-agent framework that addresses these limitations through strategic agent coordination and iterative refinement. ORCA begins with a reasoning agent that decomposes queries into logical steps, followed by a routing mechanism that activates task-specific agents from a specialized agent dock. Our framework leverages a set of specialized AI agents, each dedicated to a distinct modality, enabling fine-grained understanding and collaborative reasoning across diverse document components. To ensure answer reliability, ORCA employs a debate mechanism with stress-testing, and when necessary, a thesis-antithesis adjudication process. This is followed by a sanity checker to ensure format consistency. Extensive experiments on three benchmarks demonstrate that our approach achieves significant improvements over state-of-the-art methods, establishing a new paradigm for collaborative agent systems in vision-language reasoning.