Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphs

TL;DR

This paper introduces Failure is Feedback (FiF), a novel history-aware backtracking method for multimodal document retrieval that dynamically adapts reasoning strategies based on past failures, improving accuracy and efficiency.

Contribution

FiF presents a history-aware backtracking mechanism and an economically-rational agentic workflow for more adaptive and cost-effective multimodal retrieval.

Findings

Achieves state-of-the-art results on MultimodalQA, MMCoQA, and WebQA benchmarks.

Effectively leverages failure history to improve retrieval accuracy.

Balances retrieval quality and inference costs dynamically.

Abstract

Open-domain multimodal document retrieval aims to retrieve specific components (paragraphs, tables, or images) from large and interconnected document corpora. Existing graph-based retrieval approaches typically rely on a uniform similarity metric that overlooks hop-specific semantics, and their rigid pre-defined plans hinder dynamic error correction. These limitations suggest that a retriever should adapt its reasoning to the evolving context and recover intelligently from dead ends. To address these needs, we propose Failure is Feedback (FiF), which casts subgraph retrieval as a sequential decision process and introduces two key innovations. (i) We introduce a history-aware backtracking mechanism; unlike standard backtracking that simply reverts the state, our approach piggybacks on the context of failed traversals, leveraging insights from previous failures. (ii) We implement an economically-rational agentic workflow. Unlike conventional agents with static strategies, our orchestrator employs a cost-aware traversal method to dynamically manage the trade-off between retrieval accuracy and inference costs, escalating to intensive LLM-based reasoning only when the prior failure justifies the additional computational investment. Extensive experiments show that FiF achieves state-of-the-art retrieval on the benchmarks of MultimodalQA, MMCoQA and WebQA.