DocVXQA: Context-Aware Visual Explanations for Document Question Answering

TL;DR

DocVXQA introduces a framework for document question answering that provides accurate answers along with visual heatmaps highlighting critical regions, balancing interpretability and performance.

Contribution

It presents a novel explainability-driven learning framework that generates contextually sufficient visual explanations for document QA, improving interpretability without sacrificing accuracy.

Findings

Effective visual heatmaps highlight critical document regions.

Balanced approach improves trust and interpretability.

Human evaluation confirms explanation quality.

Abstract

We propose DocVXQA, a novel framework for visually self-explainable document question answering. The framework is designed not only to produce accurate answers to questions but also to learn visual heatmaps that highlight contextually critical regions, thereby offering interpretable justifications for the model's decisions. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning objectives. Unlike conventional methods that emphasize only the regions pertinent to the answer, our framework delivers explanations that are \textit{contextually sufficient} while remaining \textit{representation-efficient}. This fosters user trust while achieving a balance between predictive performance and interpretability in DocVQA applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method. The code is available at https://github.com/dali92002/DocVXQA.