Spatially Grounded Explanations in Vision Language Models for Document Visual Question Answering

TL;DR

This paper presents EaGERS, a training-free, model-agnostic approach for generating spatially grounded explanations in vision-language models, improving transparency and accuracy in document visual question answering without additional training.

Contribution

Introduces EaGERS, a novel pipeline that generates and grounds natural language rationales to spatial regions, enhancing interpretability and performance in DocVQA without model fine-tuning.

Findings

Outperforms base model on accuracy and similarity metrics.

Enhances transparency and reproducibility in DocVQA.

Operates without additional model training.

Abstract

We introduce EaGERS, a fully training-free and model-agnostic pipeline that (1) generates natural language rationales via a vision language model, (2) grounds these rationales to spatial sub-regions by computing multimodal embedding similarities over a configurable grid with majority voting, and (3) restricts the generation of responses only from the relevant regions selected in the masked image. Experiments on the DocVQA dataset demonstrate that our best configuration not only outperforms the base model on exact match accuracy and Average Normalized Levenshtein Similarity metrics but also enhances transparency and reproducibility in DocVQA without additional model fine-tuning.