Evaluating VisualRAG: Quantifying Cross-Modal Performance in Enterprise Document Understanding

TL;DR

This paper introduces a quantitative benchmarking framework for assessing trustworthiness in multimodal VisualRAG systems, demonstrating how optimal modality weighting improves performance and reliability in enterprise document understanding.

Contribution

The paper presents a systematic framework for measuring trustworthiness in multimodal VisualRAG systems, linking technical metrics with user trust, and evaluates foundation models' impact on enterprise AI reliability.

Findings

Optimal modality weights (30% text, 15% image, 25% caption, 30% OCR) improve performance by 57.3%.

Framework establishes quantitative relationships between technical metrics and user trust.

Foundation models differ significantly in trustworthiness for captioning and OCR tasks.

Abstract

Current evaluation frameworks for multimodal generative AI struggle to establish trustworthiness, hindering enterprise adoption where reliability is paramount. We introduce a systematic, quantitative benchmarking framework to measure the trustworthiness of progressively integrating cross-modal inputs such as text, images, captions, and OCR within VisualRAG systems for enterprise document intelligence. Our approach establishes quantitative relationships between technical metrics and user-centric trust measures. Evaluation reveals that optimal modality weighting with weights of 30% text, 15% image, 25% caption, and 30% OCR improves performance by 57.3% over text-only baselines while maintaining computational efficiency. We provide comparative assessments of foundation models, demonstrating their differential impact on trustworthiness in caption generation and OCR extraction-a vital consideration for reliable enterprise AI. This work advances responsible AI deployment by providing a rigorous framework for quantifying and enhancing trustworthiness in multimodal RAG for critical enterprise applications.