Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant

TL;DR

This paper introduces VisTEL and KaLMA, innovative methods that leverage visual text entity linking and knowledge integration to significantly improve Text-KVQA accuracy using large multimodal models.

Contribution

The paper presents VisTEL for visual text entity linking and KaLMA, a knowledge-aware multimodal assistant, advancing Text-KVQA performance with state-of-the-art results.

Findings

Surpassed previous best by 23.3% on Text-KVQA

Established new state-of-the-art performance

Provided comprehensive experimental analysis

Abstract

We revisit knowledge-aware text-based visual question answering, also known as Text-KVQA, in the light of modern advancements in large multimodal models (LMMs), and make the following contributions: (i) We propose VisTEL - a principled approach to perform visual text entity linking. The proposed VisTEL module harnesses a state-of-the-art visual text recognition engine and the power of a large multimodal model to jointly reason using textual and visual context obtained using surrounding cues in the image to link the visual text entity to the correct knowledge base entity. (ii) We present KaLMA - a knowledge-aware large multimodal assistant that augments an LMM with knowledge associated with visual text entity in the image to arrive at an accurate answer. Further, we provide a comprehensive experimental analysis and comparison of our approach with traditional visual question answering, pre-large multimodal models, and large multimodal models, as well as prior top-performing approaches. Averaging over three splits of Text-KVQA, our proposed approach surpasses the previous best approach by a substantial 23.3% on an absolute scale and establishes a new state of the art. We make our implementation publicly available.