Jaeger: A Concatenation-Based Multi-Transformer VQA Model

TL;DR

The paper introduces Jaeger, a multi-transformer VQA model that combines features from RoBERTa and GPT2-xl through concatenation and dimensionality reduction, aiming to improve efficiency and performance in document-based VQA tasks.

Contribution

Jaeger is a novel concatenation-based multi-transformer VQA model that leverages pre-trained language models for enhanced feature representation and reduced inference time.

Findings

Achieves competitive performance on PDF-VQA Dataset Task C.

Reduces inference time through dimensionality reduction.

Utilizes concatenation of features from RoBERTa and GPT2-xl.

Abstract

Document-based Visual Question Answering poses a challenging task between linguistic sense disambiguation and fine-grained multimodal retrieval. Although there has been encouraging progress in document-based question answering due to the utilization of large language and open-world prior models\cite{1}, several challenges persist, including prolonged response times, extended inference durations, and imprecision in matching. In order to overcome these challenges, we propose Jaegar, a concatenation-based multi-transformer VQA model. To derive question features, we leverage the exceptional capabilities of RoBERTa large\cite{2} and GPT2-xl\cite{3} as feature extractors. Subsequently, we subject the outputs from both models to a concatenation process. This operation allows the model to consider information from diverse sources concurrently, strengthening its representational capability. By leveraging pre-trained models for feature extraction, our approach has the potential to amplify the performance of these models through concatenation. After concatenation, we apply dimensionality reduction to the output features, reducing the model's computational effectiveness and inference time. Empirical results demonstrate that our proposed model achieves competitive performance on Task C of the PDF-VQA Dataset. If the user adds any new data, they should make sure to style it as per the instructions provided in previous sections.