TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering

TL;DR

This paper introduces TMPathVQA, a large multilingual speech-based VQA dataset for medical images, enabling hands-free interaction and benchmarking different system configurations.

Contribution

It presents the TMPathVQA dataset with over 98,000 spoken questions in multiple languages, expanding the PathVQA dataset for speech-based medical VQA applications.

Findings

Benchmarking of various acoustic and visual feature combinations.

Demonstration of the dataset's potential for multilingual speech-based medical VQA.

Enhanced accessibility for medical diagnostics through speech-based systems.

Abstract

In healthcare and medical diagnostics, Visual Question Answering (VQA) mayemergeasapivotal tool in scenarios where analysis of intricate medical images becomes critical for accurate diagnoses. Current text-based VQA systems limit their utility in scenarios where hands-free interaction and accessibility are crucial while performing tasks. A speech-based VQA system may provide a better means of interaction where information can be accessed while performing tasks simultaneously. To this end, this work implements a speech-based VQA system by introducing a Textless Multilingual Pathological VQA (TMPathVQA) dataset, an expansion of the PathVQA dataset, containing spoken questions in English, German & French. This dataset comprises 98,397 multilingual spoken questions and answers based on 5,004 pathological images along with 70 hours of audio. Finally, this work benchmarks and compares TMPathVQA systems implemented using various combinations of acoustic and visual features.