Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches

TL;DR

This paper introduces large-scale medical video datasets and approaches for automatically providing visual answers to health-related questions, aiming to improve access to medical knowledge through instructional videos.

Contribution

It presents a novel pipeline for creating large-scale medical video datasets and develops monomodal and multimodal methods for visual answer retrieval from videos.

Findings

Datasets improve model training for medical visual answer localization

Visual features significantly enhance approach performance

Pre-trained language-vision models offer promising future improvements

Abstract

The increase in the availability of online videos has transformed the way we access information and knowledge. A growing number of individuals now prefer instructional videos as they offer a series of step-by-step procedures to accomplish particular tasks. The instructional videos from the medical domain may provide the best possible visual answers to first aid, medical emergency, and medical education questions. Toward this, this paper is focused on answering health-related questions asked by the public by providing visual answers from medical videos. The scarcity of large-scale datasets in the medical domain is a key challenge that hinders the development of applications that can help the public with their health-related questions. To address this issue, we first proposed a pipelined approach to create two large-scale datasets: HealthVidQA-CRF and HealthVidQA-Prompt. Later, we proposed monomodal and multimodal approaches that can effectively provide visual answers from medical videos to natural language questions. We conducted a comprehensive analysis of the results, focusing on the impact of the created datasets on model training and the significance of visual features in enhancing the performance of the monomodal and multi-modal approaches. Our findings suggest that these datasets have the potential to enhance the performance of medical visual answer localization tasks and provide a promising future direction to further enhance the performance by using pre-trained language-vision models.