A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering

TL;DR

This paper introduces WSDAN, a dual-attention learning network with word and sentence embeddings, to improve medical visual question answering by enhancing feature extraction and understanding of medical images and questions.

Contribution

The study proposes a novel dual-attention network with a transformer-based sentence embedding module, significantly enhancing visual reasoning and understanding in MVQA tasks.

Findings

Outperforms previous state-of-the-art methods on VQA-MED 2019 and VQA-RAD datasets.

Effectively captures medical knowledge-rich features and fine-grained question understanding.

Demonstrates strong visual reasoning ability through ablation studies and Grad-CAM analysis.

Abstract

Research in medical visual question answering (MVQA) can contribute to the development of computeraided diagnosis. MVQA is a task that aims to predict accurate and convincing answers based on given medical images and associated natural language questions. This task requires extracting medical knowledge-rich feature content and making fine-grained understandings of them. Therefore, constructing an effective feature extraction and understanding scheme are keys to modeling. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text. Meanwhile, some visual and textual feature understanding schemes cannot effectively capture the correlation between regions and keywords for reasonable visual reasoning. In this study, a dual-attention learning network with word and sentence embedding (WSDAN) is proposed. We design a module, transformer with sentence embedding (TSE), to extract a double embedding representation of questions containing keywords and medical information. A dualattention learning (DAL) module consisting of self-attention and guided attention is proposed to model intensive intramodal and intermodal interactions. With multiple DAL modules (DALs), learning visual and textual co-attention can increase the granularity of understanding and improve visual reasoning. Experimental results on the ImageCLEF 2019 VQA-MED (VQA-MED 2019) and VQA-RAD datasets demonstrate that our proposed method outperforms previous state-of-the-art methods. According to the ablation studies and Grad-CAM maps, WSDAN can extract rich textual information and has strong visual reasoning ability.