Two eyes, Two views, and finally, One summary! Towards Multi-modal Multi-tasking Knowledge-Infused Medical Dialogue Summarization

TL;DR

This paper presents MMK-Summation, a multi-modal, multi-tasking model that effectively summarizes medical dialogues by integrating external knowledge and visual cues, outperforming traditional methods in accuracy and comprehensiveness.

Contribution

The paper introduces a novel multi-modal, multi-tasking, knowledge-infused model for medical dialogue summarization, demonstrating significant improvements over existing baselines.

Findings

Outperforms multiple baseline models across all evaluation metrics.

Effectively integrates external knowledge and visual cues into summaries.

Achieves superior human evaluation scores.

Abstract

We often summarize a multi-party conversation in two stages: chunking with homogeneous units and summarizing the chunks. Thus, we hypothesize that there exists a correlation between homogeneous speaker chunking and overall summarization tasks. In this work, we investigate the effectiveness of a multi-faceted approach that simultaneously produces summaries of medical concerns, doctor impressions, and an overall view. We introduce a multi-modal, multi-tasking, knowledge-infused medical dialogue summary generation (MMK-Summation) model, which is incorporated with adapter-based fine-tuning through a gated mechanism for multi-modal information integration. The model, MMK-Summation, takes dialogues as input, extracts pertinent external knowledge based on the context, integrates the knowledge and visual cues from the dialogues into the textual content, and ultimately generates concise summaries encompassing medical concerns, doctor impressions, and a comprehensive overview. The introduced model surpasses multiple baselines and traditional summarization models across all evaluation metrics (including human evaluation), which firmly demonstrates the efficacy of the knowledge-guided multi-tasking, multimodal medical conversation summarization. The code is available at https://github.com/NLP-RL/MMK-Summation.