Which Client is Reliable?: A Reliable and Personalized Prompt-based Federated Learning for Medical Image Question Answering

TL;DR

This paper proposes a personalized federated learning approach for medical visual question answering that enhances reliability and privacy by integrating learnable prompts, uncertainty quantification via Dempster-Shafer theory, and an inter-client communication mechanism.

Contribution

It introduces a novel personalized federated learning framework with uncertainty quantification and efficient communication for medical VQA, addressing privacy and reliability challenges.

Findings

Effective uncertainty quantification improves prediction reliability.

The method balances accuracy and uncertainty across clients.

Enhanced privacy-preserving medical VQA performance.

Abstract

Conventional medical artificial intelligence (AI) models face barriers in clinical application and ethical issues owing to their inability to handle the privacy-sensitive characteristics of medical data. We present a novel personalized federated learning (pFL) method for medical visual question answering (VQA) models, addressing privacy reliability challenges in the medical domain. Our method introduces learnable prompts into a Transformer architecture to efficiently train it on diverse medical datasets without massive computational costs. Then we introduce a reliable client VQA model that incorporates Dempster-Shafer evidence theory to quantify uncertainty in predictions, enhancing the model's reliability. Furthermore, we propose a novel inter-client communication mechanism that uses maximum likelihood estimation to balance accuracy and uncertainty, fostering efficient integration of insights across clients.