Weakly Supervised Concept Learning with Class-Level Priors for Interpretable Medical Diagnosis

TL;DR

This paper introduces a weakly supervised framework called PCP that predicts medical concepts without explicit annotations, improving interpretability and reliability in medical diagnosis tasks.

Contribution

The paper presents a novel Prior-guided Concept Predictor (PCP) that leverages class-level priors for weak supervision, reducing the need for costly annotations in medical imaging.

Findings

PCP improves concept-level F1-score by over 33% over zero-shot baselines.

PCP achieves competitive classification performance on multiple medical datasets.

The method enhances interpretability without relying on explicit concept annotations.

Abstract

Human-interpretable predictions are essential for deploying AI in medical imaging, yet most interpretable-by-design (IBD) frameworks require concept annotations for training data, which are costly and impractical to obtain in clinical contexts. Recent attempts to bypass annotation, such as zero-shot vision-language models or concept-generation frameworks, struggle to capture domain-specific medical features, leading to poor reliability. In this paper, we propose a novel Prior-guided Concept Predictor (PCP), a weakly supervised framework that enables concept answer prediction without explicit supervision or reliance on language models. PCP leverages class-level concept priors as weak supervision and incorporates a refinement mechanism with KL divergence and entropy regularization to align predictions with clinical reasoning. Experiments on PH2 (dermoscopy) and WBCatt (hematology) show that PCP improves concept-level F1-score by over 33% compared to zero-shot baselines, while delivering competitive classification performance on four medical datasets (PH2, WBCatt, HAM10000, and CXR4) relative to fully supervised concept bottleneck models (CBMs) and V-IP.