Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups

TL;DR

This paper presents HyperAdapt, a patient-conditioned adaptation framework that enhances diagnostic reliability across diverse patient subgroups by integrating patient attributes into a shared model, improving fairness without sacrificing accuracy.

Contribution

Introduces HyperAdapt, a novel patient-conditioned adaptation method using a hypernetwork to improve subgroup reliability in medical diagnosis models.

Findings

Outperforms baseline by 4.1% in recall on PAD-UFES-20

Achieves 4.4% higher F1 score, especially benefiting underrepresented groups

Maintains overall accuracy while improving subgroup fairness

Abstract

AI models for medical diagnosis often exhibit uneven performance across patient populations due to heterogeneity in disease prevalence, imaging appearance, and clinical risk profiles. Existing algorithmic fairness approaches typically seek to reduce such disparities by suppressing sensitive attributes. However, in medical settings these attributes often carry essential diagnostic information, and removing them can degrade accuracy and reliability, particularly in high-stakes applications. In contrast, clinical decision making explicitly incorporates patient context when interpreting diagnostic evidence, suggesting a different design direction for subgroup-aware models. In this paper, we introduce HyperAdapt, a patient-conditioned adaptation framework that improves subgroup reliability while maintaining a shared diagnostic model. Clinically relevant attributes such as age and sex are encoded into a compact embedding and used to condition a hypernetwork-style module, which generates small residual modulation parameters for selected layers of a shared backbone. This design preserves the general medical knowledge learned by the backbone while enabling targeted adjustments that reflect patient-specific variability. To ensure efficiency and robustness, adaptations are constrained through low-rank and bottlenecked parameterizations, limiting both model complexity and computational overhead. Experiments across multiple public medical imaging benchmarks demonstrate that the proposed approach consistently improves subgroup-level performance without sacrificing overall accuracy. On the PAD-UFES-20 dataset, our method outperforms the strongest competing baseline by 4.1% in recall and 4.4% in F1 score, with larger gains observed for underrepresented patient populations.