Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging

TL;DR

This paper introduces a novel hierarchical multi-label learning to defer framework for medical imaging, ensuring coherent deferrals aligned with clinical taxonomies and improving decision consistency.

Contribution

It formalizes coherent hierarchical deferral, characterizes the optimal rule, and proposes two methods—exact projection and TBP+RPO—to ensure coherence in medical imaging workflows.

Findings

Projection removes deferral incoherence exactly.

TBP+RPO reduces incoherence near zero.

Methods retain strong utility in benchmarks.

Abstract

Learning to Defer (L2D) enables a model to predict autonomously or defer to an expert, but prior work largely assumes flat label spaces. We study the first L2D setting with hierarchical multi-label decisions, motivated by medical-imaging workflows in which findings are organised by clinical taxonomies. In this setting, deferral is a delegation action rather than a label assignment, so treating it as an independent per-label decision can produce deferral incoherence, including taxonomic contradictions, delegation violations, and deferrals of labels already implied by the model's own assertions. We formalise coherent hierarchical deferral under a Selective-Exclusion handoff contract, characterise the Bayes-optimal coherent deferral rule, and show that even nodewise Bayes L2D can be action-incoherent. We then propose two remedies: exact coherent projection, a dynamic-programming decoder over the coherent action set, and Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO), a contract-aware joint action model trained through the same recursion used at inference. Across real-reader and controlled-expert medical-imaging benchmarks, naive binary-relevance L2D exhibits non-trivial incoherence. Projection removes it exactly, and fast TBP+RPO drives incoherence near zero while retaining strong utility.