How Close Are We? Limitations and Progress of AI Models in Banff Lesion Scoring

TL;DR

This study assesses the capabilities and limitations of current deep learning models in automating Banff lesion scoring for renal transplant biopsies, revealing partial successes and critical failure modes that impact interpretability.

Contribution

It introduces a modular, rule-based framework to evaluate AI performance in replicating expert Banff grading, highlighting current model limitations and guiding future improvements.

Findings

Partial success in automating Banff scoring

Identification of critical failure modes like hallucination and ambiguity

Highlighting the need for modular evaluation standards

Abstract

The Banff Classification provides the global standard for evaluating renal transplant biopsies, yet its semi-quantitative nature, complex criteria, and inter-observer variability present significant challenges for computational replication. In this study, we explore the feasibility of approximating Banff lesion scores using existing deep learning models through a modular, rule-based framework. We decompose each Banff indicator - such as glomerulitis (g), peritubular capillaritis (ptc), and intimal arteritis (v) - into its constituent structural and inflammatory components, and assess whether current segmentation and detection tools can support their computation. Model outputs are mapped to Banff scores using heuristic rules aligned with expert guidelines, and evaluated against expert-annotated ground truths. Our findings highlight both partial successes and critical failure modes, including structural omission, hallucination, and detection ambiguity. Even when final scores match expert annotations, inconsistencies in intermediate representations often undermine interpretability. These results reveal the limitations of current AI pipelines in replicating computational expert-level grading, and emphasize the importance of modular evaluation and computational Banff grading standard in guiding future model development for transplant pathology.