Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings

TL;DR

Pathryoshka is a multi-teacher knowledge distillation framework that significantly compresses pathology foundation models, reducing size by up to 92% while maintaining or improving performance across various benchmarks.

Contribution

We introduce Pathryoshka, a novel multi-teacher distillation method inspired by RADIO and Matryoshka Learning, enabling adaptable, compact pathology models with high accuracy.

Findings

Reduces model size by 86-92%

Outperforms single-teacher distillation models by median 7.0 accuracy points

Maintains high performance across ten public pathology benchmarks

Abstract

Pathology foundation models (FMs) have driven significant progress in computational pathology. However, these high-performing models can easily exceed a billion parameters and produce high-dimensional embeddings, thus limiting their applicability for research or clinical use when computing resources are tight. Here, we introduce Pathryoshka, a multi-teacher distillation framework inspired by RADIO distillation and Matryoshka Representation Learning to reduce pathology FM sizes while allowing for adaptable embedding dimensions. We evaluate our framework with a distilled model on ten public pathology benchmarks with varying downstream tasks. Compared to its much larger teachers, Pathryoshka reduces the model size by 86-92% at on-par performance. It outperforms state-of-the-art single-teacher distillation models of comparable size by a median margin of 7.0 in accuracy. By enabling efficient local deployment without sacrificing accuracy or representational richness, Pathryoshka democratizes access to state-of-the-art pathology FMs for the broader research and clinical community.