Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles

TL;DR

This paper demonstrates that architecture-agnostic soft label knowledge distillation from ensembles significantly improves respiratory sound classification accuracy, achieving state-of-the-art results with reduced inference costs.

Contribution

It introduces a soft label distillation method from ensembles that enhances respiratory sound classification across architectures, even with a single teacher model.

Findings

Single teacher distillation improves performance beyond its own capacity.

Optimal gains achieved with few teacher models.

State-of-the-art Score of 64.39 on ICHBI dataset.

Abstract

Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.