AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation

TL;DR

This paper introduces a self-evolving vision transformer for chest X-ray diagnosis that leverages knowledge distillation and self-supervised learning to improve performance with minimal labeled data, reducing reliance on manual annotations.

Contribution

The authors propose a novel framework combining self-supervised learning and self-training for medical image diagnosis that enhances model performance using mostly unlabeled data.

Findings

Model outperforms counterparts trained with the same labeled data amount.

Framework demonstrates robustness in real-world diagnostic environments.

Applicable to multiple chest X-ray diagnostic tasks.

Abstract

Although deep learning-based computer-aided diagnosis systems have recently achieved expert-level performance, developing a robust deep learning model requires large, high-quality data with manual annotation, which is expensive to obtain. This situation poses the problem that the chest x-rays collected annually in hospitals cannot be used due to the lack of manual labeling by experts, especially in deprived areas. To address this, here we present a novel deep learning framework that uses knowledge distillation through self-supervised learning and self-training, which shows that the performance of the original model trained with a small number of labels can be gradually improved with more unlabeled data. Experimental results show that the proposed framework maintains impressive robustness against a real-world environment and has general applicability to several diagnostic tasks such as tuberculosis, pneumothorax, and COVID-19. Notably, we demonstrated that our model performs even better than those trained with the same amount of labeled data. The proposed framework has a great potential for medical imaging, where plenty of data is accumulated every year, but ground truth annotations are expensive to obtain.