A New Perspective to Boost Vision Transformer for Medical Image Classification

TL;DR

This paper introduces BOLT, a self-supervised learning method for medical image classification using Transformers, which effectively learns from limited data without relying on large-scale labeled datasets like ImageNet.

Contribution

The paper proposes BOLT, a novel self-supervised approach with an auxiliary difficulty ranking task to enhance Transformer training for medical images without large-scale pretraining.

Findings

BOLT outperforms ImageNet pretrained weights in medical image tasks.

BOLT surpasses existing self-supervised methods in accuracy.

Effective learning from limited medical data with Transformer backbone.

Abstract

Transformer has achieved impressive successes for various computer vision tasks. However, most of existing studies require to pretrain the Transformer backbone on a large-scale labeled dataset (e.g., ImageNet) for achieving satisfactory performance, which is usually unavailable for medical images. Additionally, due to the gap between medical and natural images, the improvement generated by the ImageNet pretrained weights significantly degrades while transferring the weights to medical image processing tasks. In this paper, we propose Bootstrap Own Latent of Transformer (BOLT), a self-supervised learning approach specifically for medical image classification with the Transformer backbone. Our BOLT consists of two networks, namely online and target branches, for self-supervised representation learning. Concretely, the online network is trained to predict the target network representation of the same patch embedding tokens with a different perturbation. To maximally excavate the impact of Transformer from limited medical data, we propose an auxiliary difficulty ranking task. The Transformer is enforced to identify which branch (i.e., online/target) is processing the more difficult perturbed tokens. Overall, the Transformer endeavours itself to distill the transformation-invariant features from the perturbed tokens to simultaneously achieve difficulty measurement and maintain the consistency of self-supervised representations. The proposed BOLT is evaluated on three medical image processing tasks, i.e., skin lesion classification, knee fatigue fracture grading and diabetic retinopathy grading. The experimental results validate the superiority of our BOLT for medical image classification, compared to ImageNet pretrained weights and state-of-the-art self-supervised learning approaches.