Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision

TL;DR

This paper introduces CASS, a self-supervised learning method combining Transformers and CNNs, which improves healthcare data modeling efficiency, robustness, and performance with less data and reduced pretraining time.

Contribution

CASS is a novel siamese self-supervised approach that synergistically leverages Transformer and CNN architectures for efficient healthcare representation learning.

Findings

CASS outperforms existing methods across four healthcare datasets.

Achieves up to 10.13% improvement with full labeled data.

Reduces pretraining time by 69% compared to state-of-the-art.

Abstract

In healthcare and biomedical applications, extreme computational requirements pose a significant barrier to adopting representation learning. Representation learning can enhance the performance of deep learning architectures by learning useful priors from limited medical data. However, state-of-the-art self-supervised techniques suffer from reduced performance when using smaller batch sizes or shorter pretraining epochs, which are more practical in clinical settings. We present Cross Architectural - Self Supervision (CASS) in response to this challenge. This novel siamese self-supervised learning approach synergistically leverages Transformer and Convolutional Neural Networks (CNN) for efficient learning. Our empirical evaluation demonstrates that CASS-trained CNNs and Transformers outperform existing self-supervised learning methods across four diverse healthcare datasets. With only 1% labeled data for finetuning, CASS achieves a 3.8% average improvement; with 10% labeled data, it gains 5.9%; and with 100% labeled data, it reaches a remarkable 10.13% enhancement. Notably, CASS reduces pretraining time by 69% compared to state-of-the-art methods, making it more amenable to clinical implementation. We also demonstrate that CASS is considerably more robust to variations in batch size and pretraining epochs, making it a suitable candidate for machine learning in healthcare applications.