Leveraging Complementary Attention maps in vision transformers for OCT image analysis

TL;DR

This paper presents a novel pipeline combining hybrid and pure attention vision transformers for OCT biomarker detection, achieving state-of-the-art results and efficient single-model performance through knowledge distillation.

Contribution

It introduces a systematic evaluation of convolution and attention mechanisms in vision transformers for OCT analysis and demonstrates the effectiveness of ensembling and distillation techniques.

Findings

MaxViT excels at local feature detection

EVA-02 captures global features effectively

Ensembling improves biomarker detection accuracy

Abstract

Optical Coherence Tomography (OCT) scan yields all possible cross-section images of a retina for detecting biomarkers linked to optical defects. Due to the high volume of data generated, an automated and reliable biomarker detection pipeline is necessary as a primary screening stage. We outline our new state-of-the-art pipeline for identifying biomarkers from OCT scans. In collaboration with trained ophthalmologists, we identify local and global structures in biomarkers. Through a comprehensive and systematic review of existing vision architectures, we evaluate different convolution and attention mechanisms for biomarker detection. We find that MaxViT, a hybrid vision transformer combining convolution layers with strided attention, is better suited for local feature detection, while EVA-02, a standard vision transformer leveraging pure attention and large-scale knowledge distillation, excels at capturing global features. We ensemble the predictions of both models to achieve first place in the IEEE Video and Image Processing Cup 2023 competition on OCT biomarker detection, achieving a patient-wise F1 score of 0.8527 in the final phase of the competition, scoring 3.8\% higher than the next best solution. Finally, we used knowledge distillation to train a single MaxViT to outperform our ensemble at a fraction of the computation cost.