A Study on Context Length and Efficient Transformers for Biomedical Image Analysis

TL;DR

This paper systematically evaluates how context length affects transformer performance in biomedical image analysis, demonstrating that longer contexts improve accuracy and that recent long-context models offer efficiency gains with maintained performance.

Contribution

It provides a comprehensive analysis of context length effects and evaluates recent long-context models in biomedical imaging, highlighting their benefits and remaining challenges.

Findings

Longer context lengths improve segmentation and classification performance.

Recent long-context models achieve efficiency gains without sacrificing accuracy.

Performance gains are especially notable in pixel-level prediction tasks.

Abstract

Biomedical imaging modalities often produce high-resolution, multi-dimensional images that pose computational challenges for deep neural networks. These computational challenges are compounded when training transformers due to the self-attention operator, which scales quadratically with context length. Recent developments in long-context models have potential to alleviate these difficulties and enable more efficient application of transformers to large biomedical images, although a systematic evaluation on this topic is lacking. In this study, we investigate the impact of context length on biomedical image analysis and we evaluate the performance of recently proposed long-context models. We first curate a suite of biomedical imaging datasets, including 2D and 3D data for segmentation, denoising, and classification tasks. We then analyze the impact of context length on network performance using the Vision Transformer and Swin Transformer by varying patch size and attention window size. Our findings reveal a strong relationship between context length and performance, particularly for pixel-level prediction tasks. Finally, we show that recent long-context models demonstrate significant improvements in efficiency while maintaining comparable performance, though we highlight where gaps remain. This work underscores the potential and challenges of using long-context models in biomedical imaging.