MTVNet: Mapping using Transformers for Volumes -- Network for Super-Resolution with Long-Range Interactions

TL;DR

MTVNet introduces a hierarchical transformer-based approach for 3D volumetric super-resolution, effectively capturing long-range interactions across multiple scales to improve super-resolution quality on large 3D datasets.

Contribution

The paper presents a novel multi-scale transformer model with hierarchical attention and carrier tokens, enabling larger receptive fields in 3D super-resolution.

Findings

Outperforms state-of-the-art models on five 3D datasets.

Effectively captures long-range interactions in volumetric data.

Shows significant improvements on larger 3D images.

Abstract

Until now, it has been difficult for volumetric super-resolution to utilize the recent advances in transformer-based models seen in 2D super-resolution. The memory required for self-attention in 3D volumes limits the receptive field. Therefore, long-range interactions are not used in 3D to the extent done in 2D and the strength of transformers is not realized. We propose a multi-scale transformer-based model based on hierarchical attention blocks combined with carrier tokens at multiple scales to overcome this. Here information from larger regions at coarse resolution is sequentially carried on to finer-resolution regions to predict the super-resolved image. Using transformer layers at each resolution, our coarse-to-fine modeling limits the number of tokens at each scale and enables attention over larger regions than what has previously been possible. We experimentally compare our method, MTVNet, against state-of-the-art volumetric super-resolution models on five 3D datasets demonstrating the advantage of an increased receptive field. This advantage is especially pronounced for images that are larger than what is seen in popularly used 3D datasets. Our code is available at https://github.com/AugustHoeg/MTVNet