Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokens

TL;DR

This paper introduces a multi-scale vision transformer approach with relay tokens for ultra-high resolution semantic segmentation, effectively combining local detail preservation with global context understanding.

Contribution

It presents a simple method integrating relay tokens into standard transformers to enhance multi-scale reasoning without significant parameter increase.

Findings

Achieves up to 15% relative mIoU improvement on benchmarks.

Works with standard transformer backbones like ViT and Swin.

Requires fewer than 2% additional parameters.

Abstract

Current approaches for segmenting ultra high resolution images either slide a window, thereby discarding global context, or downsample and lose fine detail. We propose a simple yet effective method that brings explicit multi scale reasoning to vision transformers, simultaneously preserving local details and global awareness. Concretely, we process each image in parallel at a local scale (high resolution, small crops) and a global scale (low resolution, large crops), and aggregate and propagate features between the two branches with a small set of learnable relay tokens. The design plugs directly into standard transformer backbones (eg ViT and Swin) and adds fewer than 2 % parameters. Extensive experiments on three ultra high resolution segmentation benchmarks, Archaeoscape, URUR, and Gleason, and on the conventional Cityscapes dataset show consistent gains, with up to 15 % relative mIoU improvement. Code and pretrained models are available at https://archaeoscape.ai/work/relay-tokens/ .