Transformers For Recognition In Overhead Imagery: A Reality Check

TL;DR

This study systematically evaluates the effectiveness of transformer-based models in overhead imagery recognition, finding modest improvements mainly in hybrid models, and questioning the benefits of fully transformer-based approaches.

Contribution

It provides a comprehensive empirical comparison of transformer-enhanced segmentation models for overhead imagery, highlighting the limited advantages of fully transformer models.

Findings

Transformers offer modest performance gains in hybrid models.

Fully transformer-based models perform relatively poorly.

Hybrid models outperform fully transformer models on benchmark datasets.

Abstract

There is evidence that transformers offer state-of-the-art recognition performance on tasks involving overhead imagery (e.g., satellite imagery). However, it is difficult to make unbiased empirical comparisons between competing deep learning models, making it unclear whether, and to what extent, transformer-based models are beneficial. In this paper we systematically compare the impact of adding transformer structures into state-of-the-art segmentation models for overhead imagery. Each model is given a similar budget of free parameters, and their hyperparameters are optimized using Bayesian Optimization with a fixed quantity of data and computation time. We conduct our experiments with a large and diverse dataset comprising two large public benchmarks: Inria and DeepGlobe. We perform additional ablation studies to explore the impact of specific transformer-based modeling choices. Our results suggest that transformers provide consistent, but modest, performance improvements. We only observe this advantage however in hybrid models that combine convolutional and transformer-based structures, while fully transformer-based models achieve relatively poor performance.