OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

TL;DR

This paper demonstrates the successful pretraining of billion-parameter foundation models for high-resolution satellite imagery using exascale computing resources, highlighting the importance of data scaling and providing best practices for the geospatial community.

Contribution

It introduces a scalable pretraining approach for large vision transformers on satellite data, utilizing high-performance computing and a novel dataset, with insights into technical challenges and model performance.

Findings

Data scaling is crucial for effective model scaling.

Pretrained models improve performance on classification, segmentation, detection.

Technical challenges in training large models are discussed.

Abstract

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.