Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation

TL;DR

The paper introduces DOFA, a neural plasticity-inspired multimodal foundation model for Earth Observation that flexibly processes diverse satellite sensor data, achieving state-of-the-art results with efficient training.

Contribution

It presents a novel dynamic hypernetwork approach enabling a single model to handle multiple EO sensor modalities, improving generalization and efficiency.

Findings

State-of-the-art performance on multiple EO tasks

Effective generalization to unseen sensor modalities

Reduced computational resources with hybrid continual pretraining

Abstract

Earth observation (EO) in open-world settings presents a unique challenge: different applications rely on diverse sensor modalities, each with varying ground sampling distances, spectral ranges, and numbers of spectral bands. However, existing EO foundation models are typically tailored to specific sensor types, making them inflexible when generalizing across the heterogeneous landscape of EO data. To address this, we propose the Dynamic One-For-All (DOFA) model, a unified, multimodal foundation framework designed for diverse vision tasks in EO. Inspired by neural plasticity, DOFA utilizes a wavelength-conditioned dynamic hypernetwork to process inputs from five distinct satellite sensors flexibly. By continually pretraining on five EO modalities, DOFA achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities. Enhanced with hybrid continual pretraining, DOFA+ requires significantly fewer computational resources while outperforming counterparts trained with extensive GPU budgets. Experiments on diverse datasets highlight DOFA's potential as a foundation for general-purpose vision models in the sensor-diverse EO domain. The code and pre-trained weights are publicly available at https://github.com/zhu-xlab/DOFA.