How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

TL;DR

This paper investigates the redundancy in remote sensing foundation models, revealing they are more overparameterized than computer vision models, and demonstrates that slimmability can be used for efficient deployment and understanding of these models.

Contribution

The study provides empirical evidence that remote sensing models are highly redundant and introduces post-hoc slimmability as a practical tool for model compression and analysis.

Findings

RS FMs retain over 71% accuracy at 1% FLOPs after slimming

RS FMs are more overparameterized than CV models at smaller scales

Learned slimmable training improves model performance

Abstract

Large-scale foundation models (FMs) in remote sensing (RS) are developed based on the paradigms established in computer vision (CV) and have shown promise for various Earth observation applications. However, the direct transfer of scaling assumptions from CV to RS has not been adequately examined. We hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, where increasing parameter count primarily induces redundant representations rather than qualitatively new abstractions. To test this hypothesis, we use post-hoc slimming, where we uniformly reduce the width of pretrained encoder, as a tool to measure representational redundancy across six state-of-the-art RS FMs on four downstream classification tasks. Our findings reveal a significant contrast with those in the CV domain: while a post-hoc slimmed masked autoencoder (MAE) trained on ImageNet retains less than 10% accuracy at 1% FLOPs, RS FMs maintain over 71% relative accuracy at the same budget. This sevenfold difference provides strong empirical support for our hypothesis. We further demonstrate that learned slimmable training can improve both Momentum Contrast (MoCo)- and MAE- based models. In addition, through the explained variance ratio and the feature correlation analysis, we provide mechanistic explanations showing that RS FMs distribute task-relevant information with high redundancy. Our findings establish post-hoc slimmability as both a practical deployment strategy for resource-constrained environments and a diagnostic tool that challenges the prevailing scaling paradigm in RS. Upon acceptance, we will publish all code.