Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

TL;DR

This paper introduces a manifold-constrained optimization method called DLRT to effectively compress large vision transformers for geospatial transfer learning, enabling efficient on-device deployment without significant accuracy loss.

Contribution

The work presents a novel optimization framework that enforces structured low-dimensional parameterizations during transfer learning, outperforming existing low-rank compression methods.

Findings

Significant parameter reduction achieved

Minimal accuracy loss on geospatial benchmarks

Enables high-performance on-device models

Abstract

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption. In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer-based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.