Transferring Physical Priors into Remote Sensing Segmentation via Large Language Models

TL;DR

This paper introduces PriorSeg, a physics-aware segmentation model that integrates physical priors from a large language model-driven knowledge graph, improving remote sensing segmentation accuracy without retraining foundation models.

Contribution

The paper presents a novel paradigm for incorporating physical priors into segmentation models using a knowledge graph and a physics-aware residual refinement approach.

Findings

PriorSeg enhances segmentation accuracy and physical plausibility.

The PCKG and Phy-Sky-SA datasets are effective for integrating physical priors.

The physics-consistency loss improves model performance without retraining foundation models.

Abstract

Semantic segmentation of remote sensing imagery is fundamental to Earth observation. Achieving accurate results requires integrating not only optical images but also physical variables such as the Digital Elevation Model (DEM), Synthetic Aperture Radar (SAR) and Normalized Difference Vegetation Index (NDVI). Recent foundation models (FMs) leverage pre-training to exploit these variables but still depend on spatially aligned data and costly retraining when involving new sensors. To overcome these limitations, we introduce a novel paradigm for integrating domain-specific physical priors into segmentation models. We first construct a Physical-Centric Knowledge Graph (PCKG) by prompting large language models to extract physical priors from 1,763 vocabularies, and use it to build a heterogeneous, spatial-aligned dataset, Phy-Sky-SA. Building on this foundation, we develop PriorSeg, a physics-aware residual refinement model trained with a joint visual-physical strategy that incorporates a novel physics-consistency loss. Experiments on heterogeneous settings demonstrate that PriorSeg improves segmentation accuracy and physical plausibility without retraining the FMs. Ablation studies verify the effectiveness of the Phy-Sky-SA dataset, the PCKG, and the physics-consistency loss.