Enabling Near-realtime Remote Sensing via Satellite-Ground Collaboration of Large Vision-Language Models

TL;DR

Grace enables near-realtime remote sensing by deploying compact vision-language models on satellites and larger models on ground stations, using asynchronous retrieval and confidence-based task offloading to significantly reduce latency.

Contribution

The paper introduces Grace, a novel satellite-ground collaborative system that combines adaptive knowledge updates and confidence-based task dispatching for efficient LVLM inference in remote sensing.

Findings

Reduces average latency by 76-95% compared to existing methods.

Maintains inference accuracy while enabling near-realtime processing.

Demonstrates effectiveness with real-world satellite orbital data.

Abstract

Large vision-language models (LVLMs) have recently demonstrated great potential in remote sensing (RS) tasks (e.g., disaster monitoring) conducted by low Earth orbit (LEO) satellites. However, their deployment in real-world LEO satellite systems remains largely unexplored, hindered by limited onboard computing resources and brief satellite-ground contacts. We propose Grace, a satellite-ground collaborative system designed for near-realtime LVLM inference in RS tasks. Accordingly, we deploy compact LVLM on satellites for realtime inference, but larger ones on ground stations (GSs) to guarantee end-to-end performance. Grace is comprised of two main phases that are asynchronous satellite-GS Retrieval-Augmented Generation (RAG), and a task dispatch algorithm. Firstly, we still the knowledge archive of GS RAG to satellite archive with tailored adaptive update algorithm during limited satellite-ground data exchange period. Secondly, propose a confidence-based test algorithm that either processes the task onboard the satellite or offloads it to the GS. Extensive experiments based on real-world satellite orbital data show that Grace reduces the average latency by 76-95% compared to state-of-the-art methods, without compromising inference accuracy.