RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts

TL;DR

RSUniVLM is a comprehensive vision-language model for remote sensing that effectively handles multi-granularity tasks, multi-image analysis, and diverse RS applications with a novel architecture and large-scale dataset.

Contribution

The paper introduces RSUniVLM, a unified end-to-end remote sensing vision-language model with a novel granularity-oriented mixture of experts architecture and a large-scale instruction-following dataset.

Findings

Achieves state-of-the-art performance on various RS tasks.

Effectively handles multi-image analysis like change detection.

Maintains a compact model size of about 1 billion parameters.

Abstract

Remote Sensing Vision-Language Models (RS VLMs) have made much progress in the tasks of remote sensing (RS) image comprehension. While performing well in multi-modal reasoning and multi-turn conversations, the existing models lack pixel-level understanding and struggle with multi-image inputs. In this work, we propose RSUniVLM, a unified, end-to-end RS VLM designed for comprehensive vision understanding across multiple granularity, including image-level, region-level, and pixel-level tasks. RSUniVLM also performs effectively in multi-image analysis, with instances of change detection and change captioning. To enhance the model's ability to capture visual information at different levels without increasing model size, we design a novel architecture called Granularity-oriented Mixture of Experts to constraint the model to about 1 billion parameters. We also construct a large-scale RS instruction-following dataset based on a variety of existing datasets in both RS and general domain, encompassing various tasks such as object localization, visual question answering, and semantic segmentation. Substantial experiments have been conducted to validate the superiority of the proposed RSUniVLM up to state-of-the-art across various RS tasks. Code and model will be available at \href{https://github.com/xuliu-cyber/RSUniVLM}{here}.