Detecting Cloud Presence in Satellite Images Using the RGB-based CLIP Vision-Language Model

TL;DR

This paper investigates the use of the pre-trained CLIP vision-language model for detecting cloud presence in satellite images, evaluating zero-shot and fine-tuning methods across different datasets and sensors.

Contribution

It demonstrates the effectiveness of CLIP representations for cloud detection and explores transferability and fine-tuning strategies for satellite imagery analysis.

Findings

CLIP achieves non-trivial cloud detection performance.

Fine-tuning significantly improves true negative rate.

Methods generalize across different satellite sensors.

Abstract

This work explores capabilities of the pre-trained CLIP vision-language model to identify satellite images affected by clouds. Several approaches to using the model to perform cloud presence detection are proposed and evaluated, including a purely zero-shot operation with text prompts and several fine-tuning approaches. Furthermore, the transferability of the methods across different datasets and sensor types (Sentinel-2 and Landsat-8) is tested. The results that CLIP can achieve non-trivial performance on the cloud presence detection task with apparent capability to generalise across sensing modalities and sensing bands. It is also found that a low-cost fine-tuning stage leads to a strong increase in true negative rate. The results demonstrate that the representations learned by the CLIP model can be useful for satellite image processing tasks involving clouds.