AGI for the Earth, the path, possibilities and how to evaluate intelligence of models that work with Earth Observation Data?

TL;DR

This paper discusses the potential of AGI in understanding Earth Observation data, reviews current benchmarks, highlights their limitations, and proposes a comprehensive new benchmark to evaluate models' Earth understanding capabilities.

Contribution

It introduces the need for a new comprehensive benchmark for Earth Observation data and proposes a set of tasks to better evaluate model generalization in this domain.

Findings

Current benchmarks are limited in evaluating Earth Observation models.

Earth Observation data is crucial for advancing AGI understanding of the natural world.

A proposed set of tasks aims to improve model assessment in Earth Observation applications.

Abstract

Artificial General Intelligence (AGI) is closer than ever to becoming a reality, sparking widespread enthusiasm in the research community to collect and work with various modalities, including text, image, video, and audio. Despite recent efforts, satellite spectral imagery, as an additional modality, has yet to receive the attention it deserves. This area presents unique challenges, but also holds great promise in advancing the capabilities of AGI in understanding the natural world. In this paper, we argue why Earth Observation data is useful for an intelligent model, and then we review existing benchmarks and highlight their limitations in evaluating the generalization ability of foundation models in this domain. This paper emphasizes the need for a more comprehensive benchmark to evaluate earth observation models. To facilitate this, we propose a comprehensive set of tasks that a benchmark should encompass to effectively assess a model's ability to understand and interact with Earth observation data.