Quantifying the synthetic and real domain gap in aerial scene understanding

TL;DR

This paper introduces a new methodology to quantify the perceptual and structural differences between synthetic and real aerial imagery, highlighting the domain gap's impact on model performance and adaptation.

Contribution

It proposes a novel Multi-Model Consensus Metric and depth-based structural metrics for assessing scene complexity and domain gaps in aerial imagery.

Findings

Real-world scenes have higher model consensus than synthetic scenes.

Synthetic scenes exhibit greater variability, challenging model adaptability.

The study emphasizes the need for improved simulation fidelity and model generalization.

Abstract

Quantifying the gap between synthetic and real-world imagery is essential for improving both transformer-based models - that rely on large volumes of data - and datasets, especially in underexplored domains like aerial scene understanding where the potential impact is significant. This paper introduces a novel methodology for scene complexity assessment using Multi-Model Consensus Metric (MMCM) and depth-based structural metrics, enabling a robust evaluation of perceptual and structural disparities between domains. Our experimental analysis, utilizing real-world (Dronescapes) and synthetic (Skyscenes) datasets, demonstrates that real-world scenes generally exhibit higher consensus among state-of-the-art vision transformers, while synthetic scenes show greater variability and challenge model adaptability. The results underline the inherent complexities and domain gaps, emphasizing the need for enhanced simulation fidelity and model generalization. This work provides critical insights into the interplay between domain characteristics and model performance, offering a pathway for improved domain adaptation strategies in aerial scene understanding.