On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction

TL;DR

This paper introduces DAV, a method based on adversarial validation, to quantify dissimilarity in geospatial data, and analyzes how this dissimilarity impacts the effectiveness of different cross-validation strategies.

Contribution

The paper proposes DAV, a novel adversarial validation-based approach to measure feature space dissimilarity in geospatial data, and evaluates its impact on cross-validation accuracy.

Findings

DAV effectively quantifies dissimilarity across its entire range.

Dissimilarity influences the accuracy of CV methods, with geospatial CV outperforming RDM-CV at high dissimilarity levels.

No CV method is accurate when dissimilarity exceeds 90%.

Abstract

Recent geospatial machine learning studies have shown that the results of model evaluation via cross-validation (CV) are strongly affected by the dissimilarity between the sample data and the prediction locations. In this paper, we propose a method to quantify such a dissimilarity in the interval 0 to 100% and from the perspective of the data feature space. The proposed method is based on adversarial validation, which is an approach that can check whether sample data and prediction locations can be separated with a binary classifier. The proposed method is called dissimilarity quantification by adversarial validation (DAV). To study the effectiveness and general?ity of DAV, we tested it on a series of experiments based on both synthetic and real datasets and with gradually increasing dissimilarities. Results show that DAV effectively quantified dissimilarity across the entire range of values. Next to this, we studied how dissimilarity affects CV methods' evaluations by comparing the results of random CV method (RDM-CV) and of two geospatial CV methods, namely, block and spatial+ CV (BLK-CV and SP-CV). Our results showed the evaluations follow similar patterns in all datasets and predictions: when dissimilarity is low (usually lower than 30%), RDM-CV provides the most accurate evaluation results. As dissimilarity increases, geospatial CV methods, especially SP-CV, become more and more accurate and even outperform RDM-CV. When dissimilarity is high (>=90%), no CV method provides accurate evaluations. These results show the importance of considering feature space dissimilarity when working with geospatial machine learning predictions and can help researchers and practitioners to select more suitable CV methods for evaluating their predictions.