Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy

TL;DR

This paper investigates how different sampling strategies and sample sizes affect the accuracy of feature approximation in exploratory landscape analysis, revealing that sampling method impacts feature values and classifier performance.

Contribution

It demonstrates that sampling strategy influences feature value accuracy and classifier effectiveness, highlighting the importance of consistent sampling methods in ELA applications.

Findings

Feature approximations vary significantly across sampling strategies.

Increasing sample size improves robustness but does not ensure convergence.

Sobol' sequence sampling yields higher classifier accuracy.

Abstract

Exploratory landscape analysis (ELA) supports supervised learning approaches for automated algorithm selection and configuration by providing sets of features that quantify the most relevant characteristics of the optimization problem at hand. In black-box optimization, where an explicit problem representation is not available, the feature values need to be approximated from a small number of sample points. In practice, uniformly sampled random point sets and Latin hypercube constructions are commonly used sampling strategies. In this work, we analyze how the sampling method and the sample size influence the quality of the feature value approximations and how this quality impacts the accuracy of a standard classification task. While, not unexpectedly, increasing the number of sample points gives more robust estimates for the feature values, to our surprise we find that the feature value approximations for different sampling strategies do not converge to the same value. This implies that approximated feature values cannot be interpreted independently of the underlying sampling strategy. As our classification experiments show, this also implies that the feature approximations used for training a classifier must stem from the same sampling strategy as those used for the actual classification tasks. As a side result we show that classifiers trained with feature values approximated by Sobol' sequences achieve higher accuracy than any of the standard sampling techniques. This may indicate improvement potential for ELA-trained machine learning models.