Generalization Ability of Feature-based Performance Prediction Models: A Statistical Analysis across Benchmarks

TL;DR

This paper investigates how well feature-based performance prediction models generalize across different benchmark suites, finding that statistical similarity between training and testing problem collections correlates with better model generalization.

Contribution

It provides a statistical analysis linking the similarity of problem distributions to the generalization performance of prediction models across benchmarks.

Findings

Positive correlation between statistical similarity and model accuracy

Models generalize better when feature distributions are statistically similar

Validation across multiple benchmark collections confirms the findings

Abstract

This study examines the generalization ability of algorithm performance prediction models across various benchmark suites. Comparing the statistical similarity between the problem collections with the accuracy of performance prediction models that are based on exploratory landscape analysis features, we observe that there is a positive correlation between these two measures. Specifically, when the high-dimensional feature value distributions between training and testing suites lack statistical significance, the model tends to generalize well, in the sense that the testing errors are in the same range as the training errors. Two experiments validate these findings: one involving the standard benchmark suites, the BBOB and CEC collections, and another using five collections of affine combinations of BBOB problem instances.