Improved cross-validation for classifiers that make algorithmic choices to minimise runtime without compromising output correctness

TL;DR

This paper enhances machine learning-based variable ordering selection in computer algebra systems by modifying cross-validation methods to better distinguish non-optimal choices, thereby reducing runtime without affecting correctness.

Contribution

It introduces a modified cross-validation approach for classifiers that improves runtime minimization in algorithmic choices without compromising output correctness.

Findings

Improved classifier performance in variable ordering tasks.

Enhanced differentiation between non-optimal orderings.

Reduced computational runtime in experiments.

Abstract

Our topic is the use of machine learning to improve software by making choices which do not compromise the correctness of the output, but do affect the time taken to produce such output. We are particularly concerned with computer algebra systems (CASs), and in particular, our experiments are for selecting the variable ordering to use when performing a cylindrical algebraic decomposition of $n$-dimensional real space with respect to the signs of a set of polynomials. In our prior work we explored the different ML models that could be used, and how to identify suitable features of the input polynomials. In the present paper we both repeat our prior experiments on problems which have more variables (and thus exponentially more possible orderings), and examine the metric which our ML classifiers targets. The natural metric is computational runtime, with classifiers trained to pick the ordering which minimises this. However, this leads to the situation were models do not distinguish between any of the non-optimal orderings, whose runtimes may still vary dramatically. In this paper we investigate a modification to the cross-validation algorithms of the classifiers so that they do distinguish these cases, leading to improved results.