Optimally rotated coordinate systems for adaptive least-squares regression on sparse grids

TL;DR

This paper introduces a preprocessing method that finds an optimal rotated coordinate system to improve adaptive sparse grid least squares regression, especially for skewed or rotated data, enhancing performance on high-dimensional problems.

Contribution

It proposes a novel preprocessing approach to optimize coordinate systems for adaptive sparse grid regression, addressing issues with skewed and rotated data.

Findings

Improved regression accuracy on synthetic and real-world data.

Enhanced effectiveness of sparse grid methods for rotated datasets.

Reduction in effective dimensionality through optimal coordinate rotation.

Abstract

For low-dimensional data sets with a large amount of data points, standard kernel methods are usually not feasible for regression anymore. Besides simple linear models or involved heuristic deep learning models, grid-based discretizations of larger (kernel) model classes lead to algorithms, which naturally scale linearly in the amount of data points. For moderate-dimensional or high-dimensional regression tasks, these grid-based discretizations suffer from the curse of dimensionality. Here, sparse grid methods have proven to circumvent this problem to a large extent. In this context, space- and dimension-adaptive sparse grids, which can detect and exploit a given low effective dimensionality of nominally high-dimensional data, are particularly successful. They nevertheless rely on an axis-aligned structure of the solution and exhibit issues for data with predominantly skewed and rotated coordinates. In this paper we propose a preprocessing approach for these adaptive sparse grid algorithms that determines an optimized, problem-dependent coordinate system and, thus, reduces the effective dimensionality of a given data set in the ANOVA sense. We provide numerical examples on synthetic data as well as real-world data to show how an adaptive sparse grid least squares algorithm benefits from our preprocessing method.