Kernel Regression with Sparse Metric Learning

TL;DR

This paper introduces a novel kernel regression method that incorporates sparse metric learning, enabling simultaneous dimensionality reduction and improved prediction accuracy, validated through extensive experiments on diverse datasets and traffic flow forecasting.

Contribution

It is the first to combine kernel regression with sparse metric learning, using a mixed (2,1)-norm regularization to learn a Mahalanobis distance metric for better predictions.

Findings

Outperforms existing kernel regression methods on 19 datasets.

Effectively reduces dimensionality while maintaining prediction accuracy.

Demonstrates practical utility in short-term traffic flow forecasting.

Abstract

Kernel regression is a popular non-parametric fitting technique. It aims at learning a function which estimates the targets for test inputs as precise as possible. Generally, the function value for a test input is estimated by a weighted average of the surrounding training examples. The weights are typically computed by a distance-based kernel function and they strongly depend on the distances between examples. In this paper, we first review the latest developments of sparse metric learning and kernel regression. Then a novel kernel regression method involving sparse metric learning, which is called kernel regression with sparse metric learning (KR$\_$SML), is proposed. The sparse kernel regression model is established by enforcing a mixed $(2,1)$-norm regularization over the metric matrix. It learns a Mahalanobis distance metric by a gradient descent procedure, which can simultaneously conduct dimensionality reduction and lead to good prediction results. Our work is the first to combine kernel regression with sparse metric learning. To verify the effectiveness of the proposed method, it is evaluated on 19 data sets for regression. Furthermore, the new method is also applied to solving practical problems of forecasting short-term traffic flows. In the end, we compare the proposed method with other three related kernel regression methods on all test data sets under two criterions. Experimental results show that the proposed method is much more competitive.