Efficient Regression in Metric Spaces via Approximate Lipschitz Extension

TL;DR

This paper introduces an efficient regression method for metric spaces using approximate Lipschitz extensions, balancing speed and accuracy by leveraging the data's intrinsic dimension.

Contribution

The paper proposes a novel regression algorithm that adapts to the data's intrinsic dimension, significantly improving efficiency over naive convex programming approaches.

Findings

Finite-sample risk bounds with minimal assumptions

Algorithm's runtime depends on data's intrinsic dimension

Achieves a speed-precision tradeoff

Abstract

We present a framework for performing efficient regression in general metric spaces. Roughly speaking, our regressor predicts the value at a new point by computing a Lipschitz extension --- the smoothest function consistent with the observed data --- after performing structural risk minimization to avoid overfitting. We obtain finite-sample risk bounds with minimal structural and noise assumptions, and a natural speed-precision tradeoff. The offline (learning) and online (prediction) stages can be solved by convex programming, but this naive approach has runtime complexity $O(n^3)$, which is prohibitive for large datasets. We design instead a regression algorithm whose speed and generalization performance depend on the intrinsic dimension of the data, to which the algorithm adapts. While our main innovation is algorithmic, the statistical results may also be of independent interest.