Locally Adaptive and Differentiable Regression

TL;DR

This paper introduces a framework for creating globally continuous and differentiable regression models by combining locally learned models, improving performance and convergence in complex data scenarios.

Contribution

It proposes a novel method to ensure continuity and differentiability in over-parameterized, locally adaptive models, enhancing their theoretical and practical performance.

Findings

Achieves faster statistical convergence.

Improves practical performance on diverse data.

Ensures global continuity and differentiability.

Abstract

Over-parameterized models like deep nets and random forests have become very popular in machine learning. However, the natural goals of continuity and differentiability, common in regression models, are now often ignored in modern overparametrized, locally-adaptive models. We propose a general framework to construct a global continuous and differentiable model based on a weighted average of locally learned models in corresponding local regions. This model is competitive in dealing with data with different densities or scales of function values in different local regions. We demonstrate that when we mix kernel ridge and polynomial regression terms in the local models, and stitch them together continuously, we achieve faster statistical convergence in theory and improved performance in various practical settings.