Saturating Splines and Feature Selection

TL;DR

This paper introduces a novel method for fitting saturating splines using convex optimization, enabling simultaneous feature selection and nonlinear modeling without pre-set knot locations.

Contribution

It extends adaptive regression splines with saturation, providing an efficient algorithm that handles infinite-dimensional optimization and adapts to generalized additive models.

Findings

Efficient algorithm for saturating spline fitting without pre-specified knots

Model performs feature selection and nonlinear fitting simultaneously

Extension to higher order splines and different outside-range constraints

Abstract

We extend the adaptive regression spline model by incorporating saturation, the natural requirement that a function extend as a constant outside a certain range. We fit saturating splines to data using a convex optimization problem over a space of measures, which we solve using an efficient algorithm based on the conditional gradient method. Unlike many existing approaches, our algorithm solves the original infinite-dimensional (for splines of degree at least two) optimization problem without pre-specified knot locations. We then adapt our algorithm to fit generalized additive models with saturating splines as coordinate functions and show that the saturation requirement allows our model to simultaneously perform feature selection and nonlinear function fitting. Finally, we briefly sketch how the method can be extended to higher order splines and to different requirements on the extension outside the data range.