Differentiable Spline Approximations

TL;DR

This paper introduces a new method to extend differentiable programming to spline-based functions, enabling gradient-based optimization for a broad class of models beyond traditional differentiable functions.

Contribution

It derives the Jacobian for spline functions, revealing a block-sparse structure that allows efficient implicit computation and integration into differentiable models.

Findings

Improved performance in image segmentation tasks

Enhanced 3D point cloud reconstruction accuracy

Effective application in finite element analysis

Abstract

The paradigm of differentiable programming has significantly enhanced the scope of machine learning via the judicious use of gradient-based optimization. However, standard differentiable programming methods (such as autodiff) typically require that the machine learning models be differentiable, limiting their applicability. Our goal in this paper is to use a new, principled approach to extend gradient-based optimization to functions well modeled by splines, which encompass a large family of piecewise polynomial models. We derive the form of the (weak) Jacobian of such functions and show that it exhibits a block-sparse structure that can be computed implicitly and efficiently. Overall, we show that leveraging this redesigned Jacobian in the form of a differentiable "layer" in predictive models leads to improved performance in diverse applications such as image segmentation, 3D point cloud reconstruction, and finite element analysis.