Polyharmonic Cascade

TL;DR

The paper introduces the polyharmonic cascade, a deep learning architecture based on polyharmonic splines, enabling smooth, probabilistic function approximation with an efficient, non-gradient training method demonstrated on MNIST.

Contribution

It proposes a novel deep learning model using polyharmonic splines with a new training approach based on solving linear systems, avoiding gradient descent.

Findings

Effective approximation of nonlinear functions with smoothness and probabilistic interpretation.

Fast training method scalable to GPU with good performance on MNIST.

Preserves theoretical consistency and avoids overfitting.

Abstract

This paper presents a deep machine learning architecture, the "polyharmonic cascade" -- a sequence of packages of polyharmonic splines, where each layer is rigorously derived from the theory of random functions and the principles of indifference. This makes it possible to approximate nonlinear functions of arbitrary complexity while preserving global smoothness and a probabilistic interpretation. For the polyharmonic cascade, a training method alternative to gradient descent is proposed: instead of directly optimizing the coefficients, one solves a single global linear system on each batch with respect to the function values at fixed "constellations" of nodes. This yields synchronized updates of all layers, preserves the probabilistic interpretation of individual layers and theoretical consistency with the original model, and scales well: all computations reduce to 2D matrix operations efficiently executed on a GPU. Fast learning without overfitting on MNIST is demonstrated.