KASAM: Spline Additive Models for Function Approximation

TL;DR

This paper introduces KASAM, a novel spline-based additive model that enhances function approximation and memory retention, addressing catastrophic forgetting in continual learning, with empirical validation of its capabilities and limitations.

Contribution

The paper proposes KASAM, extending SAM with the Kolmogorov-Arnold theorem to create a universal function approximator with improved memory retention.

Findings

SAM shows robust memory retention with some interference.

KASAM is more susceptible to catastrophic forgetting.

KASAM with pseudo-rehearsal improves regression performance and memory retention.

Abstract

Neural networks have been criticised for their inability to perform continual learning due to catastrophic forgetting and rapid unlearning of a past concept when a new concept is introduced. Catastrophic forgetting can be alleviated by specifically designed models and training techniques. This paper outlines a novel Spline Additive Model (SAM). SAM exhibits intrinsic memory retention with sufficient expressive power for many practical tasks, but is not a universal function approximator. SAM is extended with the Kolmogorov-Arnold representation theorem to a novel universal function approximator, called the Kolmogorov-Arnold Spline Additive Model - KASAM. The memory retention, expressive power and limitations of SAM and KASAM are illustrated analytically and empirically. SAM exhibited robust but imperfect memory retention, with small regions of overlapping interference in sequential learning tasks. KASAM exhibited greater susceptibility to catastrophic forgetting. KASAM in combination with pseudo-rehearsal training techniques exhibited superior performance in regression tasks and memory retention.