Nonparametric Teaching of Implicit Neural Representations

TL;DR

This paper introduces Implicit Neural Teaching, a nonparametric framework for efficiently training implicit neural representations by selecting signal fragments, resulting in over 30% faster training times across modalities.

Contribution

It proposes a novel nonparametric teaching paradigm for INR training, linking MLP evolution with nonparametric function teaching, and demonstrates significant efficiency improvements.

Findings

Achieves over 30% reduction in training time.

Establishes a theoretical connection between MLP training and nonparametric teaching.

Demonstrates effectiveness across various input modalities.

Abstract

We investigate the learning of implicit neural representation (INR) using an overparameterized multilayer perceptron (MLP) via a novel nonparametric teaching perspective. The latter offers an efficient example selection framework for teaching nonparametrically defined (viz. non-closed-form) target functions, such as image functions defined by 2D grids of pixels. To address the costly training of INRs, we propose a paradigm called Implicit Neural Teaching (INT) that treats INR learning as a nonparametric teaching problem, where the given signal being fitted serves as the target function. The teacher then selects signal fragments for iterative training of the MLP to achieve fast convergence. By establishing a connection between MLP evolution through parameter-based gradient descent and that of function evolution through functional gradient descent in nonparametric teaching, we show for the first time that teaching an overparameterized MLP is consistent with teaching a nonparametric learner. This new discovery readily permits a convenient drop-in of nonparametric teaching algorithms to broadly enhance INR training efficiency, demonstrating 30%+ training time savings across various input modalities.