Extraction of linearized models from pre-trained networks via knowledge distillation

TL;DR

This paper introduces a method to derive linear models from pre-trained neural networks using Koopman operator theory and knowledge distillation, improving classification accuracy and stability on MNIST datasets.

Contribution

It presents a novel framework combining Koopman theory and knowledge distillation to extract linearized models from neural networks for classification.

Findings

The proposed model outperforms least-squares Koopman approximation in accuracy.

It demonstrates improved numerical stability over traditional methods.

Effective on MNIST and Fashion-MNIST datasets.

Abstract

Recent developments in hardware, such as photonic integrated circuits and optical devices, are driving demand for research on constructing machine learning architectures tailored for linear operations. Hence, it is valuable to explore methods for constructing learning machines with only linear operations after simple nonlinear preprocessing. In this study, we propose a framework to extract a linearized model from a pre-trained neural network for classification tasks by integrating Koopman operator theory with knowledge distillation. Numerical demonstrations on the MNIST and the Fashion-MNIST datasets reveal that the proposed model consistently outperforms the conventional least-squares-based Koopman approximation in both classification accuracy and numerical stability.