Representing Neural Network Layers as Linear Operations via Koopman Operator Theory

TL;DR

This paper introduces a novel approach to linearize neural network layers using Koopman operator theory, enabling better understanding and control of neural networks by representing them as dynamical systems.

Contribution

It applies Koopman operator theory and dynamic mode decomposition to neural networks, providing a new linear perspective and practical layer replacement methods.

Findings

Successfully linearized neural network layers with Koopman theory.

Replaced MLP layers with DMD models achieving high accuracy.

Demonstrated the approach on Yin-Yang and MNIST datasets.

Abstract

The strong performance of simple neural networks is often attributed to their nonlinear activations. However, a linear view of neural networks makes understanding and controlling networks much more approachable. We draw from a dynamical systems view of neural networks, offering a fresh perspective by using Koopman operator theory and its connections with dynamic mode decomposition (DMD). Together, they offer a framework for linearizing dynamical systems by embedding the system into an appropriate observable space. By reframing a neural network as a dynamical system, we demonstrate that we can replace the nonlinear layer in a pretrained multi-layer perceptron (MLP) with a finite-dimensional linear operator. In addition, we analyze the eigenvalues of DMD and the right singular vectors of SVD, to present evidence that time-delayed coordinates provide a straightforward and highly effective observable space for Koopman theory to linearize a network layer. Consequently, we replace layers of an MLP trained on the Yin-Yang dataset with predictions from a DMD model, achieving a mdoel accuracy of up to 97.3%, compared to the original 98.4%. In addition, we replace layers in an MLP trained on the MNIST dataset, achieving up to 95.8%, compared to the original 97.2% on the test set.