Credit Assignment for Trained Neural Networks Based on Koopman Operator Theory

TL;DR

This paper introduces a novel method for credit assignment in trained neural networks using Koopman operator theory, providing a linear dynamics perspective to evaluate component contributions post-training.

Contribution

It applies Koopman operator theory to trained neural networks, offering a new approach for credit assignment beyond the training phase.

Findings

Effective credit assignment demonstrated on typical neural networks.

The method provides algebraically interpretable metrics for component contribution.

Experimental results validate the approach's effectiveness.

Abstract

Credit assignment problem of neural networks refers to evaluating the credit of each network component to the final outputs. For an untrained neural network, approaches to tackling it have made great contributions to parameter update and model revolution during the training phase. This problem on trained neural networks receives rare attention, nevertheless, it plays an increasingly important role in neural network patch, specification and verification. Based on Koopman operator theory, this paper presents an alternative perspective of linear dynamics on dealing with the credit assignment problem for trained neural networks. Regarding a neural network as the composition of sub-dynamics series, we utilize step-delay embedding to capture snapshots of each component, characterizing the established mapping as exactly as possible. To circumvent the dimension-difference problem encountered during the embedding, a composition and decomposition of an auxiliary linear layer, termed minimal linear dimension alignment, is carefully designed with rigorous formal guarantee. Afterwards, each component is approximated by a Koopman operator and we derive the Jacobian matrix and its corresponding determinant, similar to backward propagation. Then, we can define a metric with algebraic interpretability for the credit assignment of each network component. Moreover, experiments conducted on typical neural networks demonstrate the effectiveness of the proposed method.