Tensor-based Multi-layer Decoupling

TL;DR

This paper introduces a tensor-based framework for multi-layer decoupling of multivariate functions, extending CPD methods to more complex models with theoretical and practical validation.

Contribution

It develops a novel tensor decomposition approach for multi-layer decoupling, including a structured factorization and bilevel optimization, with theoretical justification and empirical demonstrations.

Findings

Effective on synthetic systems and benchmarks

Enables neural network compression

Balances first- and zeroth-order information adaptively

Abstract

The decoupling of multivariate functions is a powerful modeling paradigm for learning multivariate input-output relations from data. For the single-layer case, established CPD-based methods are available, but the multi-layer case remained largely unexplored. This work introduces a tensor-based framework for multi-layer decoupling, which is based on ParaTuck-type tensor decompositions and constrained optimization. We provide theoretical justification behind the considered tensor decompositions and parameterizations. Furthermore, we formulate a structured coupled matrix-tensor factorization that incorporates both Jacobian and function evaluations, together with a bilevel optimization approach for adaptively balancing first- and zeroth-order information. The feasibility of the proposed methodology is illustrated on synthetic systems, a nonlinear system identification benchmark and neural network compression.