Deep learning with transfer functions: new applications in system identification

TL;DR

This paper introduces a linear dynamical operator with transfer functions for deep learning, enabling end-to-end training in system identification tasks including prediction error methods and quantized data modeling.

Contribution

It presents a novel transfer function-based operator with back-propagation for structured neural networks in system identification, integrating traditional methods with deep learning.

Findings

Effective in system identification benchmarks

Enables end-to-end training with transfer functions

Improves accuracy in quantized data modeling

Abstract

This paper presents a linear dynamical operator described in terms of a rational transfer function, endowed with a well-defined and efficient back-propagation behavior for automatic derivatives computation. The operator enables end-to-end training of structured networks containing linear transfer functions and other differentiable units {by} exploiting standard deep learning software. Two relevant applications of the operator in system identification are presented. The first one consists in the integration of {prediction error methods} in deep learning. The dynamical operator is included as {the} last layer of a neural network in order to obtain the optimal one-step-ahead prediction error. The second one considers identification of general block-oriented models from quantized data. These block-oriented models are constructed by combining linear dynamical operators with static nonlinearities described as standard feed-forward neural networks. A custom loss function corresponding to the log-likelihood of quantized output observations is defined. For gradient-based optimization, the derivatives of the log-likelihood are computed by applying the back-propagation algorithm through the whole network. Two system identification benchmarks are used to show the effectiveness of the proposed methodologies.