Error Autocorrelation Objective Function for Improved System Modeling

TL;DR

This paper introduces a new whitening cost function based on the Ljung-Box statistic for deep learning models, which reduces error correlations to improve generalization and extrapolation in system modeling tasks.

Contribution

The paper proposes a novel whitening objective function that minimizes error autocorrelations, enhancing model generalization beyond traditional MSE-based training.

Findings

Significant improvement in generalization for RNNs and image autoencoders.

Better extrapolation capabilities demonstrated in simulated and real mechanical systems.

Enhanced spatial and temporal error independence leading to more reliable control systems.

Abstract

Deep learning models are trained to minimize the error between the model's output and the actual values. The typical cost function, the Mean Squared Error (MSE), arises from maximizing the log-likelihood of additive independent, identically distributed Gaussian noise. However, minimizing MSE fails to minimize the residuals' cross-correlations, leading to over-fitting and poor extrapolation of the model outside the training set (generalization). In this paper, we introduce a "whitening" cost function, the Ljung-Box statistic, which not only minimizes the error but also minimizes the correlations between errors, ensuring that the fits enforce compatibility with an independent and identically distributed (i.i.d) gaussian noise model. The results show significant improvement in generalization for recurrent neural networks (RNNs) (1d) and image autoencoders (2d). Specifically, we look at both temporal correlations for system-id in simulated and actual mechanical systems. We also look at spatial correlation in vision autoencoders to demonstrate that the whitening objective functions lead to much better extrapolation--a property very desirable for reliable control systems.