Beyond Occam's Razor in System Identification: Double-Descent when Modeling Dynamics

TL;DR

This paper investigates the occurrence of double-descent phenomena in system identification, showing that models can improve in performance even after perfectly fitting training data, challenging traditional model complexity trade-offs.

Contribution

It demonstrates experimentally that double-descent behavior occurs in dynamic system modeling, extending the understanding of model complexity effects beyond static machine learning tasks.

Findings

Double-descent observed in system identification models.

Performance improves beyond interpolation threshold.

Validates phenomena on real-world and synthetic data.

Abstract

System identification aims to build models of dynamical systems from data. Traditionally, choosing the model requires the designer to balance between two goals of conflicting nature; the model must be rich enough to capture the system dynamics, but not so flexible that it learns spurious random effects from the dataset. It is typically observed that the model validation performance follows a U-shaped curve as the model complexity increases. Recent developments in machine learning and statistics, however, have observed situations where a "double-descent" curve subsumes this U-shaped model-performance curve. With a second decrease in performance occurring beyond the point where the model has reached the capacity of interpolating - i.e., (near) perfectly fitting - the training data. To the best of our knowledge, such phenomena have not been studied within the context of dynamic systems. The present paper aims to answer the question: "Can such a phenomenon also be observed when estimating parameters of dynamic systems?" We show that the answer is yes, verifying such behavior experimentally both for artificially generated and real-world datasets.