Metric entropy of causal, discrete-time LTI systems

TL;DR

This paper provides a clear, self-contained proof of the metric entropy of discrete-time LTI systems, clarifying previous mathematical issues and correcting related results in the context of neural network learning of such systems.

Contribution

It offers an elementary proof of metric entropy results for LTI systems, fixing minor mathematical issues and refining prior findings.

Findings

Clarified the metric entropy of LTI systems

Corrected a constant in previous results

Enhanced understanding of neural network approximation of LTI systems

Abstract

In [1] it is shown that recurrent neural networks (RNNs) can learn - in a metric entropy optimal manner - discrete time, linear time-invariant (LTI) systems. This is effected by comparing the number of bits needed to encode the approximating RNN to the metric entropy of the class of LTI systems under consideration [2, 3]. The purpose of this note is to provide an elementary self-contained proof of the metric entropy results in [2, 3], in the process of which minor mathematical issues appearing in [2, 3] are cleaned up. These corrections also lead to the correction of a constant in a result in [1] (see Remark 2.5).