An Objective Performance Evaluation of the LSTM Networks in Time Series Classification

TL;DR

This paper compares LSTM networks and model-based EM classifiers for time-series classification, revealing that EM classifiers perform better when the data matches known models, while LSTMs need more distinct noise differences to be reliable.

Contribution

It provides an objective evaluation framework for comparing data-driven LSTM classifiers with model-based EM classifiers in structured environments.

Findings

EM classifier performs well with known model structure.

LSTM requires larger noise separation for reliable classification.

LSTM performance saturates below the reference classifier when models differ only in measurement noise.

Abstract

The rapid adoption of deep learning has increasingly led to data-driven models replacing classical model-based algorithms, even in domains governed by well-understood physical laws. While data-driven models, such as long short-term memory (LSTM) networks, have become a popular choice for time-series analysis, their performance relative to model-based approaches in structured environments is rarely evaluated objectively. This paper presents a performance evaluation framework comparing an LSTM classifier against a model-based expectation maximization (EM) classifier for binary time-series classification. The evaluation is conducted on two scalar linear Gaussian state space models differing only in their noise statistics, where the Kalman filter likelihood ratio test with true parameters serves as a reference for the best achievable classification performance.Through Monte Carlo simulations, the classifiers are evaluated across three axes: task difficulty, controlled by the separation in process or measurement noise between the two models; sequence length; and training dataset size. The results show that the EM classifier, which exploits the known model structure, performs strongly when the data conform to the assumed model class. The LSTM classifier requires a larger separation in noise statistics to achieve reliable classification, and its performance saturates below the reference classifier when the models differ only in measurement noise, regardless of sequence length or training dataset size.