Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet

TL;DR

This study compares KalmanNet, a deep learning-based state estimator, to traditional filters like IMM in automotive radar scenarios, revealing that KalmanNet currently underperforms and lacks robustness for safety-critical use.

Contribution

The paper provides an empirical evaluation of KalmanNet's performance on real-world automotive radar data, highlighting its limitations compared to traditional filtering methods.

Findings

KalmanNet is outperformed by the IMM filter in accuracy.

KalmanNet shows less reliability and robustness in real-world conditions.

Traditional filters remain superior for safety-critical applications.

Abstract

Kalman Filters (KF) are fundamental to real-time state estimation applications, including radar-based tracking systems used in modern driver assistance and safety technologies. In a linear dynamical system with Gaussian noise distributions the KF is the optimal estimator. However, real-world systems often deviate from these assumptions. This deviation combined with the success of deep learning across many disciplines has prompted the exploration of data driven approaches that leverage deep learning for filtering applications. These learned state estimators are often reported to outperform traditional model based systems. In this work, one prevalent model, KalmanNet, was selected and evaluated on automotive radar data to assess its performance under real-world conditions and compare it to an interacting multiple models (IMM) filter. The evaluation is based on raw and normalized errors as well as the state uncertainty. The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.