Beyond Convergence: Identifiability of Machine Learning and Deep Learning Models

TL;DR

This paper investigates the concept of parameter identifiability in machine learning and deep learning models, demonstrating that some parameters cannot be uniquely determined from data, which impacts model reliability and requires improved data collection strategies.

Contribution

The study provides a case analysis of parameter identifiability in a human gait model using deep learning, highlighting intrinsic limitations and proposing broader implications for ML model design.

Findings

Some parameters are identifiable from data, others are not.

Unidentifiability is an inherent limitation of experimental setups.

Addressing unidentifiability involves using theoretical models and multimodal data fusion.

Abstract

Machine learning (ML) and deep learning models are extensively used for parameter optimization and regression problems. However, not all inverse problems in ML are ``identifiable,'' indicating that model parameters may not be uniquely determined from the available data and the data model's input-output relationship. In this study, we investigate the notion of model parameter identifiability through a case study focused on parameter estimation from motion sensor data. Utilizing a bipedal-spring mass human walk dynamics model, we generate synthetic data representing diverse gait patterns and conditions. Employing a deep neural network, we attempt to estimate subject-wise parameters, including mass, stiffness, and equilibrium leg length. The results show that while certain parameters can be identified from the observation data, others remain unidentifiable, highlighting that unidentifiability is an intrinsic limitation of the experimental setup, necessitating a change in data collection and experimental scenarios. Beyond this specific case study, the concept of identifiability has broader implications in ML and deep learning. Addressing unidentifiability requires proven identifiable models (with theoretical support), multimodal data fusion techniques, and advancements in model-based machine learning. Understanding and resolving unidentifiability challenges will lead to more reliable and accurate applications across diverse domains, transcending mere model convergence and enhancing the reliability of machine learning models.