Non-linear State-space Model Identification from Video Data using Deep Encoders

TL;DR

This paper introduces a novel non-linear state-space identification approach using deep neural network encoders to analyze high-dimensional video data, enabling accurate long-term predictions in complex systems.

Contribution

It presents a new method combining neural encoders and advanced optimization techniques for high-dimensional system identification from video data.

Findings

Achieved low simulation error in a simulated environment

Demonstrated excellent long-term prediction capabilities

Effectively handled high-dimensional, large datasets

Abstract

Identifying systems with high-dimensional inputs and outputs, such as systems measured by video streams, is a challenging problem with numerous applications in robotics, autonomous vehicles and medical imaging. In this paper, we propose a novel non-linear state-space identification method starting from high-dimensional input and output data. Multiple computational and conceptual advances are combined to handle the high-dimensional nature of the data. An encoder function, represented by a neural network, is introduced to learn a reconstructability map to estimate the model states from past inputs and outputs. This encoder function is jointly learned with the dynamics. Furthermore, multiple computational improvements, such as an improved reformulation of multiple shooting and batch optimization, are proposed to keep the computational time under control when dealing with high-dimensional and large datasets. We apply the proposed method to a video stream of a simulated environment of a controllable ball in a unit box. The study shows low simulation error with excellent long term prediction capability of the model obtained using the proposed method.