Inductive biases and Self Supervised Learning in modelling a physical heating system

TL;DR

This paper introduces a neural network architecture leveraging inductive biases like input delay and separability, trained with self-supervised learning, to effectively model a physical heating system for Model Predictive Control.

Contribution

The paper proposes a novel Delay architecture that incorporates inductive biases for modeling physical systems with delays, improving speed and data efficiency over traditional attention-based recurrent networks.

Findings

Delay architecture outperforms baseline in speed and data utilization.

Delay kernels are essential for effective learning.

Proposed models are nearly as accurate as baseline in prediction performance.

Abstract

Model Predictive Controllers (MPC) require a good model for the controlled process. In this paper I infer inductive biases about a physical system. I use these biases to derive a new neural network architecture that can model this real system that has noise and inertia. The main inductive biases exploited here are: the delayed impact of some inputs on the system and the separability between the temporal component and how the inputs interact to produce the output of a system. The inputs are independently delayed using shifted convolutional kernels. Feature interactions are modelled using a fully connected network that does not have access to temporal information. The available data and the problem setup allow the usage of Self Supervised Learning in order to train the models. The baseline architecture is an Attention based Reccurent network adapted to work with MPC like inputs. The proposed networks are faster, better at exploiting larger data volumes and are almost as good as baseline networks in terms of prediction performance. The proposed architecture family called Delay can be used in a real scenario to control systems with delayed responses with respect to its controls or inputs. Ablation studies show that the presence of delay kernels are vital to obtain any learning in proposed architecture. Code and some experimental data are available online.