Mutual information for fitting deep nonlinear models

TL;DR

This paper explores the use of mutual information and KL divergence as objective functions for fitting deep nonlinear models without knowing hidden layers, demonstrating their effectiveness in a proof-of-concept and cognitive performance application.

Contribution

It introduces information theoretic measures as novel objective functions for training deep nonlinear models without hidden layer knowledge.

Findings

Mutual information effectively guides model fitting depending on parameters.

KL divergence is successful with some knowledge of hidden layer statistics.

Both measures outperform traditional methods in the tested scenarios.

Abstract

Deep nonlinear models pose a challenge for fitting parameters due to lack of knowledge of the hidden layer and the potentially non-affine relation of the initial and observed layers. In the present work we investigate the use of information theoretic measures such as mutual information and Kullback-Leibler (KL) divergence as objective functions for fitting such models without knowledge of the hidden layer. We investigate one model as a proof of concept and one application of cogntive performance. We further investigate the use of optimizers with these methods. Mutual information is largely successful as an objective, depending on the parameters. KL divergence is found to be similarly succesful, given some knowledge of the statistics of the hidden layer.