Stochastic Surprisal: An inferential measurement of Free Energy in Neural Networks

TL;DR

This paper introduces stochastic surprisal, a new measure for neural networks that incorporates action during inference, improving robustness and analysis in noisy environments across multiple applications.

Contribution

It proposes stochastic surprisal as a novel measurement integrating action during inference, validated on image quality and recognition tasks with significant improvements.

Findings

Significant performance increase across all measures

Effective in noisy recognition and image quality assessment

Applicable as a plug-in on multiple networks

Abstract

This paper conjectures and validates a framework that allows for action during inference in supervised neural networks. Supervised neural networks are constructed with the objective to maximize their performance metric in any given task. This is done by reducing free energy and its associated surprisal during training. However, the bottom-up inference nature of supervised networks is a passive process that renders them fallible to noise. In this paper, we provide a thorough background of supervised neural networks, both generative and discriminative, and discuss their functionality from the perspective of free energy principle. We then provide a framework for introducing action during inference. We introduce a new measurement called stochastic surprisal that is a function of the network, the input, and any possible action. This action can be any one of the outputs that the neural network has learnt, thereby lending stochasticity to the measurement. Stochastic surprisal is validated on two applications: Image Quality Assessment and Recognition under noisy conditions. We show that, while noise characteristics are ignored to make robust recognition, they are analyzed to estimate image quality scores. We apply stochastic surprisal on two applications, three datasets, and as a plug-in on twelve networks. In all, it provides a statistically significant increase among all measures. We conclude by discussing the implications of the proposed stochastic surprisal in other areas of cognitive psychology including expectancy-mismatch and abductive reasoning.