Conditional Information Gain Networks

TL;DR

Conditional Information Gain Networks enable deep neural networks to execute conditionally, reducing computation by skipping parts of the model based on input, trained with differentiable information gain mechanisms.

Contribution

The paper introduces a novel conditional computation framework using differentiable information gain for end-to-end training of neural networks.

Findings

Achieves comparable or better accuracy than standard CNNs on MNIST and Fashion MNIST.

Uses significantly fewer parameters while maintaining performance.

Demonstrates effective conditional execution in deep neural networks.

Abstract

Deep neural network models owe their representational power to the high number of learnable parameters. It is often infeasible to run these largely parametrized deep models in limited resource environments, like mobile phones. Network models employing conditional computing are able to reduce computational requirements while achieving high representational power, with their ability to model hierarchies. We propose Conditional Information Gain Networks, which allow the feed forward deep neural networks to execute conditionally, skipping parts of the model based on the sample and the decision mechanisms inserted in the architecture. These decision mechanisms are trained using cost functions based on differentiable Information Gain, inspired by the training procedures of decision trees. These information gain based decision mechanisms are differentiable and can be trained end-to-end using a unified framework with a general cost function, covering both classification and decision losses. We test the effectiveness of the proposed method on MNIST and recently introduced Fashion MNIST datasets and show that our information gain based conditional execution approach can achieve better or comparable classification results using significantly fewer parameters, compared to standard convolutional neural network baselines.