Encouraging an Appropriate Representation Simplifies Training of Neural Networks

TL;DR

This paper challenges the assumption that neural networks can learn appropriate internal representations on their own, demonstrating that guiding representations can improve training success and generalization.

Contribution

It shows that encouraging specific internal representations can enable neural networks to solve tasks they otherwise fail at, highlighting the benefit of integrating domain knowledge.

Findings

State-of-the-art training fails without guided representations

Encouraging internal representations enables solving simple tasks

Guided representations improve generalization ability

Abstract

A common assumption about neural networks is that they can learn an appropriate internal representations on their own, see e.g. end-to-end learning. In this work we challenge this assumption. We consider two simple tasks and show that the state-of-the-art training algorithm fails, although the model itself is able to represent an appropriate solution. We will demonstrate that encouraging an appropriate internal representation allows the same model to solve these tasks. While we do not claim that it is impossible to solve these tasks by other means (such as neural networks with more layers), our results illustrate that integration of domain knowledge in form of a desired internal representation may improve the generalization ability of neural networks.