PonderNet: Learning to Ponder

TL;DR

PonderNet is a novel neural network algorithm that dynamically adjusts the number of computational steps based on input complexity, improving performance and efficiency on complex tasks.

Contribution

It introduces an end-to-end learnable method for adaptive computation in neural networks, outperforming previous methods on synthetic and real-world tasks.

Findings

Significantly improves performance on synthetic complex problems

Achieves state-of-the-art results on reasoning tasks

Uses less compute while maintaining high accuracy

Abstract

In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand. PonderNet learns end-to-end the number of computational steps to achieve an effective compromise between training prediction accuracy, computational cost and generalization. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at extrapolation tests where traditional neural networks fail. Also, our method matched the current state of the art results on a real world question and answering dataset, but using less compute. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural networks.1