Adaptive Computation with Elastic Input Sequence

TL;DR

AdaTape introduces a dynamic neural network approach with elastic input sequences, enabling adaptive computation through tape tokens, leading to improved performance and efficiency in image recognition tasks.

Contribution

The paper presents AdaTape, a novel method for adaptive neural computation using elastic input sequences generated by tape tokens, enhancing flexibility and performance.

Findings

Achieves better accuracy with comparable computational cost.

Demonstrates effective adaptive sequence generation in image recognition.

Provides a new framework for dynamic neural network computation.

Abstract

Humans have the ability to adapt the type of information they use, the procedure they employ, and the amount of time they spend when solving problems. However, most standard neural networks have a fixed function type and computation budget regardless of the sample's nature or difficulty. Adaptivity is a powerful paradigm as it not only imbues practitioners with flexibility pertaining to the downstream usage of these models but can also serve as a powerful inductive bias for solving certain challenging classes of problems. In this work, we introduce a new approach called AdaTape, which allows for dynamic computation in neural networks through adaptive tape tokens. AdaTape utilizes an elastic input sequence by equipping an architecture with a dynamic read-and-write tape. Specifically, we adaptively generate input sequences using tape tokens obtained from a tape bank which can be either trainable or derived from input data. We examine the challenges and requirements to obtain dynamic sequence content and length, and propose the Adaptive Tape Reading (ATR) algorithm to achieve both goals. Through extensive experiments on image recognition tasks, we show that AdaTape can achieve better performance while maintaining the computational cost. To facilitate further research, we have released code at https://github.com/google-research/scenic.