Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs

TL;DR

This paper introduces an incremental inference method for neural networks, especially transformers, that reuses calculations to efficiently process dynamic inputs with minimal recomputation, significantly reducing computational costs.

Contribution

It proposes a novel approach using vector quantization to enable incremental computation in transformers, improving efficiency without sacrificing accuracy.

Findings

Achieved 12.1X median reduction in operations for document edits

Maintained comparable accuracy on classification tasks

Demonstrated effectiveness on pre-trained language models

Abstract

Deep learning often faces the challenge of efficiently processing dynamic inputs, such as sensor data or user inputs. For example, an AI writing assistant is required to update its suggestions in real time as a document is edited. Re-running the model each time is expensive, even with compression techniques like knowledge distillation, pruning, or quantization. Instead, we take an incremental computing approach, looking to reuse calculations as the inputs change. However, the dense connectivity of conventional architectures poses a major obstacle to incremental computation, as even minor input changes cascade through the network and restrict information reuse. To address this, we use vector quantization to discretize intermediate values in the network, which filters out noisy and unnecessary modifications to hidden neurons, facilitating the reuse of their values. We apply this approach to the transformers architecture, creating an efficient incremental inference algorithm with complexity proportional to the fraction of the modified inputs. Our experiments with adapting the OPT-125M pre-trained language model demonstrate comparable accuracy on document classification while requiring 12.1X (median) fewer operations for processing sequences of atomic edits.