Switchable Decision: Dynamic Neural Generation Networks

TL;DR

This paper introduces a switchable decision mechanism for dynamic neural generation networks that accelerates inference by adaptively allocating computational resources, maintaining accuracy while reducing computation costs across various NLP tasks.

Contribution

It presents a novel method for dynamically deciding computation paths in neural generation models to optimize the trade-off between speed and quality.

Findings

Reduces inference computation cost without sacrificing accuracy.

Effective across multiple NLP tasks like question answering, summarization, and classification.

Demonstrates general applicability and robustness through extensive experiments.

Abstract

Auto-regressive generation models achieve competitive performance across many different NLP tasks such as summarization, question answering, and classifications. However, they are also known for being slow in inference, which makes them challenging to deploy in real-time applications. We propose a switchable decision to accelerate inference by dynamically assigning computation resources for each data instance. Automatically making decisions on where to skip and how to balance quality and computation cost with constrained optimization, our dynamic neural generation networks enforce the efficient inference path and determine the optimized trade-off. Experiments across question answering, summarization, and classification benchmarks show that our method benefits from less computation cost during inference while keeping the same accuracy. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.