Optimal Completion Distillation for Sequence Learning

TL;DR

Optimal Completion Distillation (OCD) is a novel training method for sequence models that directly optimizes edit distance, leading to state-of-the-art speech recognition results without requiring hyper-parameter tuning or pretraining.

Contribution

OCD introduces an efficient, hyper-parameter-free training procedure based on optimal suffixes for sequence-to-sequence models, improving performance on speech recognition tasks.

Findings

Achieves 9.3% WER on Wall Street Journal dataset.

Achieves 4.5% WER on Librispeech dataset.

Outperforms previous methods in sequence learning accuracy.

Abstract

We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pretraining or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm. Then, for each position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER respectively.