CUNI Non-Autoregressive System for the WMT 22 Efficient Translation Shared Task

TL;DR

This paper introduces a non-autoregressive translation system for WMT 22, focusing on establishing reliable baselines and evaluation methods, especially for decoding speed, using a 12-layer Transformer trained with CTC on distilled data.

Contribution

It presents a non-autoregressive translation model with a robust evaluation methodology, facilitating fair comparison with autoregressive models.

Findings

Provides a solid baseline for non-autoregressive translation

Highlights the importance of evaluation metrics for decoding speed

Uses knowledge distillation to improve model performance

Abstract

We present a non-autoregressive system submission to the WMT 22 Efficient Translation Shared Task. Our system was used by Helcl et al. (2022) in an attempt to provide fair comparison between non-autoregressive and autoregressive models. This submission is an effort to establish solid baselines along with sound evaluation methodology, particularly in terms of measuring the decoding speed. The model itself is a 12-layer Transformer model trained with connectionist temporal classification on knowledge-distilled dataset by a strong autoregressive teacher model.