Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding

TL;DR

This paper introduces agreement-constrained PMBR decoding, which uses a knowledge distilled model to improve translation quality and efficiency by better approximating utility scores in minimum Bayes risk decoding.

Contribution

It proposes a novel AC-PMBR decoding method that enhances matrix completion accuracy using knowledge distillation, balancing translation quality and computational cost.

Findings

Improved matrix completion errors by up to 3 times.

Achieved higher translation quality than PMBR at similar computational cost.

Demonstrated effectiveness on WMT'23 En-De translation tasks.

Abstract

Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candidate set; hence, it takes quadratic time with respect to the number of candidates. To reduce the number of utility function calls, probabilistic MBR (PMBR) decoding partially evaluates quality scores using sampled pairs of candidates and completes the missing scores with a matrix completion algorithm. Nevertheless, it degrades the translation quality as the number of utility function calls is reduced. Therefore, to improve the trade-off between quality and cost, we propose agreement-constrained PMBR (AC-PMBR) decoding, which leverages a knowledge distilled model to guide the completion of the score matrix. Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding at a comparable computational cost on the WMT'23 En$\leftrightarrow$De translation tasks.