Theoretical Guarantees for Minimum Bayes Risk Decoding

TL;DR

This paper provides theoretical analysis explaining why Minimum Bayes Risk (MBR) decoding performs well, showing it converges to the optimal solution at a rate of O(n^{-1/2}) under certain conditions, despite large language spaces.

Contribution

It offers the first analytical proof of MBR decoding's convergence rate and compares its performance gap with MAP decoding, explaining empirical success.

Findings

MBR decoding approaches the optimal solution at a rate of O(n^{-1/2})

MBR decoding converges faster than MAP decoding in certain cases

Theoretical explanation for MBR's strong empirical performance

Abstract

Minimum Bayes Risk (MBR) decoding optimizes output selection by maximizing the expected utility value of an underlying human distribution. While prior work has shown the effectiveness of MBR decoding through empirical evaluation, few studies have analytically investigated why the method is effective. As a result of our analysis, we show that, given the size $n$ of the reference hypothesis set used in computation, MBR decoding approaches the optimal solution with high probability at a rate of $O\left(n^{-\frac{1}{2}}\right)$, under certain assumptions, even though the language space $Y$ is significantly larger $|Y|\gg n$. This result helps to theoretically explain the strong performance observed in several prior empirical studies on MBR decoding. In addition, we provide the performance gap for maximum-a-posteriori (MAP) decoding and compare it to MBR decoding. The result of this paper indicates that MBR decoding tends to converge to the optimal solution faster than MAP decoding in several cases.