Post-edits Are Preferences Too

TL;DR

This paper investigates using post-edits as a reliable source of human preferences for fine-tuning large language models, showing that pre-training on post-edits improves model alignment with post-edit-like outputs.

Contribution

It demonstrates that post-edits can serve as implicit preferences for preference optimization, enhancing model performance in generating post-edit-like hypotheses.

Findings

Post-edits can be used as implicit preferences for fine-tuning.

Pre-training on post-edits leads to better alignment with post-edit-like outputs.

Post-edits help models move away from machine translation-like hypotheses.

Abstract

Preference Optimization (PO) techniques are currently one of the state of the art techniques for fine-tuning large language models (LLMs) on pairwise preference feedback from human annotators. However, in machine translation, this sort of feedback can be difficult to solicit. Additionally, Kreutzer et al. (2018) have shown that, for machine translation, pairwise preferences are less reliable than other forms of human feedback, such as 5-point ratings. We examine post-edits to see if they can be a source of reliable human preferences by construction. In PO, a human annotator is shown sequences $s_1$ and $s_2$ and asked for a preference judgment, %$s_1 > s_2$; while for post-editing, editors create $s_1$ and know that it should be better than $s_2$. We attempt to use these implicit preferences for PO and show that it helps the model move towards post-edit-like hypotheses and away from machine translation-like hypotheses. Furthermore, we show that best results are obtained by pre-training the model with supervised fine-tuning (SFT) on post-edits in order to promote post-edit-like hypotheses to the top output ranks.