Mechanism Design for LLM Fine-tuning with Multiple Reward Models

TL;DR

This paper designs incentive-compatible mechanisms for fine-tuning large language models with multiple reward models, ensuring truthful preference reporting and social welfare maximization.

Contribution

It introduces a mechanism design framework for LLM fine-tuning with multiple reward models, extending VCG payments for incentive compatibility, and analyzes robustness and practical implications.

Findings

Truthful reporting is sub-optimal under simple SW-Max rules without payments.

Extended VCG payments achieve dominant-strategy incentive compatibility.

Mechanism demonstrates approximate DSIC with input perturbations, confirming robustness.

Abstract

Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply a SW-Max rule, thereby highlighting the necessity of payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for a payment rule to implement a SW-Max rule in DSIC and other principles. Third, we demonstrate that our mechanism is approximately DSIC with perturbed input, showcasing its robustness against the inevitable errors in real-world applications. Experiments on real LLM training results further confirm the practical implications of our results.