GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs

TL;DR

GEM introduces an entropy-guided, self-optimization approach for aligning large language models with human preferences in low-resource and domain-specific settings, reducing reliance on large annotated datasets.

Contribution

It proposes a novel generative entropy-guided preference modeling framework that trains LLMs to internalize preference signals without extensive supervision.

Findings

Significant performance improvements on benchmarks with few-shot preference data.

Effective alignment in domain-specific tasks like medical dialogues and mathematical reasoning.

Demonstrates the viability of entropy-based self-evaluation for LLM alignment.

Abstract

Alignment of large language models (LLMs) with human preferences typically relies on supervised reward models or external judges that demand abundant annotations. However, in fields that rely on professional knowledge, such as medicine and law, such large-scale preference labels are often unachievable. In this paper, we propose a generative entropy-guided preference modeling approach named GEM for LLMs aligment at low-resource and domain-specific scenarios. Instead of training a discriminative reward model on preference data, we directly train the LLM to internalize a closed-loop optimization architecture that can extract and exploit the multi-dimensional, fine-grained cognitive signals implicit in human preferences. Specifically, our Cognitive Filtering module, based on entropy theory in decision making, first leverages Chain-of-Thought (CoT) prompting to generate diverse candidate reasoning chains (CoTs) from preference data. Subsequently, it introduces a token scoring mechanism to rank and weight the sampled CoTs, boosting the importance of high-confidence answers and strategically high-entropy tokens. Building on these filtered preferences, we fine-tune the LLM using a novel self-evaluated group advantage algorithm, SEGA, which effectively aggregates group-level cognitive signals and transforms the entropy-based scores into implicit rewards for policy optimization. In these ways, GEM empowers the LLM to rely on its own judgments and establishes an entropy-guided closed-loop cognitive optimization framework, enabling highly efficient few-shot alignment of LLMs. Experiments on general benchmarks and domain-specific tasks (such as mathematical reasoning and medical dialogues) demonstrate that our GEM achieves significant improvements with few-shot preference data.