Semantic-guided Diverse Decoding for Large Language Model

TL;DR

SemDiD is a novel decoding method for large language models that enhances semantic diversity in generated responses by operating in embedding space, leading to better task coverage and training efficiency.

Contribution

Introduces SemDiD, a semantic-guided decoding approach that balances diversity and quality through embedding space guidance and optimization techniques.

Findings

Improves Best-of-N coverage by 1.4-5.2% across tasks.

Accelerates RLHF training convergence by 15%.

Increases accuracy by up to 2.1%.

Abstract

Diverse decoding of large language models is crucial for applications requiring multiple semantically distinct responses, yet existing methods primarily achieve lexical rather than semantic diversity. This limitation significantly constrains Best-of-N strategies, group-based reinforcement learning, and data synthesis. While temperature sampling and diverse beam search modify token distributions or apply n-gram penalties, they fail to ensure meaningful semantic differentiation. We introduce Semantic-guided Diverse Decoding (SemDiD), operating directly in embedding space that balances quality with diversity through three complementary mechanisms: orthogonal directional guidance, dynamic inter-group repulsion, and position-debiased probability assessment. SemDiD harmonizes these competing objectives using adaptive gain functions and constraint optimization, ensuring both quality thresholds and maximal semantic differentiation. Experiments show SemDiD consistently outperforms existing methods, improving Best-of-N coverage by 1.4-5.2% across diverse tasks and accelerating RLHF training convergence by 15% while increasing accuracy by up to 2.1%.