SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation

TL;DR

This paper introduces Self-Evaluation Decoding (SED), a novel method that improves large language model generation by integrating speculation and evaluation steps, reducing errors at uncertain points during token selection.

Contribution

SED is a new decoding approach that mimics human decision-making, enhancing LLM output quality by addressing chaotic points during text generation.

Findings

SED improves generation quality across multiple tasks

Experimental results show better token selection at chaotic points

Enhances LLM performance with minimal additional computation

Abstract

Existing Large Language Models (LLMs) generate text through unidirectional autoregressive decoding methods to respond to various user queries. These methods tend to consider token selection in a simple sequential manner, making it easy to fall into suboptimal options when encountering uncertain tokens, referred to as chaotic points in our work. Many chaotic points exist in texts generated by LLMs, and they often significantly affect the quality of subsequently generated tokens, which can interfere with LLMs' generation. This paper proposes Self-Evaluation Decoding, SED, a decoding method for enhancing model generation. Analogous to the human decision-making process, SED integrates speculation and evaluation steps into the decoding process, allowing LLMs to make more careful decisions and thus optimize token selection at chaotic points. Experimental results across various tasks using different LLMs demonstrate SED's effectiveness.