Diver: Large Language Model Decoding with Span-Level Mutual Information Verification

TL;DR

Diver improves large language model decoding by using span-level mutual information verification to select outputs that better reflect input information, enhancing performance and reliability.

Contribution

The paper introduces Diver, a novel span-level PMI verification method that re-ranks LLM outputs during decoding to improve input-output alignment.

Findings

Diver outperforms existing decoding methods in multiple downstream tasks.

Diver enhances the consistency of LLM outputs with input information.

Empirical results show significant performance gains across various benchmarks.

Abstract

Large language models (LLMs) have shown impressive capabilities in adapting to various tasks when provided with task-specific instructions. However, LLMs using standard decoding strategies often struggle with deviations from the inputs. Intuitively, compliant LLM outputs should reflect the information present in the input, which can be measured by point-wise mutual information (PMI) scores. Therefore, we propose Diver, a novel approach that enhances LLM Decoding through span-level PMI verification. During inference, Diver first identifies divergence steps that may lead to multiple candidate spans. Subsequently, it calculates the PMI scores by assessing the log-likelihood gains of the input if the candidate spans are generated. Finally, the optimal span is selected based on the PMI re-ranked output distributions. We evaluate our method across various downstream tasks, and empirical results demonstrate that Diver significantly outperforms existing decoding methods in both performance and versatility.