Hierarchical Verification of Speculative Beams for Accelerating LLM Inference

TL;DR

This paper introduces the Hierarchical Verification Tree (HVT), a novel framework that accelerates large language model inference by prioritizing high-likelihood drafts and enabling early pruning, reducing computational costs without retraining.

Contribution

The paper presents a new hierarchical verification approach for speculative beam decoding that improves efficiency and correctness without modifying model architecture.

Findings

HVT reduces inference time significantly across multiple datasets.

Energy consumption decreases with minimal impact on output quality.

HVT outperforms existing speculative decoding methods in speed and efficiency.

Abstract

Large language models (LLMs) have achieved remarkable success across diverse natural language processing tasks but face persistent challenges in inference efficiency due to their autoregressive nature. While speculative decoding and beam sampling offer notable improvements, traditional methods verify draft sequences sequentially without prioritization, leading to unnecessary computational overhead. This work proposes the Hierarchical Verification Tree (HVT), a novel framework that restructures speculative beam decoding by prioritizing high-likelihood drafts and enabling early pruning of suboptimal candidates. Theoretical foundations and a formal verification-pruning algorithm are developed to ensure correctness and efficiency. Integration with standard LLM inference pipelines is achieved without requiring retraining or architecture modification. Experimental evaluations across multiple datasets and models demonstrate that HVT consistently outperforms existing speculative decoding schemes, achieving substantial reductions in inference time and energy consumption while maintaining or enhancing output quality. The findings highlight the potential of hierarchical verification strategies as a new direction for accelerating large language model inference.