C2T: A Classifier-Based Tree Construction Method in Speculative Decoding

TL;DR

C2T introduces a classifier-based approach for dynamic token tree construction in speculative decoding, significantly reducing candidate tokens and improving efficiency in large language model inference.

Contribution

The paper presents a novel classifier-driven method for dynamic token tree generation that outperforms existing strategies in efficiency and accuracy.

Findings

Reduces candidate tokens by 25% compared to SOTA methods.

Maintains or improves acceptance length in decoding.

Outperforms EAGLE-2 on multiple benchmarks.

Abstract

The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.