Flatter Tokens are More Valuable for Speculative Draft Model Training

TL;DR

This paper introduces a data-centric method for improving speculative decoding in large language models by selecting training samples with flatter predictive distributions, leading to faster training with minimal accuracy loss.

Contribution

It proposes a new flatness metric and a dataset distillation method that filters valuable samples, significantly accelerating training for speculative decoding.

Findings

SFDD achieves over 2x training speedup with 50% data

Final model inference speed remains within 4% of full data baseline

The flatness metric effectively identifies valuable training samples

Abstract

Speculative Decoding (SD) is a key technique for accelerating Large Language Model (LLM) inference, but it typically requires training a draft model on a large dataset. We approach this problem from a data-centric perspective, finding that not all training samples contribute equally to the SD acceptance rate. Specifically, our theoretical analysis and empirical validation reveals that tokens inducing flatter predictive distributions from the target model are more valuable than those yielding sharply peaked distributions. Based on this insight, we propose flatness, a new metric to quantify this property, and develop the Sample-level-flatness-based Dataset Distillation (SFDD) approach, which filters the training data to retain only the most valuable samples. Experiments on the EAGLE framework demonstrate that SFDD can achieve over 2$\times$ training speedup using only 50% of the data, while keeping the final model's inference speedup within 4% of the full-dataset baseline. This work introduces an effective, data-centric approach that substantially improves the training efficiency for Speculative Decoding. Our code is available at https://github.com/fjm9933/Flatness.