Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models

TL;DR

This paper demonstrates that cross-family speculative prefill effectively compresses prompts for large language models, reducing inference time while maintaining high performance, even when using draft models from different families.

Contribution

It introduces and evaluates a training-free prompt compression method that transfers attention-based token importance estimation across different model families, expanding its practical applicability.

Findings

Cross-family prompt compression retains 90-100% of baseline performance.

It significantly reduces time to first token (TTFT) in inference.

Attention-based token importance estimation transfers reliably across diverse models.

Abstract

Prompt length is a major bottleneck in agentic large language model (LLM) workloads, where repeated inference steps and multi-call loops incur substantial prefill cost. Recent work on speculative prefill demonstrates that attention-based token importance estimation can enable training-free prompt compression, but this assumes the existence of a draft model that shares the same tokenizer as the target model. In practice, however, agentic pipelines frequently employ models without any smaller in-family draft model. In this work, we study cross-family speculative prefill, where a lightweight draft model from one model family is used to perform prompt compression for a target model from a different family. Using the same speculative prefill mechanism as prior work, we evaluate a range of cross-family draft-target combinations, including Qwen, LLaMA, and DeepSeek models. Across a broad diversity of tasks, we find that attention-based token importance estimation transfers reliably across different model families despite differences in model architectures and tokenizers between draft and target models. Cross-model prompt compression largely retains 90~100% of full-prompt baseline performance and, in some cases, slightly improves accuracy due to denoising effects, while delivering substantial reductions in time to first token (TTFT). These results suggest that speculative prefill depends mainly on task priors and semantic structure, thus serving as a generalizable prompt compression primitive. We discuss the implications of our findings for agentic systems, where repeated long-context inference and heterogeneous model stacks make cross-model prompt compression both necessary and practical.