LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers

TL;DR

LayerCake introduces a token-aware contrastive decoding method that aligns token types with specific transformer layers to enhance factual accuracy in large language models without additional training.

Contribution

It presents a novel layer-aware contrastive decoding approach that jointly considers token types and layer dynamics to improve factuality in LLM outputs.

Findings

Consistently improves factual accuracy across multiple LLMs.

Effectively suppresses attention to certain token types at specific layers.

No additional training or model modifications required.

Abstract

Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors, limiting their reliability in knowledge-intensive tasks. While decoding-time strategies provide a promising efficient solution without training, existing methods typically treat token-level and layer-level signals in isolation, overlooking the joint dynamics between them. In this work, we introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation. Through empirical attention analysis, we identify two key patterns: punctuation tokens receive dominant attention in early layers, while conceptual tokens govern semantic reasoning in intermediate layers. By selectively suppressing attention to these token types at their respective depths, we achieve the induction of controlled factual degradation and derive contrastive signals to guide the final factual decoding. Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.