Dynamic Sparse Attention: Access Patterns and Architecture

TL;DR

This paper investigates the system-level challenges of dynamic sparse attention (DSA), analyzes cache inefficiencies during autoregressive decoding, and proposes a cache reservation system to improve serving efficiency on modern hardware.

Contribution

It introduces a lightweight indexer for DSA, analyzes cache behavior during decoding, and proposes a novel last-level cache reservation system with token-granularity eviction to enhance performance.

Findings

Cache misses significantly impact DSA efficiency.

The proposed cache reservation system reduces cache misses and improves throughput.

Analysis guides future architectural improvements for DSA deployment.

Abstract

Dynamic sparse attention (DSA) reduces the per-token attention bandwidth by restricting computation to a top-k subset of cached key-value (KV) entries, but its token-dependent selection pattern introduces a system-level challenge: the KV working set is fragmented, volatile, and difficult to prefetch, which can translate into poor cache locality and stalled decode throughput. We study these effects by implementing a lightweight indexer for DSA-style selection on multiple open-source backbones and logging per-layer KV indices during autoregressive decoding. Our analysis shows a gap in serving DSA backbones - a potential for a high volume of blocking LL (last level) cache miss events, causing inefficiency; we propose a novel LL cache reservation system to save KV tokens in the LL cache between decode steps, combined with a token-granularity LRU eviction policy, and show on the data we collected how this architecture can benefit serving with DSA implemented on different backbones. Finally, we propose directions for future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms.