Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths

TL;DR

This paper examines how increasing context lengths in large language models impacts their performance and quality, revealing non-linear degradation linked to KV cache growth and architectural challenges at scale.

Contribution

It provides a detailed analysis of performance degradation in LLMs with large context windows and explores architectural effects, highlighting infrastructure bottlenecks and behavioral anomalies.

Findings

Performance degradation is non-linear with context length.

KV cache growth significantly impacts system performance.

Architectural benefits of MoE models are masked at high token volumes.

Abstract

The scaling trend in Large Language Models (LLMs) has prioritized increasing the maximum context window to facilitate complex, long-form reasoning and document analysis. However, managing this expanded context introduces severe computational overhead. This paper investigates the critical trade-off between system performance and model quality when dense transformer architectures--specifically Llama-3.1-70B and Qwen1.5-14B--are exposed to large volumes of irrelevant and distracting context. The research identifies a non-linear performance degradation tied to the growth of the Key-Value (KV) cache. Furthermore, an extended analysis of the Mixture-of-Experts (MoE) architecture reveals unique behavioral anomalies at varying context scales, suggesting that architectural benefits may be masked by infrastructure bottlenecks at high token volumes.