Quantifying Laziness, Decoding Suboptimality, and Context Degradation in Large Language Models

TL;DR

This paper investigates behavioral issues in large language models, revealing prevalent laziness and instruction non-compliance, while surprisingly showing robustness against context degradation in long conversations, with implications for improving model reliability.

Contribution

The study provides a systematic quantification of laziness, decoding suboptimality, and context degradation in advanced LLMs through controlled experiments, highlighting areas for improvement.

Findings

Widespread laziness in multi-part instruction compliance

Limited evidence of decoding suboptimality in reasoning tasks

Models maintain core facts well over 200-turn conversations

Abstract

Large Language Models (LLMs) often exhibit behavioral artifacts such as laziness (premature truncation of responses or partial compliance with multi-part requests), decoding suboptimality (failure to select higher-quality sequences due to myopic decoding), and context degradation (forgetting or ignoring core instructions over long conversations). We conducted three controlled experiments (A, B, and C) to quantify these phenomena across several advanced LLMs (OpenAI GPT-4 variant, DeepSeek). Our results indicate widespread laziness in satisfying complex multi-part instructions: models frequently omitted required sections or failed to meet length requirements despite explicit prompting. However, we found limited evidence of decoding suboptimality in a simple reasoning task (the models' greedy answers appeared to align with their highest-confidence solution), and we observed surprising robustness against context degradation in a 200-turn chaotic conversation test - the models maintained key facts and instructions far better than expected. These findings suggest that while compliance with detailed instructions remains an open challenge, modern LLMs may internally mitigate some hypothesized failure modes (such as context forgetting) in straightforward retrieval scenarios. We discuss implications for reliability, relate our findings to prior work on instruction-following and long-context processing, and recommend strategies (such as self-refinement and dynamic prompting) to reduce laziness and bolster multi-instruction compliance.