Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once
Harnoor Dhingra

TL;DR
This paper introduces a framework to analyze LLM output diversity across different normative contexts, emphasizing the importance of task-specific evaluation over intrinsic model traits.
Contribution
The Magic, Madness, Heaven, Sin framework models output variation along a homogeneity-heterogeneity axis tailored to task objectives and normative contexts.
Findings
Optimizing for safety can reduce demographic representation and creative diversity.
The framework reveals complex interactions between different normative objectives.
Context-aware evaluation is essential for understanding LLM output variation.
Abstract
Research on Large Language Models (LLMs) studies output variation across generation, reasoning, alignment, and representational analysis, often under the umbrella of "diversity." Yet the terminology remains fragmented, largely because the normative objectives underlying tasks are rarely made explicit. We introduce the Magic, Madness, Heaven, Sin framework, which models output variation along a homogeneity-heterogeneity axis, where valuation is determined by the task and its normative objective. We organize tasks into four normative contexts: epistemic (factuality), interactional (user utility), societal (representation), and safety (robustness). For each, we examine the failure modes and vocabulary such as hallucination, mode collapse, bias, and erasure through which variation is studied. We apply the framework to analyze all pairwise cross-contextual interactions, revealing that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
