Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection

TL;DR

This paper systematically compares self-reflection and budget tuning techniques for LLM inference, analyzing their trade-offs in accuracy, cost, and speed across domains, and provides practical guidance for optimal strategy selection.

Contribution

It offers a comprehensive evaluation of self-reflection and budget tuning across multiple LLMs and domains, highlighting domain-dependent effectiveness and providing actionable insights.

Findings

Self-reflection can improve performance up to 220% in mathematical reasoning.

Performance gains vary significantly across domains and models.

Deployment in real-world systems confirms domain-specific effectiveness.

Abstract

As Large Language Models (LLMs) continue to evolve, practitioners face increasing options for enhancing inference-time performance without model retraining, including budget tuning and multi-step techniques like self-reflection. While these methods improve output quality, they create complex trade-offs among accuracy, cost, and latency that remain poorly understood across different domains. This paper systematically compares self-reflection and budget tuning across mathematical reasoning and translation tasks. We evaluate prominent LLMs, including Anthropic Claude, Amazon Nova, and Mistral families, along with other models under varying reflection depths and compute budgets to derive Pareto optimal performance frontiers. Our analysis reveals substantial domain dependent variation in self-reflection effectiveness, with performance gains up to 220\% in mathematical reasoning. We further investigate how reflection round depth and feedback mechanism quality influence performance across model families. To validate our findings in a real-world setting, we deploy a self-reflection enhanced marketing content localisation system at Lounge by Zalando, where it shows market-dependent effectiveness, reinforcing the importance of domain specific evaluation when deploying these techniques. Our results provide actionable guidance for selecting optimal inference strategies given specific domains and resource constraints. We open source our self-reflection implementation for reproducibility at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.