Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy
Francisca Lemos (1), Victor Alves (1), Filipa Ferraz (1) ((1) ALGORITMI Research Centre/LASI, University of Minho)

TL;DR
This paper introduces DSPy, a framework for programmatically optimizing prompts for LLMs across multiple use cases, showing variable but often significant performance improvements.
Contribution
The study presents DSPy, a novel framework that automates prompt creation and refinement, demonstrating its effectiveness across diverse LLM tasks.
Findings
Significant accuracy increase in prompt evaluation from 46.2% to 64.0%.
Improved routing accuracy from 85.0% to 90.0% with optimized prompts.
Variable impact of prompt optimization across different tasks.
Abstract
Although prompt engineering is central to unlocking the full potential of Large Language Models (LLMs), crafting effective prompts remains a time-consuming trial-and-error process that relies on human intuition. This study investigates Declarative Self-improving Python (DSPy), an optimization framework that programmatically creates and refines prompts, applied to five use cases: guardrail enforcement, hallucination detection in code, code generation, routing agents, and prompt evaluation. Each use case explores how prompt optimization via DSPy influences performance. While some cases demonstrated modest improvements - such as minor gains in the guardrails use case and selective enhancements in hallucination detection - others showed notable benefits. The prompt evaluation criterion task demonstrated a substantial performance increase, rising accuracy from 46.2% to 64.0%. In the router…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
