A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation

TL;DR

This paper compares five DSPy teleprompter algorithms for aligning large language model evaluation metrics with human judgments, demonstrating that optimized prompts can outperform benchmarks in hallucination detection.

Contribution

It provides a comparative analysis of five teleprompter algorithms within DSPy for aligning LLM evaluations to human annotations, highlighting their relative effectiveness.

Findings

Optimized prompts outperform benchmark methods in hallucination detection.

Certain teleprompter algorithms outperform others in specific experiments.

Prompt optimization improves alignment with human evaluations.

Abstract

We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments.