Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index

TL;DR

This paper introduces the Economical Prompting Index (EPI), a metric balancing accuracy and token cost, to evaluate prompting techniques under resource constraints, highlighting that simpler methods can be more cost-effective.

Contribution

The paper proposes the EPI metric that combines accuracy and token consumption, providing a new way to evaluate prompting techniques considering cost-efficiency.

Findings

Self-Consistency often offers minimal accuracy gains but is more costly.

Simpler prompting techniques like Chain-of-Thought can outperform complex methods in cost-effectiveness.

EPI effectively guides the selection of prompting strategies based on resource constraints.

Abstract

As prompt engineering research rapidly evolves, evaluations beyond accuracy are crucial for developing cost-effective techniques. We present the Economical Prompting Index (EPI), a novel metric that combines accuracy scores with token consumption, adjusted by a user-specified cost concern level to reflect different resource constraints. Our study examines 6 advanced prompting techniques, including Chain-of-Thought, Self-Consistency, and Tree of Thoughts, across 10 widely-used language models and 4 diverse datasets. We demonstrate that approaches such as Self-Consistency often provide statistically insignificant gains while becoming cost-prohibitive. For example, on high-performing models like Claude 3.5 Sonnet, the EPI of simpler techniques like Chain-of-Thought (0.72) surpasses more complex methods like Self-Consistency (0.64) at slight cost concern levels. Our findings suggest a reevaluation of complex prompting strategies in resource-constrained scenarios, potentially reshaping future research priorities and improving cost-effectiveness for end-users.