PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation

TL;DR

PromptSuite is a flexible, extensible framework that automatically generates diverse prompts for more reliable multi-prompt evaluation of large language models across various tasks.

Contribution

It introduces a modular, task-agnostic system for automatic prompt variation generation, enhancing robustness in LLM evaluation.

Findings

PromptSuite enables meaningful prompt variations for evaluation.

It works across a wide range of tasks and benchmarks.

Resources are publicly available at the provided URL.

Abstract

Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for a more robust multi-prompt evaluation is challenging, limiting its adoption in practice. To address this, we introduce PromptSuite, a framework that enables the automatic generation of various prompts. PromptSuite is flexible - working out of the box on a wide range of tasks and benchmarks. It follows a modular prompt design, allowing controlled perturbations to each component, and is extensible, supporting the addition of new components and perturbation types. Through a series of case studies, we show that PromptSuite provides meaningful variations to support strong evaluation practices. All resources, including the Python API, source code, user-friendly web interface, and demonstration video, are available at: https://eliyahabba.github.io/PromptSuite/.