Enhancing LLM Instruction Following: An Evaluation-Driven Multi-Agentic Workflow for Prompt Instructions Optimization

TL;DR

This paper introduces a multi-agentic workflow that improves prompt instructions for LLMs by separately optimizing task descriptions and constraints, leading to higher compliance scores in model outputs.

Contribution

It presents a novel multi-agentic approach that decouples task and constraint optimization, enhancing prompt effectiveness through iterative feedback.

Findings

Revised prompts achieve higher compliance scores.

Method improves adherence to formal constraints.

Effective across multiple LLMs.

Abstract

Large Language Models (LLMs) often generate substantively relevant content but fail to adhere to formal constraints, leading to outputs that are conceptually correct but procedurally flawed. Traditional prompt refinement approaches focus on rephrasing the description of the primary task an LLM has to perform, neglecting the granular constraints that function as acceptance criteria for its response. We propose a novel multi-agentic workflow that decouples optimization of the primary task description from its constraints, using quantitative scores as feedback to iteratively rewrite and improve them. Our evaluation demonstrates this method produces revised prompts that yield significantly higher compliance scores from models like Llama 3.1 8B and Mixtral-8x 7B.