Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification

TL;DR

This paper introduces a principle-based multi-agent prompting strategy for text classification that improves performance and efficiency over traditional methods by generating and consolidating guiding principles through multiple LLM agents.

Contribution

It proposes a novel multi-agent prompting framework that automatically generates and consolidates principles to enhance text classification accuracy and reduce inference costs.

Findings

Achieves 1.55% - 19.37% performance gains over zero-shot prompting.

Outperforms other baselines like CoT and stepback prompting.

Generated principles outperform human-crafted ones on private datasets.

Abstract

We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.