C-MOP: Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution

TL;DR

C-MOP introduces a novel framework combining boundary-aware contrastive sampling and momentum-guided clustering to stabilize and enhance prompt optimization for large language models, achieving superior performance over existing methods.

Contribution

The paper proposes C-MOP, a new prompt optimization framework that effectively stabilizes training and improves performance using boundary-aware sampling and semantic clustering with momentum guidance.

Findings

C-MOP outperforms SOTA baselines with 1.58% and 3.35% gains.

A 3B parameter LLM surpasses a 70B domain-specific dense LLM.

The framework demonstrates consistent improvements across multiple experiments.

Abstract

Automatic prompt optimization is a promising direction to boost the performance of Large Language Models (LLMs). However, existing methods often suffer from noisy and conflicting update signals. In this research, we propose C-MOP (Cluster-based Momentum Optimized Prompting), a framework that stabilizes optimization via Boundary-Aware Contrastive Sampling (BACS) and Momentum-Guided Semantic Clustering (MGSC). Specifically, BACS utilizes batch-level information to mine tripartite features--Hard Negatives, Anchors, and Boundary Pairs--to precisely characterize the typical representation and decision boundaries of positive and negative prompt samples. To resolve semantic conflicts, MGSC introduces a textual momentum mechanism with temporal decay that distills persistent consensus from fluctuating gradients across iterations. Extensive experiments demonstrate that C-MOP consistently outperforms SOTA baselines like PromptWizard and ProTeGi, yielding average gains of 1.58% and 3.35%. Notably, C-MOP enables a general LLM with 3B activated parameters to surpass a 70B domain-specific dense LLM, highlighting its effectiveness in driving precise prompt evolution. The code is available at https://github.com/huawei-noah/noah-research/tree/master/C-MOP.