Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation

TL;DR

This paper introduces a multi-agent framework for dynamically evaluating large language models by creating evolving benchmark instances, revealing more nuanced performance insights and limitations.

Contribution

The novel framework uses multi-agent operations to generate evolving benchmark instances, enabling scalable, robust, and fine-grained evaluation of LLMs' capabilities.

Findings

Most LLMs show performance decline against original results.

Evaluation reveals wider performance gaps between models and tasks.

Framework provides more accurate and detailed model assessments.

Abstract

This paper presents a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models (LLMs), aiming for a more accurate assessment of their capabilities and limitations. We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence that dynamically extend existing benchmarks. Towards a more scalable, robust and fine-grained evaluation, we implement six reframing operations to construct evolving instances testing LLMs against diverse queries, data noise and probing their problem-solving sub-abilities. With this framework, we extend benchmark datasets of four tasks. Experimental results show a general performance decline in most LLMs against their original results. This decline under our scalable and robust evaluations, alongside our fine-grained evaluation, more accurately reflect models' capabilities. Besides, our framework widens performance discrepancies both between different models and within the same model across various tasks, facilitating more informed model selection for specific tasks (Code and data are available at https://github.com/NanshineLoong/Self-Evolving-Benchmark).