ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models

TL;DR

ElecBench is a comprehensive evaluation benchmark designed to assess large language models' performance in the power sector across various professional and general scenarios, aiming to facilitate technological progress and application.

Contribution

This paper introduces ElecBench, the first specialized benchmark for evaluating LLMs in the power sector, covering sector-specific scenarios and multiple performance metrics.

Findings

Evaluated 8 LLMs across diverse scenarios and metrics.

Provided a public test set for transparent benchmarking.

Identified strengths and limitations of current LLMs in power applications.

Abstract

In response to the urgent demand for grid stability and the complex challenges posed by renewable energy integration and electricity market dynamics, the power sector increasingly seeks innovative technological solutions. In this context, large language models (LLMs) have become a key technology to improve efficiency and promote intelligent progress in the power sector with their excellent natural language processing, logical reasoning, and generalization capabilities. Despite their potential, the absence of a performance evaluation benchmark for LLM in the power sector has limited the effective application of these technologies. Addressing this gap, our study introduces "ElecBench", an evaluation benchmark of LLMs within the power sector. ElecBench aims to overcome the shortcomings of existing evaluation benchmarks by providing comprehensive coverage of sector-specific scenarios, deepening the testing of professional knowledge, and enhancing decision-making precision. The framework categorizes scenarios into general knowledge and professional business, further divided into six core performance metrics: factuality, logicality, stability, security, fairness, and expressiveness, and is subdivided into 24 sub-metrics, offering profound insights into the capabilities and limitations of LLM applications in the power sector. To ensure transparency, we have made the complete test set public, evaluating the performance of eight LLMs across various scenarios and metrics. ElecBench aspires to serve as the standard benchmark for LLM applications in the power sector, supporting continuous updates of scenarios, metrics, and models to drive technological progress and application.