ElectriQ: A Benchmark for Assessing the Response Capability of Large Language Models in Power Marketing

TL;DR

ElectriQ introduces a comprehensive benchmark and evaluation framework for assessing large language models in electric power marketing, emphasizing sector-specific knowledge, regulatory reasoning, and multi-turn dialogue stability.

Contribution

The paper presents ElectriQ, a large-scale sector-specific benchmark for LLMs in power marketing, and proposes SEEK-RAG, a retrieval-augmented finetuning method to enhance model performance and compliance.

Findings

7B models with SEEK-RAG outperform larger models in EPM tasks.

ElectriQ's benchmark includes over 550k dialogues and multiple evaluation metrics.

Domain-aligned models reduce computational costs while maintaining performance.

Abstract

As power systems decarbonise and digitalise, high penetrations of distributed energy resources and flexible tariffs make electric power marketing (EPM) a key interface between regulation, system operation and sustainable-energy deployment. Many utilities still rely on human agents and rule- or intent-based chatbots with fragmented knowledge bases that struggle with long, cross-scenario dialogues and fall short of requirements for compliant, verifiable and DR-ready interactions. Meanwhile, frontier large language models (LLMs) show strong conversational ability but are evaluated on generic benchmarks that underweight sector-specific terminology, regulatory reasoning and multi-turn process stability. To address this gap, we present ElectriQ, a large-scale benchmark and evaluation framework for LLMs in EPM. ElectriQ contains over 550k dialogues across six service domains and 24 sub-scenarios and defines a unified protocol that combines human ratings, automatic metrics and two compliance stress tests-Statutory Citation Correctness and Long-Dialogue Consistency. Building on ElectriQ, we propose SEEK-RAG, a retrieval-augmented method that injects policy and domain knowledge during finetuning and inference. Experiments on 13 LLMs show that domain-aligned 7B models with SEEK-RAG match or surpass much larger models while reducing computational cost, providing an auditable, regulation-aware basis for deploying LLM-based EPM assistants that support demand-side management, renewable integration and resilient grid operation.