Reinforcement Learning and Tree Search Methods for the Unit Commitment Problem

TL;DR

This paper applies reinforcement learning combined with tree search to the unit commitment problem in power systems, demonstrating cost reductions and robustness improvements over traditional methods, especially under uncertainty and generator outages.

Contribution

It introduces guided tree search, a novel hybrid RL and planning approach, for large-scale unit commitment problems with real data, showing competitive performance and enhanced robustness.

Findings

Cost reduction of up to 1.4% compared to deterministic methods.

Over 2% savings in operating costs considering generator outages.

Framework easily extends to incorporate robustness factors like wind and carbon prices.

Abstract

The unit commitment (UC) problem, which determines operating schedules of generation units to meet demand, is a fundamental task in power systems operation. Existing UC methods using mixed-integer programming are not well-suited to highly stochastic systems. Approaches which more rigorously account for uncertainty could yield large reductions in operating costs by reducing spinning reserve requirements; operating power stations at higher efficiencies; and integrating greater volumes of variable renewables. A promising approach to solving the UC problem is reinforcement learning (RL), a methodology for optimal decision-making which has been used to conquer long-standing grand challenges in artificial intelligence. This thesis explores the application of RL to the UC problem and addresses challenges including robustness under uncertainty; generalisability across multiple problem instances; and scaling to larger power systems than previously studied. To tackle these issues, we develop guided tree search, a novel methodology combining model-free RL and model-based planning. The UC problem is formalised as a Markov decision process and we develop an open-source environment based on real data from Great Britain's power system to train RL agents. In problems of up to 100 generators, guided tree search is shown to be competitive with deterministic UC methods, reducing operating costs by up to 1.4\%. An advantage of RL is that the framework can be easily extended to incorporate considerations important to power systems operators such as robustness to generator failure, wind curtailment or carbon prices. When generator outages are considered, guided tree search saves over 2\% in operating costs as compared with methods using conventional $N-x$ reserve criteria.