Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations

TL;DR

This paper demonstrates that expert demonstrations from steady-state wake models can significantly accelerate reinforcement learning for wind farm control, reducing training time and improving initial performance.

Contribution

The study introduces a pretraining method using expert demonstrations to initialize RL agents, enhancing early performance and convergence speed in wind farm control tasks.

Findings

Pretraining eliminates the costly initial learning phase.

Pretrained agents start near baseline performance, outperforming untrained agents.

All agents eventually surpass lookup-table controllers after fine-tuning.

Abstract

Reinforcement learning (RL) offers a promising approach for adaptive wind farm flow control, yet its practical deployment is hindered by slow training convergence and poor initial performance, factors that could translate to years of reduced power output if an untrained agent were deployed directly. This work investigates whether domain knowledge from steady-state wake models can accelerate RL training and improve initial controller performance. We propose a pretraining methodology in which expert demonstrations are generated by deploying a PyWake-based steady-state optimizer within a dynamic wake simulation (WindGym), then used to initialize both the actor and critic networks of a Soft Actor-Critic agent via behavior cloning. Experiments on a 2x2 wind farm show that pretraining eliminates the costly initial learning phase: while an untrained agent underperforms the greedy zero-yaw baseline by approximately 12%, pretraining raises initial performance to near-baseline levels. During online fine-tuning, all configurations converge within 250,000 environment steps to achieve similar performance, ultimately exceeding that of a lookup-table controller, which reaches approximately 7% power gain after 500,000 steps.