Automated co-design of high-performance thermodynamic cycles via graph-based hierarchical reinforcement learning

TL;DR

This paper presents a graph-based hierarchical reinforcement learning method for automated thermodynamic cycle design, achieving novel high-performance configurations surpassing classical designs in efficiency.

Contribution

It introduces a fully automated, scalable framework combining graph encoding, deep surrogate models, and hierarchical RL for thermodynamic cycle co-design, outperforming traditional methods.

Findings

Identified 18 novel heat pump cycles with 4.6% performance improvement.

Discovered 21 new heat engine cycles with 133.3% performance increase.

Successfully replicated classical cycle configurations.

Abstract

Thermodynamic cycles are pivotal in determining the efficacy of energy conversion systems. Traditional design methodologies, which rely on expert knowledge or exhaustive enumeration, are inefficient and lack scalability, thereby constraining the discovery of high-performance cycles. In this study, we introduce a graph-based hierarchical reinforcement learning approach for the co-design of structure parameters in thermodynamic cycles. These cycles are encoded as graphs, with components and connections depicted as nodes and edges, adhering to grammatical constraints. A deep learning-based thermophysical surrogate facilitates stable graph decoding and the simultaneous resolution of global parameters. Building on this foundation, we develop a hierarchical reinforcement learning framework wherein a high-level manager explores structural evolution and proposes candidate configurations, whereas a low-level worker optimizes parameters and provides performance rewards to steer the search towards high-performance regions. By integrating graph representation, thermophysical surrogate, and manager-worker learning, this method establishes a fully automated pipeline for encoding, decoding, and co-optimization. Using heat pump and heat engine cycles as case studies, the results demonstrate that the proposed method not only replicates classical cycle configurations but also identifies 18 and 21 novel heat pump and heat engine cycles, respectively. Relative to classical cycles, the novel configurations exhibit performance improvements of 4.6% and 133.3%, respectively, surpassing the traditional designs. This method effectively balances efficiency with broad applicability, providing a practical and scalable intelligent alternative to expert-driven thermodynamic cycle design.