Hierarchical Control Framework Integrating LLMs with RL for Decarbonized HVAC Operation

TL;DR

This paper introduces a hierarchical control framework combining LLMs and RL to improve energy efficiency and comfort in decarbonized multi-zone HVAC systems, addressing exploration and action space challenges.

Contribution

It presents a novel hierarchical approach where LLMs generate feasible action masks to enhance RL efficiency in HVAC control, validated with real-world building data.

Findings

Achieved 39.1% reduction in comfort deviation compared to DQN.

Reduced HVAC energy use to 140.90 kWh, outperforming vanilla RL baselines.

Demonstrated LLM-guided masking improves exploration and stability.

Abstract

Heating, ventilation, and air conditioning (HVAC) systems account for a substantial share of building energy consumption. Environmental uncertainty and dynamic occupancy behavior bring challenges in decarbonized HVAC control. Reinforcement learning (RL) can optimize long-horizon comfort-energy trade-offs but suffers from exponential action-space growth and inefficient exploration in multi-zone buildings. Large language models (LLMs) can encode semantic context and operational knowledge, yet when used alone they lack reliable closed-loop numerical optimization and may result in less reliable comfort-energy trade-offs. To address these limitations, we propose a hierarchical control framework in which a fine-tuned LLM, trained on historical building operation data, generates state-dependent feasible action masks that prune the combinatorial joint action space into operationally plausible subsets. A masked value-based RL agent then performs constrained optimization within this reduced space, improving exploration efficiency and training stability. Evaluated in a high-fidelity simulator calibrated with real-world sensor and occupancy data from a 7-zone office building, the proposed method achieves a mean PPD of 7.30%, corresponding to reductions of 39.1% relative to DQN, the best vanilla RL baseline in comfort, and 53.1% relative to the best vanilla LLM baseline, while reducing daily HVAC energy use to 140.90~kWh, lower than all vanilla RL baselines. The results suggest that LLM-guided action masking is a promising pathway toward efficient multi-zone HVAC control.