BEAVER: Building Environments with Assessable Variation for Evaluating Multi-Objective Reinforcement Learning

TL;DR

This paper introduces BEAVER, a benchmark environment for evaluating multi-objective reinforcement learning in building energy management, highlighting challenges in policy generalization across diverse building dynamics.

Contribution

It formalizes the multi-objective contextual RL problem for building management and provides a benchmark to assess algorithm generalization in varied operational scenarios.

Findings

Existing multi-objective RL methods achieve reasonable trade-offs.

Performance degrades with environment variations.

Incorporating dynamics-dependent context improves policy robustness.

Abstract

Recent years have seen significant advancements in designing reinforcement learning (RL)-based agents for building energy management. While individual success is observed in simulated or controlled environments, the scalability of RL approaches in terms of efficiency and generalization across building dynamics and operational scenarios remains an open question. In this work, we formally characterize the generalization space for the cross-environment, multi-objective building energy management task, and formulate the multi-objective contextual RL problem. Such a formulation helps understand the challenges of transferring learned policies across varied operational contexts such as climate and heat convection dynamics under multiple control objectives such as comfort level and energy consumption. We provide a principled way to parameterize such contextual information in realistic building RL environments, and construct a novel benchmark to facilitate the evaluation of generalizable RL algorithms in practical building control tasks. Our results show that existing multi-objective RL methods are capable of achieving reasonable trade-offs between conflicting objectives. However, their performance degrades under certain environment variations, underscoring the importance of incorporating dynamics-dependent contextual information into the policy learning process.