Higher-Order Action Regularization in Deep Reinforcement Learning: From Continuous Control to Building Energy Management

TL;DR

This paper explores higher-order derivative penalties in deep reinforcement learning to produce smoother control policies, improving real-world energy efficiency and equipment longevity in applications like building energy management.

Contribution

It introduces higher-order action regularization, especially third-order penalties, and demonstrates their effectiveness in continuous control and HVAC systems.

Findings

Third-order derivative penalties improve control smoothness.

Smooth policies reduce HVAC equipment switching by 60%.

Higher-order regularization balances performance and operational constraints.

Abstract

Deep reinforcement learning agents often exhibit erratic, high-frequency control behaviors that hinder real-world deployment due to excessive energy consumption and mechanical wear. We systematically investigate action smoothness regularization through higher-order derivative penalties, progressing from theoretical understanding in continuous control benchmarks to practical validation in building energy management. Our comprehensive evaluation across four continuous control environments demonstrates that third-order derivative penalties (jerk minimization) consistently achieve superior smoothness while maintaining competitive performance. We extend these findings to HVAC control systems where smooth policies reduce equipment switching by 60%, translating to significant operational benefits. Our work establishes higher-order action regularization as an effective bridge between RL optimization and operational constraints in energy-critical applications.