HierCVAE: Hierarchical Attention-Driven Conditional Variational Autoencoders for Multi-Scale Temporal Modeling

TL;DR

HierCVAE is a hierarchical attention-based variational autoencoder designed for multi-scale temporal modeling, effectively capturing complex dependencies and uncertainties in systems like energy consumption.

Contribution

It introduces a novel three-tier attention structure combined with conditional VAEs and ResFormer blocks for improved multi-scale temporal prediction.

Findings

15-40% improvement in prediction accuracy

Superior uncertainty calibration

Excels in long-term forecasting and multi-variate dependencies

Abstract

Temporal modeling in complex systems requires capturing dependencies across multiple time scales while managing inherent uncertainties. We propose HierCVAE, a novel architecture that integrates hierarchical attention mechanisms with conditional variational autoencoders to address these challenges. HierCVAE employs a three-tier attention structure (local, global, cross-temporal) combined with multi-modal condition encoding to capture temporal, statistical, and trend information. The approach incorporates ResFormer blocks in the latent space and provides explicit uncertainty quantification via prediction heads. Through evaluations on energy consumption datasets, HierCVAE demonstrates a 15-40% improvement in prediction accuracy and superior uncertainty calibration compared to state-of-the-art methods, excelling in long-term forecasting and complex multi-variate dependencies.