LacaDM: A Latent Causal Diffusion Model for Multiobjective Reinforcement Learning

TL;DR

LacaDM introduces a latent causal diffusion model that improves multiobjective reinforcement learning by learning causal relationships, enabling better generalization and transfer across complex, dynamic environments.

Contribution

It presents a novel causal diffusion framework that enhances adaptability and knowledge transfer in multiobjective reinforcement learning scenarios.

Findings

Outperforms state-of-the-art baselines in hypervolume and utility metrics.

Demonstrates strong generalization in unseen environments.

Effectively balances conflicting objectives in complex tasks.

Abstract

Multiobjective reinforcement learning (MORL) poses significant challenges due to the inherent conflicts between objectives and the difficulty of adapting to dynamic environments. Traditional methods often struggle to generalize effectively, particularly in large and complex state-action spaces. To address these limitations, we introduce the Latent Causal Diffusion Model (LacaDM), a novel approach designed to enhance the adaptability of MORL in discrete and continuous environments. Unlike existing methods that primarily address conflicts between objectives, LacaDM learns latent temporal causal relationships between environmental states and policies, enabling efficient knowledge transfer across diverse MORL scenarios. By embedding these causal structures within a diffusion model-based framework, LacaDM achieves a balance between conflicting objectives while maintaining strong generalization capabilities in previously unseen environments. Empirical evaluations on various tasks from the MOGymnasium framework demonstrate that LacaDM consistently outperforms the state-of-art baselines in terms of hypervolume, sparsity, and expected utility maximization, showcasing its effectiveness in complex multiobjective tasks.