Causal prompting model-based offline reinforcement learning

TL;DR

This paper introduces CPRL, a novel framework for offline reinforcement learning that effectively handles noisy, diverse datasets and generalizes across tasks using causal prompts and skill reuse, demonstrated on real-world medical data.

Contribution

The paper proposes the CPRL framework with Hip-BCPD for modeling dynamics and a skill-reuse strategy, advancing robustness and generalization in offline RL for online systems.

Findings

Outperforms existing algorithms in noisy, out-of-distribution environments

Effectively models environmental dynamics with Hip-BCPD

Enables multi-task learning through skill reuse

Abstract

Model-based offline Reinforcement Learning (RL) allows agents to fully utilise pre-collected datasets without requiring additional or unethical explorations. However, applying model-based offline RL to online systems presents challenges, primarily due to the highly suboptimal (noise-filled) and diverse nature of datasets generated by online systems. To tackle these issues, we introduce the Causal Prompting Reinforcement Learning (CPRL) framework, designed for highly suboptimal and resource-constrained online scenarios. The initial phase of CPRL involves the introduction of the Hidden-Parameter Block Causal Prompting Dynamic (Hip-BCPD) to model environmental dynamics. This approach utilises invariant causal prompts and aligns hidden parameters to generalise to new and diverse online users. In the subsequent phase, a single policy is trained to address multiple tasks through the amalgamation of reusable skills, circumventing the need for training from scratch. Experiments conducted across datasets with varying levels of noise, including simulation-based and real-world offline datasets from the Dnurse APP, demonstrate that our proposed method can make robust decisions in out-of-distribution and noisy environments, outperforming contemporary algorithms. Additionally, we separately verify the contributions of Hip-BCPDs and the skill-reuse strategy to the robustness of performance. We further analyse the visualised structure of Hip-BCPD and the interpretability of sub-skills. We released our source code and the first ever real-world medical dataset for precise medical decision-making tasks.