Closed-Loop CO2 Storage Control With History-Based Reinforcement Learning and Latent Model-Based Adaptation

TL;DR

This paper develops a deep reinforcement learning framework for adaptive, closed-loop CO2 storage management that effectively uses historical well data and latent models to handle reservoir uncertainties and failures.

Contribution

It introduces a history-conditioned RL policy and a latent model-based retuning method, improving adaptability and reducing reliance on online history matching.

Findings

History-conditioned policies nearly match privileged-state performance using only well data.

Latent model-based retuning outperforms direct model-free retuning in abnormal scenarios.

The framework offers a budget-aware alternative to online history matching for CO2 storage control.

Abstract

Closed-loop management of geological CO2 storage requires control policies that adapt to uncertain reservoir behavior while relying on observations that are realistically available during operation. This work formulates CO2 injection and brine-production control as a partially observable sequential decision problem and studies deployable deep reinforcement-learning controllers trained with high-fidelity reservoir simulation. We first compare privileged-state, well-only, history-conditioned, masking-curriculum, and asymmetric teacher-student model-free policies in order to quantify the value of temporal well-response information and training-time privileged simulator states. We then evaluate a latent model-based adaptation pipeline that reuses nominal latent dynamics and retunes controllers under known injector failure, leakage-induced dynamics and reward shift, and compartmentalized reservoir connectivity. The results show that history-conditioned policies recover nearly all of the privileged-state performance while using only deployable well-level information, and that latent model-based retuning outperforms direct model-free retuning under the same scenario-specific real-simulator budget in the abnormal operating cases. The proposed framework therefore provides a simulator-budget-aware alternative to repeated online history matching and re-optimization for closed-loop CO2 storage control.