Deep Reinforcement Learning for Field Development Optimization

TL;DR

This paper applies convolutional neural network-based deep reinforcement learning, specifically PPO, to optimize oil field development decisions, aiming for more robust and adaptable solutions compared to traditional evolutionary algorithms.

Contribution

It introduces CNN-based DRL algorithms for FDO, demonstrating their effectiveness and robustness over existing optimization methods.

Findings

CNN-based DRL policies outperform traditional methods in robustness.

PPO with CNN architectures achieves comparable or better results.

Deep reinforcement learning offers a promising approach for complex FDO problems.

Abstract

The field development optimization (FDO) problem represents a challenging mixed-integer nonlinear programming (MINLP) problem in which we seek to obtain the number of wells, their type, location, and drilling sequence that maximizes an economic metric. Evolutionary optimization algorithms have been effectively applied to solve the FDO problem, however, these methods provide only a deterministic (single) solution which are generally not robust towards small changes in the problem setup. In this work, the goal is to apply convolutional neural network-based (CNN) deep reinforcement learning (DRL) algorithms to the field development optimization problem in order to obtain a policy that maps from different states or representation of the underlying geological model to optimal decisions. The proximal policy optimization (PPO) algorithm is considered with two CNN architectures of varying number of layers and composition. Both networks obtained policies that provide satisfactory results when compared to a hybrid particle swarm optimization - mesh adaptive direct search (PSO-MADS) algorithm that has been shown to be effective at solving the FDO problem.