Approximate Dynamic Programming with Neural Networks in Linear Discrete Action Spaces

TL;DR

This paper introduces a hybrid approach combining neural networks and linear programming for approximate dynamic programming in high-dimensional, multi-period decision problems, demonstrating improved performance over polynomial value function approximations.

Contribution

It develops a novel integrated method embedding neural network VFAs into linear programs, enhancing flexibility and efficiency in complex optimization tasks.

Findings

Neural network VFAs outperform polynomial VFAs in experiments.

The method reduces manual design and tuning effort.

Numerical experiments validate the approach on transportation problems.

Abstract

Real-world problems of operations research are typically high-dimensional and combinatorial. Linear programs are generally used to formulate and efficiently solve these large decision problems. However, in multi-period decision problems, we must often compute expected downstream values corresponding to current decisions. When applying stochastic methods to approximate these values, linear programs become restrictive for designing value function approximations (VFAs). In particular, the manual design of a polynomial VFA is challenging. This paper presents an integrated approach for complex optimization problems, focusing on applications in the domain of operations research. It develops a hybrid solution method that combines linear programming and neural networks as part of approximate dynamic programming. Our proposed solution method embeds neural network VFAs into linear decision problems, combining the nonlinear expressive power of neural networks with the efficiency of solving linear programs. As a proof of concept, we perform numerical experiments on a transportation problem. The neural network VFAs consistently outperform polynomial VFAs, with limited design and tuning effort.