On-Line Policy Iteration with Trajectory-Driven Policy Generation

TL;DR

This paper introduces an online policy iteration method that generates a sequence of improving policies using trajectory data, suitable for real-time applications like path planning and combinatorial optimization.

Contribution

The paper presents a novel trajectory-driven policy iteration framework that ensures monotonic cost improvement and can be used for training neural network policies in real-time.

Findings

The method achieves monotonic cost improvement in deterministic finite-horizon problems.

Computational studies demonstrate effectiveness in path planning and combinatorial optimization.

The framework can be extended to stochastic settings and multi-agent problems.

Abstract

We consider deterministic finite-horizon optimal control problems with a fixed initial state. We introduce an on-line policy iteration method, which, starting from a given policy, however obtained, generates a sequence of cost-improving policies and corresponding trajectories. Each policy produces a trajectory, which is used in turn to generate data for training the next policy. The method is motivated by problems that are repeatedly solved starting from the same initial state, including discrete optimization and path planning for repetitive tasks. For such problems, the method is fast enough to be used on-line. Under a natural consistency condition, we show that the sequence of costs of the generated policies is monotonically improving for the given initial state (but not necessarily for other states). We illustrate our results with computational studies from combinatorial optimization and 3-dimensional path planning for drones {and a robot arm} in the presence of obstacles. We also discuss briefly a stochastic counterpart of our algorithm. Our proposed framework combines elements of rollout and policy iteration with flexible trajectory-based policy representations, and applies to problems involving a single as well as multiple decision makers. It also provides a principled way to train neural network-based policies using trajectory data, while preserving monotonic cost improvement.