Simple Policy Evaluation for Data-Rich Iterative Tasks

TL;DR

This paper introduces a model-free, data-driven policy evaluation method for iterative control tasks that constructs a piecewise affine value function approximation from safe trajectories, ensuring constraint satisfaction and performance bounds.

Contribution

It presents a novel, data-based policy evaluation approach that is applicable to iterative tasks, reducing computation time significantly while maintaining performance.

Findings

Reduces computation time by an order of magnitude.

Guarantees constraint satisfaction and performance bounds.

Effective in simulations and real-world experiments on autonomous race car.

Abstract

A data-based policy for iterative control task is presented. The proposed strategy is model-free and can be applied whenever safe input and state trajectories of a system performing an iterative task are available. These trajectories, together with a user-defined cost function, are exploited to construct a piecewise affine approximation to the value function. Approximated value functions are then used to evaluate the control policy by solving a linear program. We show that for linear system subject to convex cost and constraints, the proposed strategy guarantees closed-loop constraint satisfaction and performance bounds on the closed-loop trajectory. We evaluate the proposed strategy in simulations and experiments, the latter carried out on the Berkeley Autonomous Race Car (BARC) platform. We show that the proposed strategy is able to reduce the computation time by one order of magnitude while achieving the same performance as our model-based control algorithm.