Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off

TL;DR

This paper investigates how the choice of temporal resolution affects the efficiency of value estimation in continuous-time control systems, revealing a fundamental trade-off that can be optimized for better data-efficiency.

Contribution

It provides a theoretical analysis of the trade-off between approximation and statistical errors in value estimation due to time discretization in RL, and demonstrates its practical implications.

Findings

Optimal temporal resolution balances approximation and statistical errors.

Managing time discretization improves policy evaluation efficiency.

Trade-off is validated through simulations and benchmarks.

Abstract

A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.