Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation

TL;DR

This paper introduces a liveness-based Bellman operator for offline policy evaluation in robotic manipulation, effectively addressing finite-horizon truncation bias and improving task progress estimation.

Contribution

It proposes a novel framework that interprets policy evaluation as a task-completion problem, providing theoretical guarantees and practical improvements over classical methods.

Findings

The method reduces truncation bias in finite-horizon evaluations.

It outperforms classical baselines like TD(0) and Monte Carlo methods.

Empirical results show improved accuracy in reflecting task progress.

Abstract

Policy evaluation is a fundamental component of the development and deployment pipeline for robotic policies. In modern manipulation systems, this problem is particularly challenging: rewards are often sparse, task progression of evaluation rollouts are often non-monotonic as the policies exhibit recovery behaviors, and evaluation rollouts are necessarily of finite length. This finite length introduces truncation bias, breaking the infinite-horizon assumptions underlying standard methods relying on Bellman equations/principle of optimality. In this work, we propose a framework for offline policy evaluation from sparse rewards based on a liveness-based Bellman operator. Our formulation interprets policy evaluation as a task-completion problem and yields a conservative fixed-point value function that is robust to finite-horizon truncation. We analyze the theoretical properties of the proposed operator, including contraction guarantees, and show how it encodes task progression while mitigating truncation bias. We evaluate our method on two simulated manipulation tasks using both a Vision-Language-Action model and a diffusion policy, and a cloth folding task using human demonstrations. Empirical results demonstrate that our approach more accurately reflects task progress and substantially reduces truncation bias, outperforming classical baselines such as TD(0) and Monte Carlo policy evaluation.