Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

TL;DR

This paper introduces DROP, a non-iterative offline RL paradigm that separates value estimation and policy extraction, enabling safe, adaptive policy deployment during testing with improved performance.

Contribution

DROP proposes a novel non-iterative bi-level offline RL framework that answers key questions about information transfer and safe exploitation, with a focus on model-based optimization and adaptive inference.

Findings

DROP achieves comparable or better performance than prior methods.

DROP enables safe and adaptive policy deployment during testing.

The method effectively decomposes data and learns a conservative score model.

Abstract

In this work, we decouple the iterative bi-level offline RL (value estimation and policy extraction) from the offline training phase, forming a non-iterative bi-level paradigm and avoiding the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us to conduct inner-level optimization (value estimation) in training, while performing outer-level optimization (policy extraction) in testing. Naturally, such a paradigm raises three core questions that are not fully answered by prior non-iterative offline RL counterparts like reward-conditioned policy: (q1) What information should we transfer from the inner-level to the outer-level? (q2) What should we pay attention to when exploiting the transferred information for safe/confident outer-level optimization? (q3) What are the benefits of concurrently conducting outer-level optimization during testing? Motivated by model-based optimization (MBO), we propose DROP (design from policies), which fully answers the above questions. Specifically, in the inner-level, DROP decomposes offline data into multiple subsets, and learns an MBO score model (a1). To keep safe exploitation to the score model in the outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (a2). During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states (a3). Empirically, we evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.