Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL

TL;DR

This paper extends the CheckList testing methodology to planning-based reinforcement learning, enabling more comprehensive evaluation of an agent’s inference capabilities beyond standard value-based metrics.

Contribution

It introduces a CheckList approach for testing inference in planning-based RL agents, providing tools to identify reasoning flaws during tree search.

Findings

Effective in revealing previously unknown inference flaws

User study with AI researchers demonstrates practical utility

Provides insights into expert testing behaviors

Abstract

Reinforcement learning (RL) agents are commonly evaluated via their expected value over a distribution of test scenarios. Unfortunately, this evaluation approach provides limited evidence for post-deployment generalization beyond the test distribution. In this paper, we address this limitation by extending the recent CheckList testing methodology from natural language processing to planning-based RL. Specifically, we consider testing RL agents that make decisions via online tree search using a learned transition model and value function. The key idea is to improve the assessment of future performance via a CheckList approach for exploring and assessing the agent's inferences during tree search. The approach provides the user with an interface and general query-rule mechanism for identifying potential inference flaws and validating expected inference invariances. We present a user study involving knowledgeable AI researchers using the approach to evaluate an agent trained to play a complex real-time strategy game. The results show the approach is effective in allowing users to identify previously-unknown flaws in the agent's reasoning. In addition, our analysis provides insight into how AI experts use this type of testing approach, which may help improve future instantiations.