Common Benchmarks Undervalue the Generalization Power of Programmatic Policies

TL;DR

This paper challenges the belief that programmatic policies inherently generalize better than neural policies on out-of-distribution tasks, showing neural policies can perform equally well with simple training modifications.

Contribution

The paper demonstrates that neural policies can match programmatic policies' OOD generalization by simple training pipeline changes and advocates for new benchmarks emphasizing algorithmic reasoning.

Findings

Neural policies can generalize as effectively as programmatic policies on OOD problems.

Simple neural architecture modifications improve OOD generalization.

Reward function design influences policy safety and generalization.

Abstract

Algorithms for learning programmatic representations for sequential decision-making problems are often evaluated on out-of-distribution (OOD) problems, with the common conclusion that programmatic policies generalize better than neural policies on OOD problems. In this position paper, we argue that commonly used benchmarks undervalue the generalization capabilities of programmatic representations. We analyze the experiments of four papers from the literature and show that neural policies, which were shown not to generalize, can generalize as effectively as programmatic policies on OOD problems. This is achieved with simple changes in the neural policies training pipeline. Namely, we show that simpler neural architectures with the same type of sparse observation used with programmatic policies can help attain OOD generalization. Another modification we have shown to be effective is the use of reward functions that allow for safer policies (e.g., agents that drive slowly can generalize better). Also, we argue for creating benchmark problems highlighting concepts needed for OOD generalization that may challenge neural policies but align with programmatic representations, such as tasks requiring algorithmic constructs like stacks.