Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

TL;DR

This paper introduces Predicate Action Skills (PACTS), a joint generative model for actions and symbolic outcomes that enables robots to perform zero-shot skill composition through planning.

Contribution

It proposes a novel joint modeling approach for actions and predicates, allowing zero-shot skill composition and improved internal representations.

Findings

PACTS can generate coherent action and predicate trajectories.

Zero-shot skill composition is achieved via planning with predicate predictions.

The approach improves both action generation and predicate classification accuracy.

Abstract

Learning from Demonstration (LfD) enables robots to learn complex behaviors from expert examples, yet existing approaches often fail to generalize to new compositions of known skills without retraining. Modern generative policies model distributions over action trajectories alone, thus are unable to reason about the symbolic outcomes required for robust composition. We propose that skills should jointly model action trajectories and the symbolic outcomes they induce. To address this gap, we introduce Predicate Action Skills (PACTS), a class of closed-loop visuomotor policies that model skills as a joint generative process over action and predicate belief trajectories, producing coherent action-outcome rollouts within a single model. Jointly generating actions and predicates enables PACTS to learn internal representations that improve both action generation and predicate classification. Furthermore, we demonstrate zero-shot composition of learned skills via planning by leveraging online predicate predictions from PACTS as a symbolic interface for sequencing and monitoring execution. Project website: https://planpacts.github.io/