EXPIL: Explanatory Predicate Invention for Learning in Games

TL;DR

EXPIL introduces a method to automatically extract predicates from neural agents, enabling interpretable logic-based policies in games with less reliance on predefined background knowledge.

Contribution

The paper presents EXPIL, a novel approach for predicate invention that enhances interpretability and reduces background knowledge requirements in game learning agents.

Findings

Effective predicate extraction from neural agents.

Improved interpretability of logic-based agents.

Reduced need for predefined background knowledge.

Abstract

Reinforcement learning (RL) has proven to be a powerful tool for training agents that excel in various games. However, the black-box nature of neural network models often hinders our ability to understand the reasoning behind the agent's actions. Recent research has attempted to address this issue by using the guidance of pretrained neural agents to encode logic-based policies, allowing for interpretable decisions. A drawback of such approaches is the requirement of large amounts of predefined background knowledge in the form of predicates, limiting its applicability and scalability. In this work, we propose a novel approach, Explanatory Predicate Invention for Learning in Games (EXPIL), that identifies and extracts predicates from a pretrained neural agent, later used in the logic-based agents, reducing the dependency on predefined background knowledge. Our experimental evaluation on various games demonstrate the effectiveness of EXPIL in achieving explainable behavior in logic agents while requiring less background knowledge.