A Direct Approximation of AIXI Using Logical State Abstractions

TL;DR

This paper introduces a practical method to approximate AIXI using logical state abstractions, enabling reinforcement learning in complex, structured, and history-dependent environments with improved model selection and Bayesian learning.

Contribution

It combines logical state abstraction with AIXI, using higher-order logic and a generalized Context Tree Weighting for Bayesian model learning, expanding AIXI's applicability to complex environments.

Findings

Validated on epidemic control in large contact networks

Demonstrated effective state abstraction and model selection

Achieved scalable Bayesian model learning

Abstract

We propose a practical integration of logical state abstraction with AIXI, a Bayesian optimality notion for reinforcement learning agents, to significantly expand the model class that AIXI agents can be approximated over to complex history-dependent and structured environments. The state representation and reasoning framework is based on higher-order logic, which can be used to define and enumerate complex features on non-Markovian and structured environments. We address the problem of selecting the right subset of features to form state abstractions by adapting the $\Phi$-MDP optimisation criterion from state abstraction theory. Exact Bayesian model learning is then achieved using a suitable generalisation of Context Tree Weighting over abstract state sequences. The resultant architecture can be integrated with different planning algorithms. Experimental results on controlling epidemics on large-scale contact networks validates the agent's performance.