Incremental learning abstract discrete planning domains and mappings to continuous perceptions

TL;DR

This paper introduces a formal framework and algorithm for agents to incrementally learn and update discrete planning domains and their perception mappings to continuous variables, enabling dynamic state learning and improved coherence.

Contribution

It presents a novel approach that integrates learning, planning, and acting, allowing agents to adapt their models over time with scalable performance.

Findings

Agents learn increasingly coherent models over time

The system scales to models with around 1 million states

Dynamic state and perception mapping learning improves planning accuracy

Abstract

Most of the works on planning and learning, e.g., planning by (model based) reinforcement learning, are based on two main assumptions: (i) the set of states of the planning domain is fixed; (ii) the mapping between the observations from the real word and the states is implicitly assumed or learned offline, and it is not part of the planning domain. Consequently, the focus is on learning the transitions between states. In this paper, we drop such assumptions. We provide a formal framework in which (i) the agent can learn dynamically new states of the planning domain; (ii) the mapping between abstract states and the perception from the real world, represented by continuous variables, is part of the planning domain; (iii) such mapping is learned and updated along the "life" of the agent. We define an algorithm that interleaves planning, acting, and learning, and allows the agent to update the planning domain depending on how much it trusts the model w.r.t. the new experiences learned by executing actions. We define a measure of coherence between the planning domain and the real world as perceived by the agent. We test our approach showing that the agent learns increasingly coherent models, and that the system can scale to deal with models with an order of $10^6$ states.