When a Reinforcement Learning Agent Encounters Unknown Unknowns

TL;DR

This paper introduces a new reinforcement learning framework, EMDP-GA, that enables agents to handle unknown unknown states by expanding value functions with noninformative beliefs, ensuring asymptotic optimality and manageable complexity.

Contribution

The paper proposes the EMDP-GA model with NIVE approach for reinforcement learning agents to adapt to unknown unknowns, with theoretical guarantees on regret and complexity.

Findings

Asymptotic regret matches state-of-the-art methods.

Computational and space complexity are comparable to existing approaches.

The method effectively discovers unknown unknowns with reasonable speed.

Abstract

An AI agent might surprisingly find she has reached an unknown state which she has never been aware of -- an unknown unknown. We mathematically ground this scenario in reinforcement learning: an agent, after taking an action calculated from value functions $Q$ and $V$ defined on the {\it {aware domain}}, reaches a state out of the domain. To enable the agent to handle this scenario, we propose an {\it episodic Markov decision {process} with growing awareness} (EMDP-GA) model, taking a new {\it noninformative value expansion} (NIVE) approach to expand value functions to newly aware areas: when an agent arrives at an unknown unknown, value functions $Q$ and $V$ whereon are initialised by noninformative beliefs -- the averaged values on the aware domain. This design is out of respect for the complete absence of knowledge in the newly discovered state. The upper confidence bound momentum Q-learning is then adapted to the growing awareness for training the EMDP-GA model. We prove that (1) the regret of our approach is asymptotically consistent with the state of the art (SOTA) without exposure to unknown unknowns in an extremely uncertain environment, and (2) our computational complexity and space complexity are comparable with the SOTA -- these collectively suggest that though an unknown unknown is surprising, it will be asymptotically properly discovered with decent speed and an affordable cost.