A Model-free Biomimetics Algorithm for Deterministic Partially Observable Markov Decision Process

TL;DR

This paper introduces BIOMAP, a model-free biomimetics algorithm that converts DET-POMDP problems into fully observable MDPs, improving decision-making under uncertainty with deceptive observations.

Contribution

The paper presents a novel model-free biomimetics algorithm, BIOMAP, for solving DET-POMDPs by transforming them into fully observable MDPs, enhancing robustness against environmental deception.

Findings

BIOMAP outperforms existing POMDP solvers in environments with deception.

The algorithm maintains operational effectiveness under uncertain and deceptive conditions.

Experimental results demonstrate BIOMAP's superior stability and environmental reparability.

Abstract

Partially Observable Markov Decision Process (POMDP) is a mathematical framework for modeling decision-making under uncertainty, where the agent's observations are incomplete and the underlying system dynamics are probabilistic. Solving the POMDP problem within the model-free paradigm is challenging for agents due to the inherent difficulty in accurately identifying and distinguishing between states and observations. We define such a difficult problem as a DETerministic Partially Observable Markov Decision Process (DET-POMDP) problem, which is a specific setting of POMDP. In this problem, states and observations are in a many-to-one relationship. The state is obscured, and its relationship is less apparent to the agent. This creates obstacles for the agent to infer the state through observations. To effectively address this problem, we convert DET-POMDP into a fully observable MDP using a model-free biomimetics algorithm called BIOMAP. BIOMAP is based on the MDP Graph Automaton framework to distinguish authentic environmental information from fraudulent data. Thus, it enhances the agent's ability to develop stable policies against DET-POMDP. The experimental results highlight the superior capabilities of BIOMAP in maintaining operational effectiveness and environmental reparability in the presence of environmental deceptions when compared with existing POMDP solvers. This research opens up new avenues for the deployment of reliable POMDP-based systems in fields that are particularly susceptible to DET-POMDP problems.