# Minimizing the Negative Side Effects of Planning with Reduced Models

## TL;DR

This paper introduces a planning approach that uses multiple reduced models to minimize risky replanning in large Markov decision processes, improving safety and efficiency.

## Contribution

It proposes a portfolio-based planning paradigm that alternates between outcome selection approaches to reduce negative side effects of reduced models.

## Key findings

- Effective in three domains, including real-world electric vehicle charging data.
- Reduces risky replanning by selectively choosing safe states for replanning.
- Improves planning safety without fully reverting to full models.

## Abstract

Reduced models of large Markov decision processes accelerate planning by considering a subset of outcomes for each state-action pair. This reduction in reachable states leads to replanning when the agent encounters states without a precomputed action during plan execution. However, not all states are suitable for replanning. In the worst case, the agent may not be able to reach the goal from the newly encountered state. Agents should be better prepared to handle such risky situations and avoid replanning in risky states. Hence, we consider replanning in states that are unsafe for deliberation as a negative side effect of planning with reduced models. While the negative side effects can be minimized by always using the full model, this defeats the purpose of using reduced models. The challenge is to plan with reduced models, but somehow account for the possibility of encountering risky situations. An agent should thus only replan in states that the user has approved as safe for replanning. To that end, we propose planning using a portfolio of reduced models, a planning paradigm that minimizes the negative side effects of planning using reduced models by alternating between different outcome selection approaches. We empirically demonstrate the effectiveness of our approach on three domains: an electric vehicle charging domain using real-world data from a university campus and two benchmark planning problems.