When Remembering and Planning are Worth it: Navigating under Change

TL;DR

This paper investigates how memory and learning strategies can enhance spatial navigation in dynamic, uncertain environments, emphasizing the importance of multi-strategy architectures for efficient exploration and planning.

Contribution

It introduces a multi-strategy navigation architecture that combines memory updating and planning, improving efficiency in non-stationary environments with limited sensing.

Findings

Multi-strategy architectures outperform simple agents in complex tasks.

Non-stationary probability learning improves memory relevance over time.

Efficiency gains are significant when environmental uncertainty is moderate.

Abstract

We explore how different types and uses of memory can aid spatial navigation in changing uncertain environments. In the simple foraging task we study, every day, our agent has to find its way from its home, through barriers, to food. Moreover, the world is non-stationary: from day to day, the location of the barriers and food may change, and the agent's sensing such as its location information is uncertain and very limited. Any model construction, such as a map, and use, such as planning, needs to be robust against these challenges, and if any learning is to be useful, it needs to be adequately fast. We look at a range of strategies, from simple to sophisticated, with various uses of memory and learning. We find that an architecture that can incorporate multiple strategies is required to handle (sub)tasks of a different nature, in particular for exploration and search, when food location is not known, and for planning a good path to a remembered (likely) food location. An agent that utilizes non-stationary probability learning techniques to keep updating its (episodic) memories and that uses those memories to build maps and plan on the fly (imperfect maps, i.e. noisy and limited to the agent's experience) can be increasingly and substantially more efficient than the simpler (minimal-memory) agents, as the task difficulties such as distance to goal are raised, as long as the uncertainty, from localization and change, is not too large.