What do navigation agents learn about their environment?

TL;DR

This paper introduces iSEE, a system for interpreting embodied AI navigation agents, revealing how they encode environmental and agent-specific information, and demonstrating the impact of individual neurons on their behavior.

Contribution

The paper presents iSEE, a novel interpretability system for embodied AI navigation agents, providing insights into their internal representations and behaviors.

Findings

Agents encode reachable locations to avoid obstacles

Visibility of the target is represented internally

Masking critical neurons significantly alters agent behavior

Abstract

Today's state of the art visual navigation agents typically consist of large deep learning models trained end to end. Such models offer little to no interpretability about the learned skills or the actions of the agent taken in response to its environment. While past works have explored interpreting deep learning models, little attention has been devoted to interpreting embodied AI systems, which often involve reasoning about the structure of the environment, target characteristics and the outcome of one's actions. In this paper, we introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal and Object Goal navigation agents. We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment. We demonstrate interesting insights about navigation agents using iSEE, including the ability to encode reachable locations (to avoid obstacles), visibility of the target, progress from the initial spawn location as well as the dramatic effect on the behaviors of agents when we mask out critical individual neurons. The code is available at: https://github.com/allenai/iSEE