Learning to Look: Seeking Information for Decision Making via Policy Factorization

TL;DR

This paper introduces DISaM, a dual-policy approach for robotic tasks requiring active information seeking, enabling better exploration and exploitation in manipulation tasks through a factorized decision-making process.

Contribution

The paper proposes a novel dual-policy framework for factorized Contextual Markov Decision Processes, separating exploration and exploitation for improved robotic manipulation.

Findings

DISaM outperforms existing methods in five manipulation tasks

Effective separation of information-seeking and exploitation policies

Successful real-world robotic experiments demonstrating approach's practicality

Abstract

Many robot manipulation tasks require active or interactive exploration behavior in order to be performed successfully. Such tasks are ubiquitous in embodied domains, where agents must actively search for the information necessary for each stage of a task, e.g., moving the head of the robot to find information relevant to manipulation, or in multi-robot domains, where one scout robot may search for the information that another robot needs to make informed decisions. We identify these tasks with a new type of problem, factorized Contextual Markov Decision Processes, and propose DISaM, a dual-policy solution composed of an information-seeking policy that explores the environment to find the relevant contextual information and an information-receiving policy that exploits the context to achieve the manipulation goal. This factorization allows us to train both policies separately, using the information-receiving one to provide reward to train the information-seeking policy. At test time, the dual agent balances exploration and exploitation based on the uncertainty the manipulation policy has on what the next best action is. We demonstrate the capabilities of our dual policy solution in five manipulation tasks that require information-seeking behaviors, both in simulation and in the real-world, where DISaM significantly outperforms existing methods. More information at https://robin-lab.cs.utexas.edu/learning2look/.