# Joint Perception and Control as Inference with an Object-based Implementation

## TL;DR

This paper introduces a unified Bayesian inference framework called PCI that integrates perception and control in partially observable environments, emphasizing object-based representations for improved decision-making.

## Contribution

It proposes Object-based Perception Control (OPC), an unsupervised end-to-end method that automatically discovers object representations to enhance control in complex environments.

## Key findings

- OPC achieves high-quality perceptual grouping.
- OPC outperforms strong baselines in accumulated rewards.
- The perception model converges effectively during training.

## Abstract

Existing model-based reinforcement learning methods often study perception modeling and decision making separately. We introduce joint Perception and Control as Inference (PCI), a general framework to combine perception and control for partially observable environments through Bayesian inference. Based on the fact that object-level inductive biases are critical in human perceptual learning and reasoning, we propose Object-based Perception Control (OPC), an instantiation of PCI which manages to facilitate control using automatic discovered object-based representations. We develop an unsupervised end-to-end solution and analyze the convergence of the perception model update. Experiments in a high-dimensional pixel environment demonstrate the learning effectiveness of our object-based perception control approach. Specifically, we show that OPC achieves good perceptual grouping quality and outperforms several strong baselines in accumulated rewards.