Bayesian deep learning of affordances from RGB images

TL;DR

This paper introduces a Bayesian deep learning approach using multiscale CNNs to predict affordances from RGB images, incorporating uncertainty estimation for more robust and reliable understanding of object interactions.

Contribution

It presents a novel Bayesian CNN model for affordance detection from RGB images that captures both aleatoric and epistemic uncertainties, improving robustness over deterministic models.

Findings

Bayesian model outperforms previous deterministic models in affordance detection.

Uncertainty estimates align with object types and scenarios.

Deep ensembles slightly outperform MC-dropout in performance metrics.

Abstract

Autonomous agents, such as robots or intelligent devices, need to understand how to interact with objects and its environment. Affordances are defined as the relationships between an agent, the objects, and the possible future actions in the environment. In this paper, we present a Bayesian deep learning method to predict the affordances available in the environment directly from RGB images. Based on previous work on socially accepted affordances, our model is based on a multiscale CNN that combines local and global information from the object and the full image. However, previous works assume a deterministic model, but uncertainty quantification is fundamental for robust detection, affordance-based reason, continual learning, etc. Our Bayesian model is able to capture both the aleatoric uncertainty from the scene and the epistemic uncertainty associated with the model and previous learning process. For comparison, we estimate the uncertainty using two state-of-the-art techniques: Monte Carlo dropout and deep ensembles. We also compare different types of CNN encoders for feature extraction. We have performed several experiments on an affordance database on socially acceptable behaviours and we have shown improved performance compared with previous works. Furthermore, the uncertainty estimation is consistent with the the type of objects and scenarios. Our results show a marginal better performance of deep ensembles, compared to MC-dropout on the Brier score and the Expected Calibration Error.