# IVOA: Introspective Vision for Obstacle Avoidance

## TL;DR

This paper introduces IVOA, a method that uses a supervisory sensor to identify and predict failures in vision-based obstacle detection for autonomous robots, improving safety and reliability.

## Contribution

The paper presents a novel introspective approach that learns to predict vision perception failures by leveraging occasional supervisory signals and image patch analysis.

## Key findings

- IVOA accurately predicts vision perception failures in real-world scenarios.
- The approach works with different vision algorithms and environments.
- It improves obstacle avoidance reliability in autonomous robots.

## Abstract

Vision, as an inexpensive yet information rich sensor, is commonly used for perception on autonomous mobile robots. Unfortunately, accurate vision-based perception requires a number of assumptions about the environment to hold -- some examples of such assumptions, depending on the perception algorithm at hand, include purely lambertian surfaces, texture-rich scenes, absence of aliasing features, and refractive surfaces. In this paper, we present an approach for introspective vision for obstacle avoidance (IVOA) -- by leveraging a supervisory sensor that is occasionally available, we detect failures of stereo vision-based perception from divergence in plans generated by vision and the supervisory sensor. By projecting the 3D coordinates where the plans agree and disagree onto the images used for vision-based perception, IVOA generates a training set of reliable and unreliable image patches for perception. We then use this training dataset to learn a model of which image patches are likely to cause failures of the vision-based perception algorithm. Using this model, IVOA is then able to predict whether the relevant image patches in the observed images are likely to cause failures due to vision (both false positives and false negatives). We empirically demonstrate with extensive real-world data from both indoor and outdoor environments, the ability of IVOA to accurately predict the failures of two distinct vision algorithms.