Feedback Driven Multi Stereo Vision System for Real-Time Event Analysis

TL;DR

This paper presents a real-time 3D stereo vision system that fuses multiple cameras for robust scene understanding, enabling applications like event recognition and subject tracking in complex environments.

Contribution

It introduces a novel multi-camera stereo vision pipeline with feedback mechanisms for adaptive scene analysis in interactive systems.

Findings

Effective multi-camera fusion for scene reconstruction

Preliminary success in event recognition and tracking

Roadmap for production deployment

Abstract

2D cameras are often used in interactive systems. Other systems like gaming consoles provide more powerful 3D cameras for short range depth sensing. Overall, these cameras are not reliable in large, complex environments. In this work, we propose a 3D stereo vision based pipeline for interactive systems, that is able to handle both ordinary and sensitive applications, through robust scene understanding. We explore the fusion of multiple 3D cameras to do full scene reconstruction, which allows for preforming a wide range of tasks, like event recognition, subject tracking, and notification. Using possible feedback approaches, the system can receive data from the subjects present in the environment, to learn to make better decisions, or to adapt to completely new environments. Throughout the paper, we introduce the pipeline and explain our preliminary experimentation and results. Finally, we draw the roadmap for the next steps that need to be taken, in order to get this pipeline into production