A Framework for Event-based Computer Vision on a Mobile Device

TL;DR

This paper introduces the first Android framework for streaming data from event cameras to mobile devices, enabling low-power, real-time vision applications like gesture recognition and optical flow on smartphones.

Contribution

It presents a novel mobile framework that connects event cameras with Android devices, facilitating real-time event data processing for mobile computer vision tasks.

Findings

Framework achieves low latency and high throughput

Enables real-time gesture recognition and optical flow

Supports asynchronous event data processing on mobile devices

Abstract

We present the first publicly available Android framework to stream data from an event camera directly to a mobile phone. Today's mobile devices handle a wider range of workloads than ever before and they incorporate a growing gamut of sensors that make devices smarter, more user friendly and secure. Conventional cameras in particular play a central role in such tasks, but they cannot record continuously, as the amount of redundant information recorded is costly to process. Bio-inspired event cameras on the other hand only record changes in a visual scene and have shown promising low-power applications that specifically suit mobile tasks such as face detection, gesture recognition or gaze tracking. Our prototype device is the first step towards embedding such an event camera into a battery-powered handheld device. The mobile framework allows us to stream events in real-time and opens up the possibilities for always-on and on-demand sensing on mobile phones. To liaise the asynchronous event camera output with synchronous von Neumann hardware, we look at how buffering events and processing them in batches can benefit mobile applications. We evaluate our framework in terms of latency and throughput and show examples of computer vision tasks that involve both event-by-event and pre-trained neural network methods for gesture recognition, aperture robust optical flow and grey-level image reconstruction from events. The code is available at https://github.com/neuromorphic-paris/frog