Reading in the Dark with Foveated Event Vision

TL;DR

This paper introduces a novel event-based OCR system for smart glasses that leverages eye gaze and event cameras to read text in challenging conditions, significantly reducing bandwidth and outperforming traditional methods.

Contribution

It presents a new event-based OCR approach using eye gaze and deep binary reconstruction, achieving high performance in low-light and fast-motion scenarios with minimal bandwidth.

Findings

Reads text in low-light environments where RGB cameras fail

Reduces bandwidth by approximately 98% compared to RGB cameras

Achieves up to 2400 times less bandwidth usage

Abstract

Current smart glasses equipped with RGB cameras struggle to perceive the environment in low-light and high-speed motion scenarios due to motion blur and the limited dynamic range of frame cameras. Additionally, capturing dense images with a frame camera requires large bandwidth and power consumption, consequently draining the battery faster. These challenges are especially relevant for developing algorithms that can read text from images. In this work, we propose a novel event-based Optical Character Recognition (OCR) approach for smart glasses. By using the eye gaze of the user, we foveate the event stream to significantly reduce bandwidth by around 98% while exploiting the benefits of event cameras in high-dynamic and fast scenes. Our proposed method performs deep binary reconstruction trained on synthetic data and leverages multimodal LLMs for OCR, outperforming traditional OCR solutions. Our results demonstrate the ability to read text in low light environments where RGB cameras struggle while using up to 2400 times less bandwidth than a wearable RGB camera.