DeepLight: Robust & Unobtrusive Real-time Screen-Camera Communication for Real-World Displays

TL;DR

DeepLight presents a machine learning-based system for robust, human-imperceptible real-time screen-camera communication that works reliably in real-world conditions, overcoming limitations of previous methods.

Contribution

The paper introduces DeepLight, a novel DNN-based decoder and encoding scheme that enhances robustness and imperceptibility in screen-camera communication under diverse conditions.

Findings

Achieves high decoding accuracy with frame error rate < 0.2

Supports data rates of at least 0.95 Kbps

Maintains IoU >= 83% for screen extraction

Abstract

The paper introduces a novel, holistic approach for robust Screen-Camera Communication (SCC), where video content on a screen is visually encoded in a human-imperceptible fashion and decoded by a camera capturing images of such screen content. We first show that state-of-the-art SCC techniques have two key limitations for in-the-wild deployment: (a) the decoding accuracy drops rapidly under even modest screen extraction errors from the captured images, and (b) they generate perceptible flickers on common refresh rate screens even with minimal modulation of pixel intensity. To overcome these challenges, we introduce DeepLight, a system that incorporates machine learning (ML) models in the decoding pipeline to achieve humanly-imperceptible, moderately high SCC rates under diverse real-world conditions. Deep-Light's key innovation is the design of a Deep Neural Network (DNN) based decoder that collectively decodes all the bits spatially encoded in a display frame, without attempting to precisely isolate the pixels associated with each encoded bit. In addition, DeepLight supports imperceptible encoding by selectively modulating the intensity of only the Blue channel, and provides reasonably accurate screen extraction (IoU values >= 83%) by using state-of-the-art object detection DNN pipelines. We show that a fully functional DeepLight system is able to robustly achieve high decoding accuracy (frame error rate < 0.2) and moderately-high data goodput (>=0.95Kbps) using a human-held smartphone camera, even over larger screen-camera distances (approx =2m).