Synesthesia: Detecting Screen Content via Remote Acoustic Side Channels

TL;DR

This paper demonstrates that subtle acoustic noises from computer screens can be exploited as side channels to infer displayed content and user input, posing privacy risks during video calls and recordings.

Contribution

It introduces a novel side-channel attack exploiting screen-generated sounds to detect on-screen content and user activity in real-time.

Findings

Able to detect on-screen text and virtual keyboard input

Can infer web browsing activity during video calls

Effective at distances up to 10 meters with directional microphones

Abstract

We show that subtle acoustic noises emanating from within computer screens can be used to detect the content displayed on the screens. This sound can be picked up by ordinary microphones built into webcams or screens, and is inadvertently transmitted to other parties, e.g., during a videoconference call or archived recordings. It can also be recorded by a smartphone or "smart speaker" placed on a desk next to the screen, or from as far as 10 meters away using a parabolic microphone. Empirically demonstrating various attack scenarios, we show how this channel can be used for real-time detection of on-screen text, or users' input into on-screen virtual keyboards. We also demonstrate how an attacker can analyze the audio received during video call (e.g., on Google Hangout) to infer whether the other side is browsing the web in lieu of watching the video call, and which web site is displayed on their screen.