OverHear: Headphone based Multi-sensor Keystroke Inference

TL;DR

This paper presents OverHear, a framework that uses headphone sensors to infer keystrokes through acoustic and accelerometer data, revealing potential security vulnerabilities in headphone devices.

Contribution

We introduce OverHear, a novel multi-sensor keystroke inference system utilizing headphone microphones and accelerometers, demonstrating its effectiveness in real-world scenarios.

Findings

Achieved around 80% top-5 accuracy for mechanical keyboards

Achieved around 60% top-5 accuracy for membrane keyboards

Top-100 word prediction accuracy over 70% across keyboard types

Abstract

Headphones, traditionally limited to audio playback, have evolved to integrate sensors like high-definition microphones and accelerometers. While these advancements enhance user experience, they also introduce potential eavesdropping vulnerabilities, with keystroke inference being our concern in this work. To validate this threat, we developed OverHear, a keystroke inference framework that leverages both acoustic and accelerometer data from headphones. The accelerometer data, while not sufficiently detailed for individual keystroke identification, aids in clustering key presses by hand position. Concurrently, the acoustic data undergoes analysis to extract Mel Frequency Cepstral Coefficients (MFCC), aiding in distinguishing between different keystrokes. These features feed into machine learning models for keystroke prediction, with results further refined via dictionary-based word prediction methods. In our experimental setup, we tested various keyboard types under different environmental conditions. We were able to achieve top-5 key prediction accuracy of around 80% for mechanical keyboards and around 60% for membrane keyboards with top-100 word prediction accuracies over 70% for all keyboard types. The results highlight the effectiveness and limitations of our approach in the context of real-world scenarios.