Revisiting the Threat Space for Vision-based Keystroke Inference Attacks

TL;DR

This paper demonstrates that deep neural networks trained on synthetic data can effectively perform vision-based keystroke inference attacks, achieving high accuracy even with limited real-world data, by leveraging transfer learning and domain adaptation.

Contribution

It introduces a large-scale synthetic dataset for keystroke inference and shows that pre-training on this data improves deep learning model performance on real-world data.

Findings

Pre-training on synthetic data boosts accuracy to 95.6%.

Transfer learning from synthetic to real data is effective.

Synthetic data can mitigate small dataset issues in attack scenarios.

Abstract

A vision-based keystroke inference attack is a side-channel attack in which an attacker uses an optical device to record users on their mobile devices and infer their keystrokes. The threat space for these attacks has been studied in the past, but we argue that the defining characteristics for this threat space, namely the strength of the attacker, are outdated. Previous works do not study adversaries with vision systems that have been trained with deep neural networks because these models require large amounts of training data and curating such a dataset is expensive. To address this, we create a large-scale synthetic dataset to simulate the attack scenario for a keystroke inference attack. We show that first pre-training on synthetic data, followed by adopting transfer learning techniques on real-life data, increases the performance of our deep learning models. This indicates that these models are able to learn rich, meaningful representations from our synthetic data and that training on the synthetic data can help overcome the issue of having small, real-life datasets for vision-based key stroke inference attacks. For this work, we focus on single keypress classification where the input is a frame of a keypress and the output is a predicted key. We are able to get an accuracy of 95.6% after pre-training a CNN on our synthetic data and training on a small set of real-life data in an adversarial domain adaptation framework. Source Code for Simulator: https://github.com/jlim13/keystroke-inference-attack-synthetic-dataset-generator-