A Click Ahead: Real-Time Forecasting of Keyboard and Mouse Actions using RNNs and Computer Vision

TL;DR

This paper presents a real-time system using RNNs and computer vision to predict user keyboard and mouse actions with over 34% accuracy after a week of training, aiming to enhance workflow efficiency.

Contribution

It introduces a novel approach combining RNNs and computer vision for real-time prediction of complex user interactions on computers, a significant advancement over traditional click-based models.

Findings

Achieved 34.63% accuracy in predicting next user actions.

Requires approximately one week of user data for training.

Potential applications in workflow improvement tools.

Abstract

Computer input is more complex than a sequence of single mouse clicks and keyboard presses. We introduce a novel method to identify and represent the user interactions and build a system which predicts - in real-time - the action a user is most likely going to take next. For this, a recurrent neural network (RNN) is trained on a person's usage of the computer. We demonstrate that it is enough to train the RNN on a user's activity over approximately a week to achieve an accuracy of 34.63 % when predicting the next action from a set of almost 500 possible actions. Specific examples for how these predictions may be leveraged to build tools for improving and speeding up workflows of computer users are discussed.