# Building Usage Profiles Using Deep Neural Nets

## TL;DR

This paper presents a method using deep convolutional neural networks to automatically recognize user actions in tutorial videos, enabling the construction of software usage profiles for testing and quality improvement.

## Contribution

It introduces an automatic approach leveraging DCNNs to extract user actions from instructional videos, a novel application in software usage profiling.

## Key findings

- Achieved a mean average precision of 94.42% in classifying user actions.
- Demonstrated the effectiveness of DCNNs in extracting software usage information from videos.
- Validated the approach on 236 publicly available tutorial videos.

## Abstract

To improve software quality, one needs to build test scenarios resembling the usage of a software product in the field. This task is rendered challenging when a product's customer base is large and diverse. In this scenario, existing profiling approaches, such as operational profiling, are difficult to apply. In this work, we consider publicly available video tutorials of a product to profile usage. Our goal is to construct an automatic approach to extract information about user actions from instructional videos. To achieve this goal, we use a Deep Convolutional Neural Network (DCNN) to recognize user actions. Our pilot study shows that a DCNN trained to recognize user actions in video can classify five different actions in a collection of 236 publicly available Microsoft Word tutorial videos (published on YouTube). In our empirical evaluation we report a mean average precision of 94.42% across all actions. This study demonstrates the efficacy of DCNN-based methods for extracting software usage information from videos. Moreover, this approach may aid in other software engineering activities that require information about customer usage of a product.