Finding the Needle in a Haystack: On the Automatic Identification of Accessibility User Reviews

TL;DR

This paper presents a machine learning model that automatically identifies accessibility-related user reviews in mobile apps, significantly aiding developers in prioritizing accessibility improvements and creating more inclusive applications.

Contribution

It introduces a keyword-based learning model for automatic accessibility review detection, outperforming baseline methods with high accuracy on a small dataset.

Findings

Model achieves 85% accuracy in identifying accessibility reviews.

Performance improves with larger training datasets.

Outperforms keyword-based and random classifiers.

Abstract

In recent years, mobile accessibility has become an important trend with the goal of allowing all users the possibility of using any app without many limitations. User reviews include insights that are useful for app evolution. However, with the increase in the amount of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews. The goal of this paper is to support the automated identification of accessibility in user reviews, to help technology professionals in prioritizing their handling, and thus, creating more inclusive apps. Particularly, we design a model that takes as input accessibility user reviews, learns their keyword-based features, in order to make a binary decision, for a given review, on whether it is about accessibility or not. The model is evaluated using a total of 5,326 mobile app reviews. The findings show that (1) our model can accurately identify accessibility reviews, outperforming two baselines, namely keyword-based detector and a random classifier; (2) our model achieves an accuracy of 85% with relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset.