Unsatisfied Today, Satisfied Tomorrow: a simulation framework for performance evaluation of crowdsourcing-based network monitoring

TL;DR

This paper introduces a simulation framework to evaluate how effectively crowdsourcing and machine learning can identify under-performing network cells based on user satisfaction, considering diverse scenarios and mobility patterns.

Contribution

It presents a novel empirical simulation framework for assessing the detection of under-performing network cells using subjective user satisfaction data and predictive models.

Findings

Framework effectively simulates diverse network scenarios.

Detection performance varies with user density and mobility models.

Guidelines for network operators to improve cell performance detection.

Abstract

Network operators need to continuosly upgrade their infrastructures in order to keep their customer satisfaction levels high. Crowdsourcing-based approaches are generally adopted, where customers are directly asked to answer surveys about their user experience. Since the number of collaborative users is generally low, network operators rely on Machine Learning models to predict the satisfaction levels/QoE of the users rather than directly measuring it through surveys. Finally, combining the true/predicted user satisfaction levels with information on each user mobility (e.g, which network sites each user has visited and for how long), an operator may reveal critical areas in the networks and drive/prioritize investments properly. In this work, we propose an empirical framework tailored to assess the quality of the detection of under-performing cells starting from subjective user experience grades. The framework allows to simulate diverse networking scenarios, where a network characterized by a small set of under-performing cells is visited by heterogeneous users moving through it according to realistic mobility models. The framework simulates both the processes of satisfaction surveys delivery and users satisfaction prediction, considering different delivery strategies and evaluating prediction algorithms characterized by different prediction performance. We use the simulation framework to test empirically the performance of under-performing sites detection in general scenarios characterized by different users density and mobility models to obtain insights which are generalizable and that provide interesting guidelines for network operators.