Help the machine to help you: an evaluation in the wild of egocentric data cleaning via skeptical learning

TL;DR

This paper evaluates Skeptical Learning (SKEL) in real-world conditions with actual users, demonstrating its potential to reduce annotation effort and improve data quality in egocentric data cleaning for digital assistants.

Contribution

It provides the first real-world evaluation of SKEL involving end-users, highlighting practical challenges and benefits in egocentric data cleaning.

Findings

SKEL reduces user annotation effort.

SKEL improves data quality in real-world settings.

Balancing user effort and data accuracy remains challenging.

Abstract

Any digital personal assistant, whether used to support task performance, answer questions, or manage work and daily life, including fitness schedules, requires high-quality annotations to function properly. However, user annotations, whether actively produced or inferred from context (e.g., data from smartphone sensors), are often subject to errors and noise. Previous research on Skeptical Learning (SKEL) addressed the issue of noisy labels by comparing offline active annotations with passive data, allowing for an evaluation of annotation accuracy. However, this evaluation did not include confirmation from end-users, the best judges of their own context. In this study, we evaluate SKEL's performance in real-world conditions with actual users who can refine the input labels based on their current perspectives and needs. The study involves university students using the iLog mobile application on their devices over a period of four weeks. The results highlight the challenges of finding the right balance between user effort and data quality, as well as the potential benefits of using SKEL, which include reduced annotation effort and improved quality of collected data.