When the Ground Truth is not True: Modelling Human Biases in Temporal Annotations

TL;DR

This paper introduces a method to model human biases in temporal annotations, demonstrating that soft labels better approximate true activity timings, especially when annotations are affected by cognitive biases and imprecision.

Contribution

The paper proposes a novel approach to model human biases in temporal annotations and advocates for using soft labels to improve annotation quality and evaluation.

Findings

Soft labels better approximate true activity timings.

Modeling biases improves annotation reliability.

Method outperforms traditional hard labeling in experiments.

Abstract

In supervised learning, low quality annotations lead to poorly performing classification and detection models, while also rendering evaluation unreliable. This is particularly apparent on temporal data, where annotation quality is affected by multiple factors. For example, in the post-hoc self-reporting of daily activities, cognitive biases are one of the most common ingredients. In particular, reporting the start and duration of an activity after its finalisation may incorporate biases introduced by personal time perceptions, as well as the imprecision and lack of granularity due to time rounding. Here we propose a method to model human biases on temporal annotations and argue for the use of soft labels. Experimental results in synthetic data show that soft labels provide a better approximation of the ground truth for several metrics. We showcase the method on a real dataset of daily activities.