# Generating Labels for Regression of Subjective Constructs using Triplet Embeddings

## TL;DR

This paper introduces a novel triplet embedding approach for generating regression labels of subjective constructs, leveraging relative comparisons to improve annotation accuracy and robustness in noisy, real-world settings.

## Contribution

It proposes a new annotation method using triplet comparisons to create time-indexed embeddings for subjective constructs, enhancing label quality over traditional absolute ratings.

## Key findings

- Accurately represents synthetic hidden constructs under noisy sampling.
- Recovers underlying structure of subjective constructs from human annotations.
- Effective in real-world noisy annotation scenarios.

## Abstract

Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model training and testing. Current research trends focus on correcting artifacts and biases introduced by annotators during the annotation process while fusing them into a single annotation. In this work, we propose a novel annotation approach using triplet embeddings. By lifting the absolute annotation process to relative annotations where the annotator compares individual target constructs in triplets, we leverage the accuracy of comparisons over absolute ratings by human annotators. We then build a 1-dimensional embedding in Euclidean space that is indexed in time and serves as a label for regression. In this setting, the annotation fusion occurs naturally as a union of sets of sampled triplet comparisons among different annotators. We show that by using our proposed sampling method to find an embedding, we are able to accurately represent synthetic hidden constructs in time under noisy sampling conditions. We further validate this approach using human annotations collected from Mechanical Turk and show that we can recover the underlying structure of the hidden construct up to bias and scaling factors.