Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection

TL;DR

This paper introduces Conditional Multi-Task Learning (CondMTL) for demographic-specific toxicity detection, improving recall for minority groups by enabling group-specific representations without requiring labels for all groups in each example.

Contribution

The paper proposes CondMTL, a novel multi-task learning approach that handles missing group labels, enhancing toxicity detection fairness across demographics.

Findings

CondMTL improves recall for minority groups.

It maintains similar overall accuracy.

It outperforms baseline models in synthetic and real data.

Abstract

Algorithmic bias often arises as a result of differential subgroup validity, in which predictive relationships vary across groups. For example, in toxic language detection, comments targeting different demographic groups can vary markedly across groups. In such settings, trained models can be dominated by the relationships that best fit the majority group, leading to disparate performance. We propose framing toxicity detection as multi-task learning (MTL), allowing a model to specialize on the relationships that are relevant to each demographic group while also leveraging shared properties across groups. With toxicity detection, each task corresponds to identifying toxicity against a particular demographic group. However, traditional MTL requires labels for all tasks to be present for every data point. To address this, we propose Conditional MTL (CondMTL), wherein only training examples relevant to the given demographic group are considered by the loss function. This lets us learn group specific representations in each branch which are not cross contaminated by irrelevant labels. Results on synthetic and real data show that using CondMTL improves predictive recall over various baselines in general and for the minority demographic group in particular, while having similar overall accuracy.