Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms

TL;DR

This paper investigates which types of self-disclosure information are most effective for modeling annotators' judgments of social norms, revealing that limited comments and diverse samples can improve prediction accuracy.

Contribution

It categorizes self-disclosures and analyzes their impact on annotator modeling, challenging assumptions about the amount and diversity of information needed.

Findings

Few comments related to the original post suffice for accurate prediction.

Diverse self-disclosures do not necessarily improve model performance.

Larger, unfiltered comment pools yield better annotator prediction results.

Abstract

Recent work has explored the use of personal information in the form of persona sentences or self-disclosures to improve modeling of individual characteristics and prediction of annotator labels for subjective tasks. The volume of personal information has historically been restricted and thus little exploration has gone into understanding what kind of information is most informative for predicting annotator labels. In this work, we categorize self-disclosures and use them to build annotator models for predicting judgments of social norms. We perform several ablations and analyses to examine the impact of the type of information on our ability to predict annotation patterns. Contrary to previous work, only a small number of comments related to the original post are needed. Lastly, a more diverse sample of annotator self-disclosures did not lead to the best performance. Sampling from a larger pool of comments without filtering still yields the best performance, suggesting that there is still much to uncover in terms of what information about an annotator is most useful for verdict prediction.