Examining Human-AI Collaboration for Co-Writing Constructive Comments Online

TL;DR

This study explores how large language models can assist humans in writing more constructive online comments on divisive issues, highlighting benefits and challenges in human-AI collaboration.

Contribution

It provides empirical evidence on the effectiveness and limitations of LLMs in co-writing constructive comments and analyzes perceptual differences between humans and AI.

Findings

LLMs increased comment constructiveness and positivity.

Participants rated AI-assisted comments as more constructive than human-only comments.

LLMs sometimes distorted original views, especially with non-polarizing stances.

Abstract

This paper examines if large language models (LLMs) can help people write constructive comments on divisive social issues due to the difficulty of expressing constructive disagreement online. Through controlled experiments with 600 participants from India and the US, who reviewed and wrote constructive comments on threads related to Islamophobia and homophobia, we observed potential misalignment between how LLMs and humans perceive constructiveness in online comments. While the LLM was more likely to prioritize politeness and balance among contrasting viewpoints when evaluating constructiveness, participants emphasized logic and facts more than the LLM did. Despite these differences, participants rated both LLM-generated and human-AI co-written comments as significantly more constructive than those written independently by humans. Our analysis also revealed that LLM-generated comments integrated significantly more linguistic features of constructiveness compared to human-written comments. When participants used LLMs to refine their comments, the resulting comments were more constructive, more positive, less toxic, and retained the original intent. However, occasionally LLMs distorted people's original views -- especially when their stances were not outright polarizing. Based on these findings, we discuss ethical and design considerations in using LLMs to facilitate constructive discourse online.