Fine-Grained Analysis of Team Collaborative Dialogue

TL;DR

This paper presents an initial approach to analyze team collaborative dialogues in software development, introducing a hierarchical labeling scheme and metrics to understand team dynamics from Slack chat data.

Contribution

It develops a novel hierarchical labeling scheme and descriptive metrics, and applies a transformer + CRF model to analyze long-range context in team chat dialogues.

Findings

Hierarchical labeling scheme for dialogue acts

Metrics based on dialogue act frequency

Transformer + CRF model captures long-range context

Abstract

Natural language analysis of human collaborative chat dialogues is an understudied domain with many unique challenges: a large number of dialogue act labels, underspecified and dynamic tasks, interleaved topics, and long-range contextual dependence. While prior work has studied broad metrics of team dialogue and associated performance using methods such as LSA, there has been little effort in generating fine-grained descriptions of team dynamics and individual performance from dialogue. We describe initial work towards developing an explainable analytics tool in the software development domain using Slack chats mined from our organization, including generation of a novel, hierarchical labeling scheme; design of descriptive metrics based on the frequency of occurrence of dialogue acts; and initial results using a transformer + CRF architecture to incorporate long-range context.