How Do We Answer Complex Questions: Discourse Structure of Long-form Answers

TL;DR

This paper investigates the discourse structure of long-form answers in QA, developing an ontology of sentence roles, analyzing human and model answers, and providing tools for automatic classification to improve understanding and evaluation.

Contribution

It introduces a six-role ontology for long-form answers, annotates a large dataset, and develops a classifier for discourse analysis, advancing research in long-form QA modeling and evaluation.

Findings

Different answer collection methods lead to distinct discourse structures.

Annotators show less agreement on model-generated answers.

A strong classifier for sentence role identification was developed.

Abstract

Long-form answers, consisting of multiple sentences, can provide nuanced and comprehensive answers to a broader set of questions. To better understand this complex and understudied task, we study the functional structure of long-form answers collected from three datasets, ELI5, WebGPT and Natural Questions. Our main goal is to understand how humans organize information to craft complex answers. We develop an ontology of six sentence-level functional roles for long-form answers, and annotate 3.9k sentences in 640 answer paragraphs. Different answer collection methods manifest in different discourse structures. We further analyze model-generated answers -- finding that annotators agree less with each other when annotating model-generated answers compared to annotating human-written answers. Our annotated data enables training a strong classifier that can be used for automatic analysis. We hope our work can inspire future research on discourse-level modeling and evaluation of long-form QA systems.