CQASUMM: Building References for Community Question Answering Summarization Corpora

TL;DR

This paper introduces CQASUMM, a large annotated dataset for community question answering summarization, and proposes OpinioSumm, a new multi-document summarization method tailored for opinion-rich CQA content.

Contribution

The paper creates the first large-scale CQA summarization dataset and develops OpinioSumm, a multi-document summarizer that handles opinionated and diverse CQA data effectively.

Findings

OpinioSumm outperforms baseline methods by 4.6% ROUGE-1 score.

Existing MDS methods struggle with opinionated and diverse CQA content.

The dataset enables new research in community question answering summarization.

Abstract

Community Question Answering forums such as Quora, Stackoverflow are rich knowledge resources, often catering to information on topics overlooked by major search engines. Answers submitted to these forums are often elaborated, contain spam, are marred by slurs and business promotions. It is difficult for a reader to go through numerous such answers to gauge community opinion. As a result summarization becomes a prioritized task for CQA forums. While a number of efforts have been made to summarize factoid CQA, little work exists in summarizing non-factoid CQA. We believe this is due to the lack of a considerably large, annotated dataset for CQA summarization. We create CQASUMM, the first huge annotated CQA summarization dataset by filtering the 4.4 million Yahoo! Answers L6 dataset. We sample threads where the best answer can double up as a reference summary and build hundred word summaries from them. We treat other answers as candidates documents for summarization. We provide a script to generate the dataset and introduce the new task of Community Question Answering Summarization. Multi document summarization has been widely studied with news article datasets, especially in the DUC and TAC challenges using news corpora. However documents in CQA have higher variance, contradicting opinion and lesser amount of overlap. We compare the popular multi document summarization techniques and evaluate their performance on our CQA corpora. We look into the state-of-the-art and understand the cases where existing multi document summarizers (MDS) fail. We find that most MDS workflows are built for the entirely factual news corpora, whereas our corpus has a fair share of opinion based instances too. We therefore introduce OpinioSumm, a new MDS which outperforms the best baseline by 4.6% w.r.t ROUGE-1 score.