cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers

TL;DR

cPAPERS is a novel dataset of multimodal conversational question-answer pairs grounded in scientific paper components, enabling research in situated and multimodal interactions within scientific documents.

Contribution

The paper introduces the cPAPERS dataset, a new resource for studying multimodal conversational interactions in scientific papers, along with baseline LLM approaches.

Findings

Baseline LLMs can effectively utilize the dataset for conversational tasks.

The dataset covers diverse scientific paper components like text, figures, and tables.

OpenResearch data collection strategy enhances dataset quality.

Abstract

An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed of text, equations, figures, and tables, SIMMC methods must be developed specifically for each component to support the depth of inquiry and interactions required by research scientists. This work introduces Conversational Papers (cPAPERS), a dataset of conversational question-answer pairs from reviews of academic papers grounded in these paper components and their associated references from scientific documents available on arXiv. We present a data collection strategy to collect these question-answer pairs from OpenReview and associate them with contextual information from LaTeX source files. Additionally, we present a series of baseline approaches utilizing Large Language Models (LLMs) in both zero-shot and fine-tuned configurations to address the cPAPERS dataset.