ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

TL;DR

This paper introduces ExpStar, a model for automatic scientific experiment commentary generation, supported by a new dataset, ExpInstruct, covering multiple disciplines and incorporating scientific principles and safety guidelines.

Contribution

The paper presents the first dataset for experiment commentary and a retrieval-augmented model that outperforms existing large multimodal models in this task.

Findings

ExpStar significantly outperforms 14 baseline models.

ExpInstruct contains over 7,000 step-level commentaries across 21 scientific subjects.

The approach effectively integrates external scientific knowledge.

Abstract

Experiment commentary is crucial in describing the experimental procedures, delving into underlying scientific principles, and incorporating content-related safety guidelines. In practice, human teachers rely heavily on subject-specific expertise and invest significant time preparing such commentary. To address this challenge, we introduce the task of automatic commentary generation across multi-discipline scientific experiments. While recent progress in large multimodal models (LMMs) has demonstrated promising capabilities in video understanding and reasoning, their ability to generate fine-grained and insightful experiment commentary remains largely underexplored. In this paper, we make the following contributions: (i) We construct \textit{ExpInstruct}, the first dataset tailored for experiment commentary generation, featuring over 7\textit{K} step-level commentaries across 21 scientific subjects from 3 core disciplines (\ie, science, healthcare and engineering). Each sample includes procedural descriptions along with potential scientific principles (\eg, chemical equations and physical laws) and safety guidelines. (ii) We propose ExpStar, an automatic experiment commentary generation model that leverages a retrieval-augmented mechanism to adaptively access, evaluate, and utilize external knowledge. (iii) Extensive experiments show that our ExpStar substantially outperforms 14 leading LMMs, which highlights the superiority of our dataset and model. We believe that ExpStar holds great potential for advancing AI-assisted scientific experiment instruction.