SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

TL;DR

SummVis is an open-source visualization tool designed to facilitate detailed analysis of neural text summarization models, data, and evaluation metrics, addressing the black-box nature and evaluation challenges in the field.

Contribution

It introduces SummVis, a novel interactive visualization platform that enables in-depth analysis of summarization models, data, and metrics, improving interpretability and understanding.

Findings

Enables exploration of factual consistency and abstractiveness.

Provides fine-grained lexical and semantic visualizations.

Supports analysis of model performance and failure modes.

Abstract

Novel neural architectures, training strategies, and the availability of large-scale corpora haven been the driving force behind recent progress in abstractive text summarization. However, due to the black-box nature of neural models, uninformative evaluation metrics, and scarce tooling for model and data analysis, the true performance and failure modes of summarization models remain largely unknown. To address this limitation, we introduce SummVis, an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization. Through its lexical and semantic visualizations, the tools offers an easy entry point for in-depth model prediction exploration across important dimensions such as factual consistency or abstractiveness. The tool together with several pre-computed model outputs is available at https://github.com/robustness-gym/summvis.