Sort Story: Sorting Jumbled Images and Captions into Stories

TL;DR

This paper introduces the task of sequencing jumbled image-caption pairs into coherent stories, proposing multiple models that leverage text and image features to understand temporal common sense.

Contribution

It formulates a new sequencing task for stories and develops ensemble models combining unary and pairwise predictions with multimodal features.

Findings

Ensemble models outperform individual approaches.

Text and image features provide complementary benefits.

Models learn meaningful temporal common sense.

Abstract

Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication. We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story. We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense.