Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark

TL;DR

This paper introduces Meta-RecipeQA, a new benchmark for multimodal machine comprehension that systematically addresses dataset biases and evaluates models' generalization abilities across varying difficulty levels.

Contribution

It proposes a framework with control knobs to generate diverse datasets, introduces a hierarchical transformer reasoning network, and provides a comprehensive evaluation of models on the new benchmark.

Findings

HTRN outperforms SOTA models by ~18% in Visual Cloze task.

Models perform worse on Meta-RecipeQA than RecipeQA, indicating less bias.

The benchmark effectively measures generalization capabilities.

Abstract

We focus on Multimodal Machine Reading Comprehension (M3C) where a model is expected to answer questions based on given passage (or context), and the context and the questions can be in different modalities. Previous works such as RecipeQA have proposed datasets and cloze-style tasks for evaluation. However, we identify three critical biases stemming from the question-answer generation process and memorization capabilities of large deep models. These biases makes it easier for a model to overfit by relying on spurious correlations or naive data patterns. We propose a systematic framework to address these biases through three Control-Knobs that enable us to generate a test bed of datasets of progressive difficulty levels. We believe that our benchmark (referred to as Meta-RecipeQA) will provide, for the first time, a fine grained estimate of a model's generalization capabilities. We also propose a general M3C model that is used to realize several prior SOTA models and motivate a novel hierarchical transformer based reasoning network (HTRN). We perform a detailed evaluation of these models with different language and visual features on our benchmark. We observe a consistent improvement with HTRN over SOTA (~18% in Visual Cloze task and ~13% in average over all the tasks). We also observe a drop in performance across all the models when testing on RecipeQA and proposed Meta-RecipeQA (e.g. 83.6% versus 67.1% for HTRN), which shows that the proposed dataset is relatively less biased. We conclude by highlighting the impact of the control knobs with some quantitative results.