Modulating early visual processing by language

TL;DR

This paper introduces MOdulated RESnet (MRN), a novel method that conditions the entire visual processing pipeline on linguistic input, leading to improved performance in visual question answering tasks.

Contribution

It proposes a new approach to modulate early visual processing using language embeddings by adjusting batch normalization parameters in a pretrained ResNet.

Findings

Modulating early visual stages improves VQA performance.

Language-conditioned batch normalization enhances visual processing.

The method outperforms strong baselines on two VQA datasets.

Abstract

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.