Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis

TL;DR

This paper introduces a differentiable outlier detection layer within an end-to-end vision-language model, leveraging explicit knowledge graphs and an OOD filtering mechanism to improve robustness and efficiency in multimodal tasks.

Contribution

It proposes a novel interactive OOD layer and integrates explicit knowledge graphs into deep models, enabling robust multimodal analysis with fewer samples and less training time.

Findings

Achieves competitive results with fewer training samples.

Effectively filters noise from external knowledge bases.

Enhances robustness in vision-language tasks.

Abstract

Often, deep network models are purely inductive during training and while performing inference on unseen data. Thus, when such models are used for predictions, it is well known that they often fail to capture the semantic information and implicit dependencies that exist among objects (or concepts) on a population level. Moreover, it is still unclear how domain or prior modal knowledge can be specified in a backpropagation friendly manner, especially in large-scale and noisy settings. In this work, we propose an end-to-end vision and language model incorporating explicit knowledge graphs. We also introduce an interactive out-of-distribution (OOD) layer using implicit network operator. The layer is used to filter noise that is brought by external knowledge base. In practice, we apply our model on several vision and language downstream tasks including visual question answering, visual reasoning, and image-text retrieval on different datasets. Our experiments show that it is possible to design models that perform similarly to state-of-art results but with significantly fewer samples and training time.