Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learning

TL;DR

This paper introduces a novel approach for document-based zero-shot learning that extracts and aligns multi-view semantic concepts from documents and images, improving performance by focusing on partial rather than full concept alignment.

Contribution

The work proposes a semantic decomposition network with specialized loss functions to enable partial alignment of visual and textual semantic concepts, addressing redundancy and diversity issues.

Findings

Outperforms state-of-the-art methods on three benchmarks

Learned partial associations are interpretable

Effective semantic concept extraction from documents and images

Abstract

Recent work shows that documents from encyclopedias serve as helpful auxiliary information for zero-shot learning. Existing methods align the entire semantics of a document with corresponding images to transfer knowledge. However, they disregard that semantic information is not equivalent between them, resulting in a suboptimal alignment. In this work, we propose a novel network to extract multi-view semantic concepts from documents and images and align the matching rather than entire concepts. Specifically, we propose a semantic decomposition module to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. To alleviate the issue of information redundancy among embeddings, we propose the local-to-semantic variance loss to capture distinct local details and multiple semantic diversity loss to enforce orthogonality among embeddings. Subsequently, two losses are introduced to partially align visual-semantic embedding pairs according to their semantic relevance at the view and word-to-patch levels. Consequently, we consistently outperform state-of-the-art methods under two document sources in three standard benchmarks for document-based zero-shot learning. Qualitatively, we show that our model learns the interpretable partial association.