UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training

TL;DR

UKnow introduces a unified multimodal knowledge protocol and dataset that enhances reasoning and vision-language pre-training by organizing data into a logical knowledge graph structure.

Contribution

The paper proposes UKnow, a novel knowledge protocol for multimodal data, and constructs a large-scale knowledge graph dataset following this protocol for improved reasoning and pre-training.

Findings

The dataset contains over 1.3 million nodes and 3.6 million triplets.

Experiments show UKnow improves reasoning and pre-training performance.

Unified knowledge organization benefits vision-language tasks.

Abstract

This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. Particularly focusing on visual and linguistic modalities, we categorize data knowledge into five unit types, namely, in-image, in-text, cross-image, cross-text, and image-text, and set up an efficient pipeline to help construct the multimodal knowledge graph from any data collection. Thanks to the logical information naturally contained in knowledge graph, organizing datasets under UKnow format opens up more possibilities of data usage compared to the commonly used image-text pairs. Following UKnow protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset that consists of 1,388,568 nodes (with 571,791 vision-related ones) and 3,673,817 triplets. The dataset is also annotated with rich event tags, including 11 coarse labels and 9,185 fine labels. Experiments on 4 benchmarks demonstrate the potential of UKnow in supporting common-sense reasoning and boosting vision-language pre-training with a single dataset, benefiting from its unified form of knowledge organization. See Appendix to download the dataset.