VisKnow: Constructing Visual Knowledge Base for Object Understanding

TL;DR

VisKnow introduces a framework for building structured visual knowledge bases that integrate multi-modal data, enhancing object understanding and supporting advanced visual tasks like zero-shot recognition and fine-grained VQA.

Contribution

The paper presents a novel construction framework for a visual knowledge base that combines text and image data at object and part levels, exemplified by the creation of AnimalKB.

Findings

AnimalKB improves zero-shot recognition accuracy.

AnimalKB enhances fine-grained visual question answering.

Constructed knowledge base serves as a benchmark for graph completion and segmentation.

Abstract

Understanding objects is fundamental to computer vision. Beyond object recognition that provides only a category label as typical output, in-depth object understanding represents a comprehensive perception of an object category, involving its components, appearance characteristics, inter-category relationships, contextual background knowledge, etc. Developing such capability requires sufficient multi-modal data, including visual annotations such as parts, attributes, and co-occurrences for specific tasks, as well as textual knowledge to support high-level tasks like reasoning and question answering. However, these data are generally task-oriented and not systematically organized enough to achieve the expected understanding of object categories. In response, we propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow that extracts multi-modal, object-level knowledge for object understanding. This framework integrates enriched aligned text and image-source knowledge with region annotations at both object and part levels through a combination of expert design and large-scale model application. As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories, which contains 22K textual knowledge triplets extracted from encyclopedic documents, 420K images, and corresponding region annotations. A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA, and serves as challenging benchmarks for knowledge graph completion and part segmentation. Our findings highlight the potential of automatically constructing visual knowledge bases to advance visual understanding and its practical applications. The project page is available at https://vipl-vsu.github.io/VisKnow.