Learning semantic Image attributes using Image recognition and knowledge graph embeddings

TL;DR

This paper introduces a shared learning model that combines image recognition with knowledge graph embeddings to extract semantic attributes from images, enhancing knowledge base completeness with limited data.

Contribution

It presents a novel approach that integrates image attributes and knowledge graph embeddings, improving semantic understanding with limited knowledge base data.

Findings

Significant accuracy improvements over previous methods

Effective linking of image entities through knowledge graph embeddings

Bridges gap between large data frameworks and limited predicate frameworks

Abstract

Extracting structured knowledge from texts has traditionally been used for knowledge base generation. However, other sources of information, such as images can be leveraged into this process to build more complete and richer knowledge bases. Structured semantic representation of the content of an image and knowledge graph embeddings can provide a unique representation of semantic relationships between image entities. Linking known entities in knowledge graphs and learning open-world images using language models has attracted lots of interest over the years. In this paper, we propose a shared learning approach to learn semantic attributes of images by combining a knowledge graph embedding model with the recognized attributes of images. The proposed model premises to help us understand the semantic relationship between the entities of an image and implicitly provide a link for the extracted entities through a knowledge graph embedding model. Under the limitation of using a custom user-defined knowledge base with limited data, the proposed model presents significant accuracy and provides a new alternative to the earlier approaches. The proposed approach is a step towards bridging the gap between frameworks which learn from large amounts of data and frameworks which use a limited set of predicates to infer new knowledge.