Contrastive Object Detection Using Knowledge Graph Embeddings

TL;DR

This paper explores the use of knowledge graph embeddings for object detection, showing they improve semantic understanding and misclassification analysis without sacrificing detection performance.

Contribution

It introduces a knowledge-embedded design for object detection architectures, integrating semantic class representations from knowledge graphs.

Findings

Knowledge-based class embeddings lead to more semantically meaningful errors.

Performance on COCO and Cityscapes benchmarks is comparable to traditional methods.

The approach generalizes across multiple object detection architectures.

Abstract

Object recognition for the most part has been approached as a one-hot problem that treats classes to be discrete and unrelated. Each image region has to be assigned to one member of a set of objects, including a background class, disregarding any similarities in the object types. In this work, we compare the error statistics of the class embeddings learned from a one-hot approach with semantically structured embeddings from natural language processing or knowledge graphs that are widely applied in open world object detection. Extensive experimental results on multiple knowledge-embeddings as well as distance metrics indicate that knowledge-based class representations result in more semantically grounded misclassifications while performing on par compared to one-hot methods on the challenging COCO and Cityscapes object detection benchmarks. We generalize our findings to multiple object detection architectures by proposing a knowledge-embedded design for keypoint-based and transformer-based object detection architectures.