A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks

TL;DR

This paper introduces ITO, a comprehensive, curated knowledge graph of AI tasks, benchmarks, and metrics designed to facilitate analysis of AI progress and research prioritization.

Contribution

The creation of ITO, a large-scale, ontology-based knowledge graph that integrates AI tasks, benchmarks, and performance data for enhanced analysis and collaboration.

Findings

ITO contains 685,560 edges and 1,100 classes.

It enables network-based analysis of AI capabilities.

The dataset and tools are openly available.

Abstract

Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully -- or still unsuccessfully -- applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized. To help address these issues, we created the Intelligence Task Ontology and Knowledge Graph (ITO), a comprehensive, richly structured and manually curated resource on artificial intelligence tasks, benchmark results and performance metrics. The current version of ITO contain 685,560 edges, 1,100 classes representing AI processes and 1,995 properties representing performance metrics. The goal of ITO is to enable precise and network-based analyses of the global landscape of AI tasks and capabilities. ITO is based on technologies that allow for easy integration and enrichment with external data, automated inference and continuous, collaborative expert curation of underlying ontological models. We make the ITO dataset and a collection of Jupyter notebooks utilising ITO openly available.