Libraries, Integrations and Hubs for Decentralized AI using IPFS

TL;DR

This paper explores how decentralized technologies like IPFS, Web3, and DAOs can address limitations of centralized AI services, presenting libraries, integrations, and a proof-of-concept decentralized AI Hub.

Contribution

It introduces new libraries and integrations for decentralized AI storage and compute, and demonstrates a decentralized AI Hub built on IPFS.

Findings

Decentralized storage can reduce costs and improve control over AI assets.

Libraries enable interaction with IPFS in familiar data science languages.

A proof-of-concept decentralized AI Hub showcases practical application.

Abstract

AI requires heavy amounts of storage and compute. As a result, AI developers are regular users of centralised cloud services such as AWS, GCP and Azure, compute environments such as Jupyter and Colab notebooks, and AI Hubs such as HuggingFace and ActiveLoop. There services are associated with certain benefits and limitations that stem from the underlying infrastructure and governance systems with which they are built. These limitations include high costs, lack of monetization and reward, lack of control and difficulty of reproducibility. At the same time, there are few libraries that allow data scientists to interact with decentralised storage in the language that data scientists are used to, and few hubs where they can discover and interact with AI assets. In this report, we explore the potential of decentralized technologies - such as Web3 wallets, peer-to-peer marketplaces, decentralized storage (IPFS and Filecoin) and compute, and DAOs - to address some of the above limitations. We showcase some of the libraries and integrations that we have built to tackle these issues, as well as a proof of concept of a decentralized AI Hub app, that all use IPFS as a core infrastructural component.