Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference

TL;DR

This paper presents a model-agnostic sharding framework utilizing blockchain and compression techniques to enable secure, efficient, and decentralized inference of large AI models on diverse hardware.

Contribution

It introduces a novel blockchain-based sharding framework for decentralized AI inference that is model-agnostic and incorporates compression and security measures.

Findings

Efficient distributed inference on consumer hardware.

Compression techniques do not reduce model accuracy.

Enhanced data security with hardware trusted execution environments.

Abstract

The rapid growth of large-scale AI models, particularly large language models has brought significant challenges in data privacy, computational resources, and accessibility. Traditional centralized architectures often struggle to meet required data security and scalability needs which hinders the democratization of AI systems. Nesa introduces a model-agnostic sharding framework designed for decentralized AI inference. Our framework uses blockchain-based sequential deep neural network sharding to distribute computational tasks across a diverse network of nodes based on a personalised heuristic and routing mechanism. This enables efficient distributed training and inference for recent large-scale models even on consumer-grade hardware. We use compression techniques like dynamic blockwise quantization and mixed matrix decomposition to reduce data transfer and memory needs. We also integrate robust security measures, including hardware-based trusted execution environments to ensure data integrity and confidentiality. Evaluating our system across various natural language processing and vision tasks shows that these compression strategies do not compromise model accuracy. Our results highlight the potential to democratize access to cutting-edge AI technologies by enabling secure and efficient inference on a decentralized network.