Efficient Data Management for IPFS dApps

TL;DR

This paper introduces a new off-chain storage layer for IPFS-based dApps that uses erasure coding and decentralized maintenance to improve space efficiency and data availability, with modular components for encoding, decoding, monitoring, and repair.

Contribution

The paper presents a novel storage layer with encoding and repair components for IPFS, enhancing efficiency and reliability of data management in decentralized applications.

Findings

Reduced storage space using alpha entanglement codes

Efficient failure detection and repair mechanism

Modular design allows migration to other systems

Abstract

Inefficient data management has been the Achilles heel of blockchain-based decentralized applications (dApps). An off-chain storage layer, which lies between the application and the blockchain layers, can improve space efficiency and data availability with erasure codes and decentralized maintenance. This paper presents two fundamental components of such storage layer designed and implemented for the IPFS network. The IPFS Community is a component built on top of the IPFS network that encodes and decodes data before uploading to the network. Since data is encoded with alpha entanglement codes, the solution requires less storage space than the native IPFS solution which replicates data by pinning content with the IPFS Cluster. To detect and repair failures in a timely manner, we introduce the monitoring and repair component. This novel component is activated by any node and distributes the load of repairs among various nodes. These two components are implemented as pluggable modules, and can, therefore, be easily migrated to other distributed file systems by adjusting the connector component.