Delta Tensor: Efficient Vector and Tensor Storage in Delta Lake

TL;DR

This paper introduces Delta Tensor, a new method for storing vectors and tensors efficiently in Delta Lake, significantly improving space and time performance for AI and ML applications in cloud environments.

Contribution

It adapts array database strategies and sparse encoding to Delta Lake, enabling efficient tensor storage in a Lakehouse architecture.

Findings

Notable improvements in space efficiency

Enhanced time performance over traditional serialization

Effective for AI and ML data management

Abstract

The exponential growth of artificial intelligence (AI) and machine learning (ML) applications has necessitated the development of efficient storage solutions for vector and tensor data. This paper presents a novel approach for tensor storage in a Lakehouse architecture using Delta Lake. By adopting the multidimensional array storage strategy from array databases and sparse encoding methods to Delta Lake tables, experiments show that this approach has demonstrated notable improvements in both space and time efficiencies when compared to traditional serialization of tensors. These results provide valuable insights for the development and implementation of optimized vector and tensor storage solutions in data-intensive applications, contributing to the evolution of efficient data management practices in AI and ML domains in cloud-native environments