Content-defined Merkle Trees for Efficient Container Delivery

TL;DR

This paper introduces Content-defined Merkle Trees (MT) for container images, enabling efficient updates and transfers by indexing deduplicated blocks, significantly reducing disk and network I/O during container operations.

Contribution

The paper presents MT, a novel data structure that improves container image synchronization by efficiently indexing deduplicated blocks and enabling logarithmic change detection.

Findings

MT reduces network I/O during container updates.

MT achieves scalable performance with multiple container images.

Content-defined chunking verifies block-level deduplication effectiveness.

Abstract

Containerization simplifies the sharing and deployment of applications when environments change in the software delivery chain. To deploy an application, container delivery methods push and pull container images. These methods operate on file and layer (set of files) granularity, and introduce redundant data within a container. Several container operations such as upgrading, installing, and maintaining become inefficient, because of copying and provisioning of redundant data. In this paper, we reestablish recent results that block-level deduplication reduces the size of individual containers, by verifying the result using content-defined chunking. Block-level deduplication, however, does not improve the efficiency of push/pull operations which must determine the specific blocks to transfer. We introduce a content-defined Merkle Tree (\CDMT{}) over deduplicated storage in a container. \CDMT{} indexes deduplicated blocks and determines changes to blocks in logarithmic time on the client. \CDMT{} efficiently pushes and pulls container images from a registry, especially as containers are upgraded and (re-)provisioned on a client. We also describe how a registry can efficiently maintain the \CDMT{} index as new image versions are pushed. We show the scalability of \CDMT{} over Merkle Trees in terms of disk and network I/O savings using 15 container images and 233 image versions from Docker Hub.