Verifying Concurrent Multicopy Search Structures

TL;DR

This paper introduces a verification framework for concurrent multicopy search structures like LSM trees, enabling formal proofs of correctness and the first verified in-memory implementation.

Contribution

It presents a general verification framework for multicopy search structures, allowing proof reuse and formal verification of diverse implementations.

Findings

Framework for verifying linearizability of multicopy structures

Template algorithms for LSM and differential file structures

First verified concurrent in-memory LSM tree implementation

Abstract

Multicopy search structures such as log-structured merge (LSM) trees are optimized for high insert/update/delete (collectively known as upsert) performance. In such data structures, an upsert on key $k$, which adds $(k,v)$ where $v$ can be a value or a tombstone, is added to the root node even if $k$ is already present in other nodes. Thus there may be multiple copies of $k$ in the search structure. A search on $k$ aims to return the value associated with the most recent upsert. We present a general framework for verifying linearizability of concurrent multicopy search structures that abstracts from the underlying representation of the data structure in memory, enabling proof-reuse across diverse implementations. Based on our framework, we propose template algorithms for a) LSM structures forming arbitrary directed acyclic graphs and b) differential file structures, and formally verify these templates in the concurrent separation logic Iris. We also instantiate the LSM template to obtain the first verified concurrent in-memory LSM tree implementation.