Adaptive Logging for Distributed In-memory Databases

TL;DR

This paper introduces an adaptive logging method for distributed in-memory databases that balances transaction processing speed and recovery efficiency by combining data and command logging techniques.

Contribution

It extends command logging to distributed systems and proposes an adaptive approach that optimizes logging strategies based on application needs.

Findings

Adaptive logging achieves 10x faster recovery.

Transaction throughput is comparable to command logging.

Outperforms traditional data logging in distributed environments.

Abstract

A new type of logs, the command log, is being employed to replace the traditional data log (e.g., ARIES log) in the in-memory databases. Instead of recording how the tuples are updated, a command log only tracks the transactions being executed, thereby effectively reducing the size of the log and improving the performance. Command logging on the other hand increases the cost of recovery, because all the transactions in the log after the last checkpoint must be completely redone in case of a failure. In this paper, we first extend the command logging technique to a distributed environment, where all the nodes can perform recovery in parallel. We then propose an adaptive logging approach by combining data logging and command logging. The percentage of data logging versus command logging becomes an optimization between the performance of transaction processing and recovery to suit different OLTP applications. Our experimental study compares the performance of our proposed adaptive logging, ARIES-style data logging and command logging on top of H-Store. The results show that adaptive logging can achieve a 10x boost for recovery and a transaction throughput that is comparable to that of command logging.