MatrixGate: A High-performance Data Ingestion Tool for Time-series Databases

TL;DR

MatrixGate is a novel high-performance data ingestion tool for massive time-series databases, utilizing parallel processing, lock-free queues, and adaptive policies to significantly improve loading speed and reduce query latency.

Contribution

This paper introduces MatrixGate, a new data ingestion approach that combines parallel procedures and unique strategies to enhance efficiency and scalability for time-series data.

Findings

Outperforms existing methods by 3 to 100 times in loading speed

Reduces query latency by approximately 80%

Achieves 86% scalability in distributed architectures

Abstract

Recent years have seen massive time-series data generated in many areas. This different scenario brings new challenges, particularly in terms of data ingestion, where existing technologies struggle to handle such massive time-series data, leading to low loading speed and poor timeliness. To address these challenges, this paper presents MatrixGate, a new and efficient data ingestion approach for massive time-series data. MatrixGate implements both single-instance and multi-instance parallel procedures, which is based on its unique ingestion strategies. First, MatrixGate uses policies to tune the slots that are synchronized with segments to ingest data, which eliminates the cost of starting transactions and enhance the efficiency. Second, multi-coroutines are responsible for transfer data, which can increase the degree of parallelism significantly. Third, lock-free queues are used to enable direct data transfer without the need for disk storage or lodging in the master instance. Experiment results on multiple datasets show that MatrixGate outperforms state-of-the-art methods by 3 to 100 times in loading speed, and cuts down about 80% query latency. Furthermore, MatrixGate scales out efficiently under distributed architecture, achieving scalability of 86%.