Efficient Management of High-Frequency Sensor Data Streams Using a Read-Optimized Learned Index

TL;DR

This paper introduces DyGLIN, a new index system for managing high-frequency sensor data in IoT environments, which improves query performance and handles updates more efficiently.

Contribution

DyGLIN introduces a decoupled leaf architecture with hierarchical filtering and a Delta Buffer mechanism for efficient spatial indexing in dynamic IoT workloads.

Findings

DyGLIN reduces query latency by 26.4% compared to GLIN.

It achieves 30.0% higher insertion throughput with minimal memory overhead increase.

The system shows superior deletion performance in dynamic workloads.

Abstract

The rapid growth of sensor data in IoT and Digital Twins necessitates high-performance spatial indexing. Traditional indexes like Rtrees suffer from high storage overhead, while state-of-the-art learned indexes like GLIN encounter a “Refinement Bottleneck” due to coarse-grained Minimum Bounding Rectangle (MBR) filtering. Furthermore, existing solutions often trade update throughput for query accuracy, failing in dynamic IoT workloads with concurrent reads and writes. We propose DyGLIN (Dynamic Generate Learning-Based Index), a dynamic, read-optimized learned spatial index tailored for high-frequency sensor streams. DyGLIN introduces a decoupled leaf architecture separating query processing from data maintenance. To accelerate queries, we implement a hierarchical filtering pipeline using hierarchical MBRs (HMBR) and Cuckoo Filters to aggressively prune false positives. For maintenance, a Delta Buffer mechanism amortizes update costs, while logical deletion ensures high throughput. Experiments on real-world datasets show that DyGLIN reduces query latency by 26.4% [95% CI: 20.1%, 38.6%] compared to GLIN. It achieves 30.0% [95% CI: 21.4%, 35.9%] higher insertion throughput and superior deletion performance, with only an 18.5% [95% CI: 16.8%, 19.8%] increase in memory overhead.

Figures