Search on Secondary Attributes in Geo-Distributed Systems

TL;DR

This paper discusses the design of an indexing and query system that enables real-time searches on secondary attributes in geo-distributed, weakly consistent storage systems, addressing challenges of low latency and high throughput.

Contribution

It introduces an early-stage system design for real-time secondary attribute queries in geo-distributed, weakly consistent storage environments.

Findings

Proposed an indexing system for real-time secondary attribute search.

Addressed challenges of low latency and high throughput in geo-distributed settings.

Initial implementation demonstrates feasibility of real-time querying.

Abstract

In the age of big data, more and more applications need to query and analyse large volumes of continuously updated data in real-time. In response, cloud-scale storage systems can extend their interface that allows fast lookups on the primary key with the ability to retrieve data based on non-primary attributes. However, the need to ingest content rapidly and make it searchable immediately while supporting low-latency, high-throughput query evaluation, as well as the geo-distributed nature and weak consistency guarantees of modern storage systems pose several challenges to the implementation of indexing and search systems. We present our early-stage work on the design and implementation of an indexing and query processing system that enables realtime queries on secondary attributes of data stored in geo-distributed, weakly consistent storage systems.