Annotative Indexing

TL;DR

This paper presents annotative indexing, a versatile framework unifying various database indexing methods, supporting diverse data types and complex queries, and enabling fully dynamic, transactional indexing for large-scale, concurrent data access.

Contribution

Introduces annotative indexing as a general framework that unifies multiple indexing paradigms and supports dynamic, transactional operations across diverse data types.

Findings

Supports retrieval augmented generation and knowledge graphs.

Enables fully dynamic, ACID-compliant indexing with high concurrency.

Demonstrates applicability to text, JSON, and other data types.

Abstract

This paper introduces annotative indexing, a novel framework that unifies and generalizes traditional inverted indexes, column stores, object stores, and graph databases. As a result, annotative indexing can provide the underlying indexing framework for databases that support retrieval augmented generation, knowledge graphs, entity retrieval, semi-structured data, and ranked retrieval. While we primarily focus on human language data in the form of text, annotative indexing is sufficiently general to support a range of other datatypes, and we provide examples of SQL-like queries over a JSON store that includes numbers and dates. Taking advantage of the flexibility of annotative indexing, we also demonstrate a fully dynamic annotative index incorporating support for ACID properties of transactions with hundreds of multiple concurrent readers and writers.