# Incremental Techniques for Large-Scale Dynamic Query Processing

## TL;DR

This paper reviews incremental query processing techniques for large-scale, real-time big data, highlighting recent algorithms that improve performance and address challenges in high-frequency updates and distributed streaming platforms.

## Contribution

It provides a comprehensive overview of legacy and new algorithms for incremental query processing in big data environments, emphasizing recent advances and future directions.

## Key findings

- New algorithms reduce processing complexity for high-frequency updates
- Distributed streaming platforms like Spark Streaming and Flink are leveraged
- Analysis of various approaches' characteristics and performance

## Abstract

Many applications from various disciplines are now required to analyze fast evolving big data in real time. Various approaches for incremental processing of queries have been proposed over the years. Traditional approaches rely on updating the results of a query when updates are streamed rather than re-computing these queries, and therefore, higher execution performance is expected. However, they do not perform well for large databases that are updated at high frequencies. Therefore, new algorithms and approaches have been proposed in the literature to address these challenges by, for instance, reducing the complexity of processing updates. Moreover, many of these algorithms are now leveraging distributed streaming platforms such as Spark Streaming and Flink. In this tutorial, we briefly discuss legacy approaches for incremental query processing, and then give an overview of the new challenges introduced due to processing big data streams. We then discuss in detail the recently proposed algorithms that address some of these challenges. We emphasize the characteristics and algorithmic analysis of various proposed approaches and conclude by discussing future research directions.