ML-Powered Index Tuning: An Overview of Recent Progress and Open Challenges

TL;DR

This paper reviews recent advances and open challenges in automated index tuning using machine learning, emphasizing scalability, minimal performance regressions, and cross-platform applicability in modern cloud data systems.

Contribution

It provides a comprehensive overview of ML techniques for index tuning, identifies key gaps, and proposes a preliminary cross-platform design to democratize index tuning.

Findings

ML techniques improve workload selection and candidate filtering.

Methods reduce query optimizer calls and performance regressions.

Identifies gaps for effective ML integration in index tuning.

Abstract

The scale and complexity of workloads in modern cloud services have brought into sharper focus a critical challenge in automated index tuning -- the need to recommend high-quality indexes while maintaining index tuning scalability. This challenge is further compounded by the requirement for automated index implementations to introduce minimal query performance regressions in production deployments, representing a significant barrier to achieving scalability and full automation. This paper directs attention to these challenges within automated index tuning and explores ways in which machine learning (ML) techniques provide new opportunities in their mitigation. In particular, we reflect on recent efforts in developing ML techniques for workload selection, candidate index filtering, speeding up index configuration search, reducing the amount of query optimizer calls, and lowering the chances of performance regressions. We highlight the key takeaways from these efforts and underline the gaps that need to be closed for their effective functioning within the traditional index tuning framework. Additionally, we present a preliminary cross-platform design aimed at democratizing index tuning across multiple SQL-like systems -- an imperative in today's continuously expanding data system landscape. We believe our findings will help provide context and impetus to the research and development efforts in automated index tuning.