Extensible Database Simulator for Fast Prototyping In-Database Algorithms

TL;DR

This paper introduces DBSim, an extensible database simulator built in Python, enabling rapid prototyping and testing of in-database algorithms without modifying core database kernels.

Contribution

The paper presents DBSim, a flexible, Python-based database simulator that supports easy extension of SQL syntax, query optimization, and execution components for in-database algorithm development.

Findings

Supports fast prototyping of in-database algorithms.

Enables easy extension of database components without kernel modification.

Facilitates debugging with visualization and analysis tools.

Abstract

With the rapid increasing of data scale, in-database analytics and learning has become one of the most studied topics in data science community, because of its significance on reducing the gap between the management and the analytics of data. By extending the capability of database on analytics and learning, data scientists can save much time on exchanging data between databases and external analytic tools. For this goal, researchers are attempting to integrate more data science algorithms into database. However, implementing the algorithms in mainstream databases is super time-consuming, especially when it is necessary to have a deep dive into the database kernels. Thus there are demands for an easy-to-extend database simulator to help fast prototype and verify the in-database algorithms before implementing them in real databases. In this demo, we present such an extensible relational database simulator, DBSim, to help data scientists prototype their in-database analytics and learning algorithms and verify the effectiveness of their ideas with minimal cost. DBSim simulates a real relational database by integrating all the major components in mainstream RDBMS, including SQL parser, relational operators, query optimizer, etc. In addition, DBSim provides various interfaces for users to flexibly plug their custom extension modules into any of the major components, without modifying the kernel. By those interfaces, DBSim supports easy extensions on SQL syntax, relational operators, query optimizer rules and cost models, and physical plan execution. Furthermore, DBSim provides utilities to facilitate users' developing and debugging, like query plan visualizer and interactive analyzer on optimization rules. We develop DBSim using pure Python to support seamless implementation of most data science algorithms into it, since many of them are written in Python.