PandaDB: Understanding Unstructured Data in Graph Database

TL;DR

PandaDB is an open-source graph database designed to unify the management and querying of both structured and unstructured data, enabling semantic queries and efficient processing in industrial applications.

Contribution

This work introduces a novel graph data model, query language, and optimization techniques for integrated structured and unstructured data management in a single database system.

Findings

Supports large-scale unstructured data queries efficiently

Improves query processing speed through new optimization techniques

Widely applicable in industrial domains like FinTech and recommendation systems

Abstract

Unstructured data(e.g., images, videos, PDF files, etc.) contain semantic information, for example, the facial feature of a person and the plate number of a vehicle. There could be semantic relationships between data items which are not explicitly represented. For example, a person's face may appear in two irrelevant photos. Also, much information is represented as structured data(e.g., the person's name and age). End-users prefer to query the semantic information from unstructured data together with structured data based on the potential relationships among them. However, due to the lack of a unified database system for structured and unstructured data, developers have to comprise multiple systems and runtime together to answer these queries. In this work, we build an open-source graph database named PandaDB to consistently manage and query structured and unstructured data. We first introduce a graph data model to manage structured and unstructured data, then propose a new query language to understand the semantics of the unstructured data in the graph. Next, we develop a new cost model and related query optimization techniques to speed up the unstructured data processing pipeline. Finally, we optimize the unstructured data storage and provide the index to speed up the query processing over unstructured data. PandaDB is widely used in industrial applications like FinTech, Knowledge Graph, and Recommendation System. The results show PandaDB can support a large scale of unstructured data query processing in a graph.