DIMSpan - Transactional Frequent Subgraph Mining with Distributed In-Memory Dataflow Systems

TL;DR

DIMSpan is a scalable distributed in-memory system for frequent subgraph mining in large graph collections, leveraging dataflow systems like Spark and Flink to improve performance and reduce network traffic.

Contribution

It introduces a novel distributed approach for subgraph mining that is optimized for runtime, memory-awareness, and minimal network traffic in big data environments.

Findings

Demonstrates high scalability on large graph datasets

Effective pruning and optimization techniques improve performance

Outperforms existing solutions in distributed environments

Abstract

Transactional frequent subgraph mining identifies frequent subgraphs in a collection of graphs. This research problem has wide applicability and increasingly requires higher scalability over single machine solutions to address the needs of Big Data use cases. We introduce DIMSpan, an advanced approach to frequent subgraph mining that utilizes the features provided by distributed in-memory dataflow systems such as Apache Spark or Apache Flink. It determines the complete set of frequent subgraphs from arbitrary string-labeled directed multigraphs as they occur in social, business and knowledge networks. DIMSpan is optimized to runtime and minimal network traffic but memory-aware. An extensive performance evaluation on large graph collections shows the scalability of DIMSpan and the effectiveness of its pruning and optimization techniques.