# Ingesting High-Velocity Streaming Graphs from Social Media Sources

## TL;DR

This paper presents an adaptive buffering and graph compression method to efficiently ingest high-velocity, bursty social media streaming graphs into databases, addressing challenges of data velocity and complexity.

## Contribution

The paper introduces a novel adaptive buffering algorithm that considers data rate, content, and CPU resources, along with an ingestion-time graph compression technique, to optimize streaming graph data ingestion.

## Key findings

- The adaptive buffering reduces ingestion latency under bursty data conditions.
- Graph compression improves storage efficiency and ingestion speed.
- Experimental results confirm the effectiveness of the proposed methods.

## Abstract

Many data science applications like social network analysis use graphs as their primary form of data. However, acquiring graph-structured data from social media presents some interesting challenges. The first challenge is the high data velocity and bursty nature of the social media data. The second challenge is that the complex nature of the data makes the ingestion process expensive. If we want to store the streaming graph data in a graph database, we face a third challenge -- the database is very often unable to sustain the ingestion of high-velocity, high-burst data. We have developed an adaptive buffering mechanism and a graph compression technique that effectively mitigates the problem. A novel aspect of our method is that the adaptive buffering algorithm uses the data rate, the data content as well as the CPU resources of the database machine to determine an optimal data ingestion mechanism. We further show that an ingestion-time graph-compression strategy improves the efficiency of the data ingestion into the database. We have verified the efficacy of our ingestion optimization strategy through extensive experiments.