Design Guidelines for Apache Kafka Driven Data Management and Distribution in Smart Cities

TL;DR

This paper presents a modular architecture utilizing Apache Kafka and NiFi for efficient data management in smart cities, providing guidelines based on experimental validation to enhance real-time services and data-driven decision-making.

Contribution

It introduces a comprehensive, scalable architecture for data acquisition, management, and distribution in smart cities, with practical guidelines validated through industrial experiments.

Findings

Effective data flow management improves real-time service responsiveness.

Scalable platform supports diverse smart city applications.

Guidelines enhance data handling efficiency in complex scenarios.

Abstract

Smart city management is going through a remarkable transition, in terms of quality and diversity of services provided to the end-users. The stakeholders that deliver pervasive applications are now able to address fundamental challenges in the big data value chain, from data acquisition, data analysis and processing, data storage and curation, and data visualisation in real scenarios. Industry 4.0 is pushing this trend forward, demanding for servitization of products and data, also for the smart cities sector where humans, sensors and devices are operating in strict collaboration. The data produced by the ubiquitous devices must be processed quickly to allow the implementation of reactive services such as situational awareness, video surveillance and geo-localization, while always ensuring the safety and privacy of involved citizens. This paper proposes a modular architecture to (i) leverage innovative technologies for data acquisition, management and distribution (such as Apache Kafka and Apache NiFi), (ii) develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in smart cities environment, and (iii) tackle the main issues in tasks involving complex data flows and provide general guidelines to solve them. We derived some guidelines from an experimental setting performed together with leading industrial technical departments to accomplish an efficient system for monitoring and servitization of smart city assets, with a scalable platform that confirms its usefulness in numerous smart city use cases with different needs.