Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning

TL;DR

This paper analyzes a year-long project on distributed real-time analytics using edge computing and machine learning, highlighting integration challenges, organizational barriers, and proposing strategies for improvement.

Contribution

It identifies critical challenges of big-bang integration in distributed projects and recommends early mock-based deployment and structured integration for better success.

Findings

Only six minutes of system functionality achieved versus 40 expected.

Poor communication and lack of early testing caused major setbacks.

Early mock deployment and structured integration improve project outcomes.

Abstract

This experience report analyses a one year project focused on building a distributed real-time analytics system using edge computing and machine learning. The project faced critical setbacks due to a big-bang integration approach, where all components developed by multiple geographically dispersed partners were merged at the final stage. The integration effort resulted in only six minutes of system functionality, far below the expected 40 minutes. Through root cause analysis, the study identifies technical and organisational barriers, including poor communication, lack of early integration testing, and resistance to topdown planning. It also considers psychological factors such as a bias toward fully developed components over mockups. The paper advocates for early mock based deployment, robust communication infrastructures, and the adoption of topdown thinking to manage complexity and reduce risk in reactive, distributed projects. These findings underscore the limitations of traditional Agile methods in such contexts and propose simulation-driven engineering and structured integration cycles as key enablers for future success.