Managing Technical Debt in a Multidisciplinary Data Intensive Software Team: an Observational Case Study

TL;DR

This paper presents an observational case study on how multidisciplinary data-intensive software teams identify, assess, and manage technical debt, focusing on technical data components and pipeline debt, highlighting the need for new tools and practices.

Contribution

It provides empirical insights into technical debt management practices specific to multidisciplinary data-intensive teams, an area with limited prior research.

Findings

Identification of technical data components debt and pipeline debt.

Description of how teams assess and treat technical debt.

Discussion on the need for new patterns and tool support.

Abstract

Context: There is an increase in the investment and development of data-intensive (DI) solutions, systems that manage large amounts of data. Without careful management, this growing investment will also grow associated technical debt (TD). Delivery of DI solutions requires a multidisciplinary skill set, but there is limited knowledge about how multidisciplinary teams develop DI systems and manage TD. Objective: This research contributes empirical, practice based insights about multidisciplinary DI team TD management practices. Method: This research was conducted as an exploratory observation case study. We used socio-technical grounded theory (STGT) for data analysis to develop concepts and categories that articulate TD and TDs debt management practices. Results: We identify TD that the DI team deals with, in particular technical data components debt and pipeline debt. We explain how the team manages the TD, assesses TD, what TD treatments they consider and how they implement TD treatments to fit sprint capacity constraints. Conclusion: We align our findings to existing TD and TDM taxonomies, discuss their implications and highlight the need for new implementation patterns and tool support for multidisciplinary DI teams.