Dealing with Data Challenges when Delivering Data-Intensive Software Solutions

TL;DR

This study explores the challenges faced by multi-disciplinary data-intensive software teams, identifying key issues with data access, quality, and understanding, and proposing strategies like governance and team development to address them.

Contribution

It presents a grounded theory of data challenges in MDSTs, detailing causes, consequences, and strategies, based on interviews with practitioners across roles and organizational levels.

Findings

Data access and quality are primary challenges.

Strategies include data governance and advanced tools.

Team dynamics and communication are crucial for success.

Abstract

The predicted increase in demand for data-intensive solution development is driving the need for software, data, and domain experts to effectively collaborate in multi-disciplinary data-intensive software teams (MDSTs). We conducted a socio-technical grounded theory study through interviews with 24 practitioners in MDSTs to better understand the challenges these teams face when delivering data-intensive software solutions. The interviews provided perspectives across different types of roles including domain, data and software experts, and covered different organisational levels from team members, team managers to executive leaders. We found that the key concern for these teams is dealing with data-related challenges. In this paper, we present the theory of dealing with data challenges that explains the challenges faced by MDSTs including gaining access to data, aligning data, understanding data, and resolving data quality issues; the context in and condition under which these challenges occur, the causes that lead to the challenges, and the related consequences such as having to conduct remediation activities, inability to achieve expected outcomes and lack of trust in the delivered solutions. We also identified contingencies or strategies applied to address the challenges including high-level strategic approaches such as implementing data governance, implementing new tools and techniques such as data quality visualisation and monitoring tools, as well as building stronger teams by focusing on people dynamics, communication skill development and cross-skilling. Our findings have direct implications for practitioners and researchers to better understand the landscape of data challenges and how to deal with them.