A survey study of success factors in data science projects

TL;DR

This survey study explores success factors and project management practices in data science, revealing that most professionals use agile frameworks but few follow formal methodologies, emphasizing stakeholder needs and collaboration.

Contribution

Provides new empirical data on data science project management practices and identifies key success factors and differences based on methodology adherence.

Findings

Most use agile lifecycle, but only 25% follow formal methodologies.

Key success factors include stakeholder needs, communication, and collaboration.

Methodology adherents focus more on risks, version control, and data security.

Abstract

In recent years, the data science community has pursued excellence and made significant research efforts to develop advanced analytics, focusing on solving technical problems at the expense of organizational and socio-technical challenges. According to previous surveys on the state of data science project management, there is a significant gap between technical and organizational processes. In this article we present new empirical data from a survey to 237 data science professionals on the use of project management methodologies for data science. We provide additional profiling of the survey respondents' roles and their priorities when executing data science projects. Based on this survey study, the main findings are: (1) Agile data science lifecycle is the most widely used framework, but only 25% of the survey participants state to follow a data science project methodology. (2) The most important success factors are precisely describing stakeholders' needs, communicating the results to end-users, and team collaboration and coordination. (3) Professionals who adhere to a project methodology place greater emphasis on the project's potential risks and pitfalls, version control, the deployment pipeline to production, and data security and privacy.