The Viability of Continuous Experimentation in Early-Stage Software Startups: A Descriptive Multiple-Case Study

TL;DR

This study explores the adoption of continuous experimentation in early-stage startups, revealing that resource constraints, prior experience, and immediate perceived benefits influence its implementation.

Contribution

It provides empirical insights into factors affecting continuous experimentation adoption in small startups, highlighting practical barriers and conditions for successful implementation.

Findings

Practices like iterative development are common in startups.

Systematic experimentation is rare among the studied startups.

Adoption depends on prior experience, resources, and immediate benefits.

Abstract

Background: Continuous experimentation (CE) has been proposed as a data-driven approach to software product development. Several challenges with this approach have been described in large organisations, but its application in smaller companies with early-stage products remains largely unexplored. Aims: The goal of this study is to understand what factors could affect the adoption of CE in early-stage software startups. Method: We present a descriptive multiple-case study of five startups in Finland which differ in their utilisation of experimentation. Results: We find that practices often mentioned as prerequisites for CE, such as iterative development and continuous integration and delivery, were used in the case companies. CE was not widely recognised or used as described in the literature. Only one company performed experiments and used experimental data systematically. Conclusions: Our study indicates that small companies may be unlikely to adopt CE unless 1) at least some company employees have prior experience with the practice, 2) the company's limited available resources are not exceeded by its adoption, and 3) the practice solves a problem currently experienced by the company, or the company perceives almost immediate benefit of adopting it. We discuss implications for advancing CE in early-stage startups and outline directions for future research on the approach.