Effect of Requirements Analyst Experience on Elicitation Effectiveness: A Family of Empirical Studies

TL;DR

This study investigates how different types of experience affect requirements analysts' effectiveness in eliciting requirements, revealing that domain familiarity influences the impact of interview and professional experience.

Contribution

It provides empirical evidence on the nuanced effects of various experience types on analyst performance across familiar and unfamiliar domains.

Findings

Interview experience positively impacts effectiveness in familiar domains.

Professional experience has a moderate negative effect in familiar domains.

In unfamiliar domains, experience does not significantly influence effectiveness.

Abstract

Context. Nowadays there is a great deal of uncertainty surrounding the effects of experience on Requirements Engineering (RE). There is a widespread idea that experience improves analyst performance. However, there are empirical studies that demonstrate the exact opposite. Aim. Determine whether experience influences requirements analyst performance. Method. Quasi-experiments run with students and professionals. The experimental task was to elicit requirements using the open interview technique immediately followed by the consolidation of the elicited information in domains with which the analysts were and were not familiar. Results. In unfamiliar domains, interview, requirements, development, and professional experience does not influence analyst effectiveness. In familiar domains, effectiveness varies depending on the type of experience. Interview experience has a strong positive effect, whereas professional experience has a moderate negative effect. Requirements experience appears to have a moderately positive effect; however, the statistical power of the analysis is insufficient to be able to confirm this point. Development experience has no effect either way. Conclusion. Experience effects analyst effectiveness differently depending on the problem domain type (familiar, unfamiliar). Generally, experience does not account for all the observed variability, which means there are other influential factors.