An Exploratory Study of Applying a Scrum Development Process for Safety-Critical Systems

TL;DR

This study explores applying an adapted Scrum process, S-Scrum, for safety-critical systems through an empirical case study, demonstrating its feasibility and safety benefits despite some agility trade-offs.

Contribution

It introduces an optimized S-Scrum process integrating safety analysis, validated through a real-world student project and industry feedback, advancing agile safety-critical system development.

Findings

S-Scrum ensures safety during each sprint with integrated safety analysis.

Optimizations improved safety and agility in the development process.

Industry feedback was positive, supporting feasibility in safety-critical contexts.

Abstract

Agile techniques recently have received attention in developing safety-critical systems. However, a lack of empirical knowledge of performing safety assurance techniques in practice, especially safety analysis into agile development processes prevents further steps. In this article, we aim at investigating the feasibility and the effects of our S-Scrum development process, and stepwise improving and proposing an Optimized S-Scrum development process for safety-critical systems in a real environment. We conducted an exploratory case study in a one-year student project "Smart Home" at the University of Stuttgart, Germany. We participated in the project and collected quantitative and qualitative data from questionnaire, interviews, participant observation, physical artifacts, and documentation review. Furthermore, we evaluated the Optimized S-Scrum in industry by conducting interviews. The first-stage results showed that by integrating STPA (System-Theoretic Process Analysis) can ensure the safety during each sprint and enhance the safety of delivered products, while the agility of S-Scrum is slightly worse than the original Scrum. Six challenges have been explored: Management changes the team's priorities during an iteration; Disturbed safety-related communication; Non-functional requirements are determined too late; Insufficient upfront planning; Insufficient well-defined completion criteria; Excessive time to perform upfront planning. We investigated further the causalities and optimizations. The second-stage results revealed that the safety and agility have been improved after the optimizations. We have gained a positive assessment and suggestions from industry. The optimized S-Scrum is feasible for developing safety-critical systems concerning the capability to ensure safety and the acceptable agility in a student project. Further attempt is still needed in industrial projects.

Tables

Table 1. Table 1: Research strategy in stage 1 (“DL”-Developer, “SH”-Stakeholder, “SM”-Scrum Master)

Time	Sprint 1 to sprint 5	Sprint 6 to sprint 7	Sprint 8	Sprint 9
Process	Scrum	S-Scrum	S-Scrum	S-Scrum
Data collection	Participant observation Scrum artifacts Documentation review	Participant observation Scrum artifacts Documentation review	Questionnaires	Semi-structured interviews
Participants	DLs SHs	DLs SHs	13 voluntary DLs	5 voluntary DLs 1 SM
Data types	Quantitative	Quantitative	Quantitative	Qualitative
Analysis	Sum of the numbers	Sum of the numbers	Median MAD	Coding
Output	No safety data	Safety data: M16.1-M16.3 M17.1-M17.3	Agility data: M1-M15	Challenges and optimizations of S-Scrum

Table 2. Table 2: Research strategy in stage 2 (“DL”-Developer, “SH”-Stakeholder, “SM”-Scrum Master, “PO”-Product Owner)

Time	Sprint 10 to sprint 11	Sprint 12	Sprint 13
Process	optimized S-Scrum	optimized S-Scrum	optimized S-Scrum
Data collection	Participant observation Scrum artifacts Documentation review	Questionnaires	Semi-structured interviews
Participants	DLs SHs	8 voluntary DLs	1 PO (from EPLAN) 1 SM (from EPLAN)
Data types	Quantitative	Quantitative	Qualitative
Analysis	Sum of the numbers (compare with the data from stage 1)	Median and MAD (compare with the data from stage 1)	Coding
Output	Safety data: M16.1-M16.3 M17.1-M17.3	Agility data: M1-M15	Preliminary discussion in industry

Table 3. Table 3: Normal Scrum, S-Scrum and optimized S-Scrum in Smart Home (“DL”-Developer, “SM”-Scrum Master, “PO”-Product Owner, “SE”-Safety Expert)

Normal Scrum	14 DLs 1 SM 1 PO	Sprint planning meeting Weekly Scrum meeting (2 times/week) Sprint review meeting Sprint retrospective meeting	Story map Product backlog Sprint backlog
S-Scrum	14 DLs 1 SM 1 PO 1 SE	Sprint planning meeting (with safety planning) Weekly Scrum meeting (2 times/week) (with safety discussion) Sprint review meeting (with safety review) Sprint retrospective meeting	Story map Functional product backlog Safety product backlog Sprint backlog
Optimized S-Scrum	13 DLs 1 SM 1 PO 1 external SE 1 internal SE	Pre-planning meeting Sprint planning meeting (brainstorming requirements and criteria) Weekly Scrum meeting (2 times/week) Weekly safety meeting (1 time/week) Sprint review meeting (with safety review) Sprint retrospective meeting	Story map (with safety epics) Functional product backlog Safety product backlog (with safety stories) Sprint backlog Safety plan

Full text

1 Introduction

To reduce the risks and costs for reworking and rescheduling, agile techniques have aroused attention for the development of safety-critical systems. Traditionally standardised safety assurance, such as IEC 61508 [1], is based on the V-model. Even though there is no prohibition to adapt standards for lightweight development processes with iterations, some limitations cannot be avoided during the adaptation [2]. Existing research in agile techniques for safety-critical systems is striving for consistency to standards. Safe Scrum [3] is a considerable success due to a comprehensive combination between Scrum and IEC 61508. However, an integrated safety analysis to face the changing architectures inside each sprint still needs to be enhanced. Therefore, in 2016, we proposed S-Scrum to integrate a systems theory based safety analysis technique, STPA (System-Theoretic Process Analysis) [7], which was proposed by Leveson in 2012, inside each sprint to guide a safe design [16].

Problem statement. We proposed to integrate STPA in a Scrum development process to enhance the safety in agile development. However, it has not been validated in practice. As far as we know, there exists no empirical data on applying Scrum for a safety-critical project with the integration of STPA.

Research objective and research questions. In this article, we aim to explore the agility and safety of S-Scrum as well as challenges and their relevant optimizations for developing a safety-critical system called “Smart Home”. The research questions are as follows:

RQ 1 How does S-Scrum handle agility and safety in safety-critical systems?

RQ 2 What are the challenges of S-Scrum in such a context?

RQ 3 How could S-Scrum be optimized to overcome the challenges?

RQ 4 What are the effects of the optimized S-Scrum on safety and agility?

Contribution. This paper provides the first case study on applying a Scrum development process for safety-critical systems. We investigated the effects and challenges of S-Scrum in the 1st stage of the case study. We proposed an optimized S-Scrum and validated it in the 2nd stage of the case study. To this end, we preliminarily discussed the optimized S-Scrum in industry.

Outline. The paper is organized as follows. First, we present the related work on using Scrum for safety-critical systems and normal Scrum development process improvement (Sect. 2). Then, we present the background about STPA and our previous work about S-Scrum (Sect. 3). After that, we describe the approach and results of the 1st stage of the case study (Sect. 4.1), and the 2nd stage of the case study (Sect. 4.2). Finally, we discuss the threats to validity (Sect. 5), and draw the conclusions (Sect. 6).

2 Related Work

To the best of our knowledge, few empirical studies of applying Scrum or other agile processes for safety-critical systems exist. Most of the research is still in the stage of theoretical illustration and validation [8] [9].

Safe Scrum is a Scrum development process for safety-critical systems, which was developed to adhere to the general functional safety standard IEC 61508 [3] [6]. Previous research of Safe Scrum has been synergized with other safety standards in different domains [4] [5]. However, purely theoretical validation is unable to cover the details of the process. More practical experiences are becoming crucial.

Despite the limited practical experiences in applying Scrum for safety-critical systems, there are a lot of Scrum development process experiences that could be taken as a reference for the agile software process improvement of our project [25] [26]. Diebold et al. [10] investigated the industrial usage of Scrum with various sprint length, events, team size, requirements engineering, roles, effort estimations and quality assurance. Cho [11] conducted an in-depth case study in two organizations. The data was analyzed along 4 dimensions, including human resource management; structured development process; environment; information systems and technology. These factors were covered in our assessment of agility considering our criteria to improve the S-Scrum.

3 STPA and S-Scrum

STPA is a new hazard analysis technique by Leveson in 2012. It has been successfully used in various domains, such as aviation, automobiles and healthcare. Compared with the traditional safety analysis techniques, such as FMEA (Failure Mode and Effects Analysis) and FTA (Fault tree analysis), STPA bases on the systems theory rather than the traditional reliability theory. Due to an increasing complexity of systems, the accidents are not caused by single function failures or chains of failure events, but resulted from inadequate control actions. To ensure the safety of today’s complex systems, the use of STPA is becoming necessary. Besides, we proposed using STPA in a Scrum development process [16], as current safety analysis techniques start from a complete design, which is not consistent to agile methodologies, which advocate a lightweight up-front planning and design. STPA, on the contrary, provides the necessary information to start from a high-level architecture and to guide the incremental design process. In S-Scrum, we integrate STPA mainly in three aspects: (1) During each sprint, we integrate STPA as safety-guided design. (2) At the end of each sprint, we use STPA on the product instead of a Reliability, Availibility, Maintainability and Safety (RAMS) validation. (3) We replace the final RAMS validation with STPA. The other parts are kept consistent to Safe Scrum: (1) The environment description and the SSRS phases 1-4 (concept, overall scope definitions, hazard and risk analysis and overall safety requirements). (2) Test Driven Development. (3) Safety product backlog. (4) A safety expert [33]. We aim to fill the gap of a lack of safety analysis in agile development and enhance the safety on the basis of a standard-based Scrum development process for safety-critical systems.

4 Case Study

To explore S-Scrum further, we conduct this study following the guideline by Runeson [17] and Yin [18]. We design this case study with a multi-staged procedure. Each stage has different objectives and research questions. We explored the challenges and optimizations in S-Scrum in stage 1, while we validated the optimized S-Scrum in stage 2.

4.1 Research Context

The case study (including stage 1 and stage 2) was performed in the project developing safety-critical systems, Smart Home, between March, 2016 and March, 2017 at the Institute of Software Technology, University of Stuttgart. The project had 400 planned working hours per head with a headcount of 14 students. The students have taken part in a training program for agile development and STPA before joining the project and a course on automation systems during the project. The Scrum Master was one research assistant with experienced project management background, while the Product Owner and Safety Expert was another research assistant majoring in using agile for safety-critical systems. All the students were supervised by three research assistants. The project was to work on an IoT based smart home with a smart coffee machine, smart light alarm system, autonomous parking system, door-open system, and smoke detector alarm system through the IoT server - KAA111https://www.kaaproject.org/overview/. The project “Smart Home” is openly available in GitHub222https://github.com/ywISTE/student-project—Smart-Home.

4.2 Case study - stage 1

The objective of stage 1 is to validate the safety and agility of S-Scrum and optimize it. In stage 1, we focus on answering RQ 1, RQ 2, and RQ 3. The general research strategy in stage 1 is shown in Table 1.

4.2.1 Data collection in stage 1

Stage 1 spans from sprint 1 to sprint 9. Each sprint lasts three weeks. The agility-related quantitative data, M1 to M15, were collected through 13 questionnaires333The questionnaire is available: https://zenodo.org/record/439696#.WODCovl96Uk. Our participant observation as the Product Owner (the first author), the Scrum Master, and the customer imposed also an evaluation and review of the results. The safety-related data, M16.1 to M16.3 and M17.1 to M17.3, were quantitatively collected during sprint 6 and sprint 7. From sprint 1 to sprint 5, we executed normal Scrum without safety analysis for the adaptation and preparation for the project. The STPA was performed by the safety expert and recorded privately by using the STPA tool, XSTAMPP444http://www.xstampp.de/, while the hazards and safety requirements were recorded in the safety product backlog in Jira.

Based on the quantitative data for agility and safety, we then designed semi-structured interviews with 6 voluntary participants from the development team, including the Scrum Master and five developers. The interviews lasted 270 minutes overall. The questions began with a specific set of questions regarding the observations. Further, we asked about the causalities. Finally, the optimizations were collected in an open-ended mode. The interview guideline555The interview guideline is available: https://zenodo.org/record/439696#.WODCovl96Uk was provided before each interview. We recorded interview data in field notes and we used the audio recordings for text transcription.

4.2.2 Data analysis in stage 1

We analyzed the data using the combination of GSN [14] and GQM [15] referring partially to the VMF framework [13], as shown in Fig. 1. The data are from two aspects: agility (S1) and safety (S2). To evaluate and optimize agility (S1), we set 15 goals (G1 to G15) considering Comparative Agility Survey [12]. They are: G1 (Team work composition); G2 (Team work management); G3 (Communication); G4 (Requirement emergency); G5 (Technical design); G6 (Planning levels); G7 (Critical variables); G8 (Progress tracking); G9 (Sources of dates and estimates); G10 (When do we plan); G11 (Customer acceptance test); G12 (Timing); G13 (Quality focus); G14 (Reflection); G15 (Outcome measure). To reach G1 to G15, we analyzed M1 to M15 indirectly by setting sub-metrics. For example, M1 (Team work composition) was analyzed by M1.1 (Team members are kept as long as possible), M1.2 (Specialists are willing to work outside their specialty to achieve team goals), M1.3 (Everyone required to go from requirements to finished system is on the team), and M1.4 (People are no more than two teams). Each sub-metric was analyzed on an ordinal scale of 5 (e.g., from 1 to 5 means “Negative”, “More negative than positive”, “Neither negative nor positive”, “More positive than negative”, and “Positive”). To investigate the in-depth challenges, we found out either the negative values of the results or the significant differences between the normal Scrum and S-Scrum to formulate further interview questions. To analyze the interview results, we used NVivo11 for text encoding [19]. Concerning safety, G16 is extended with 3 questions together with 3 metrics including: number of software hazards (M16.1), number of software safety requirements (M16.2), and number of safety requirements traceable to hazards (M16.3). G17 is extended to be evaluated by the number of mitigated hazards (M17.1), number of accepted safety requirements (M17.2) in the present sprint, and number of rejected safety requirements (M17.3) in the project.

4.2.3 Results in stage 1 - RQ 1: How does S-Scrum handle agility and safety in safety-critical systems?

We investigate the effect on agility by comparing the normal Scrum and the S-Scrum according to the 15 metrics in Fig. 2. From the general overview, we can conclude that most of the values regarding agility in S-Scrum are slightly worse than those in the normal Scrum, while one metric shows strongly negative values (“when do we plan”). We discussed the results with the technical support from the Comparative Agility Survey and got the feedback: when most of the values are more positive than negative (more than “3”), we could say that the process is agile enough. Moreover, most values show relatively small differences between normal Scrum and S-Scrum. Thus, we consider the agility of S-Scrum to be acceptable. Yet, optimizations are needed. Regarding the safety of S-Scrum, we performed STPA two rounds in sprint 6. We found 6 software hazards (M16.1) and 15 safety requirements (M16.2), which can all be traced back to software hazards (M16.3). Three hazards were mitigated (M17.1), while 14 safety requirements were accepted (M17.2). In sprint 7, we performed two rounds of STPA analysis. We found 10 software hazards (M16.1) and 24 safety requirements (M16.2), which can also all be traced back to software hazards (M16.3). Six hazards were mitigated (M17.1), while 23 safety requirements were accepted (M17.2). Each sprint has 1 rejected safety requirement due to hardware limitation (M17.3).

4.2.4 Results in stage 1 - RQ 2 & RQ 3: What are the challenges of S-Scrum in such context? & How could S-Scrum be optimized to overcome the challenges?

To optimize S-Scrum, we derived six challenges from the six abnormal values (see data analysis in stage 1) from the sub-metrics inside these 15 metrics.

Challenge 1: The priority management of safety requirements and functional requirements has conflict. In the normal Scrum, the management and development team determine the sprint backlog with functional requirements in the sprint planning meeting. All the team members have a clear overview of and commitment to the sprint plan with relatively high-level features. The developers accomplish each item with their own detailed tasks. The requirements from the management and the concrete realizations from the developer reach a consensus during each sprint. In S-Scrum, the integrated STPA and the safety requirements break the balance. The functional requirements are correlated with the safety requirements. However, some developers preferred: functional requirements are more important than the safety requirements. It was found that the need for long-term quality was given a lower priority than the need for short-term progress [29]. Moreover, the safety expert spent a relatively short time working with the team members which influences also the decision making. As one developer mentioned: The safety expert is not working in the same room with the development team and has an inconsistent working time. Thus, a lack of an in-time decision maker on the safety requirements together with the ignorance of safety requirements in the development team cause the conflict.

To face this challenge, a safety culture should be integrated into a light-weight development process. We suggest to include an internal safety expert in the development team to (1) spread the safety culture; (2) increase the safety expert’s working time with the team members; (3) clarify the bewilded safety requirements. An external safety expert is necessary to keep the communication with other stakeholders. To fill the gap between the external safety expert and the development team, the development team suggests that the external safety expert should join at least once the weekly Scrum meeting. The discussion between the management, the external safety expert and the internal safety expert could strive a fresh balance on the priorities.

Challenge 2: The communication between team members and safety expert is disturbed. To start with, the unclear safety-related documentation influences an effective communication. The team members mentioned: it is difficult to comprehend the purpose of the safety expert and integrate into our daily work from the existing documents. Moreover, a lack of safety-related knowledge of the development team influences the discussion concerning safety issues. Finally, the insufficient time spent between safety expert and development team causes also a poor communication. Without a non-obstacle work place to communicate within the team about the work progress, the safety assurance could either be a superficial decoration or even worse, a roadblock during fast product delivery.

To face this challenge, in addition to the separated internal safety expert and external safety expert, a weekly safety meeting is suggested by an interviewee: The internal safety expert and external safety expert should meet each other at least once a week to exchange the status of the development team. Because the discussion should be deep in the safety area, it is not supposed to be established during the normal weekly Scrum meeting. Last but not least, we improve our safety epics and safety stories to support an effective communication [34], as shown in Sect. 4.3 (Optimized S-Scrum).

Challenge 3: The safety requirements are not determined early enough to appropriately influence design and testing. In sprint 6 and sprint 7, the safety requirements were determined by the development team and the safety expert together in the sprint planning meeting. However, as one interviewee mentioned: the determination of safety requirements from the safety product backlog is too late to avoid a conflict between the functional requirements and their suitability for the coming sprint. Thus, sometimes the functional design and testing have to start without the in-time safety requirements.

To face this challenge, we propose a pre-planning meeting for solving the time pressure problem. First, the internal, external safety experts and product owner discuss the safety product backlog and the functional product backlog in the pre-planning meeting. Then they brainstorm the results with the whole development team in the sprint planning meeting to gather more ideas and make each safety requirement clear.

Challenge 4: The planning at the start of each iteration is insufficient. In the normal Scrum, the development team and the product owner plan the upcoming sprint in the sprint planning meeting by formulating the sprint backlog with estimated items, which makes the development team sufficiently confident about their plan. However, the estimation and planning for the safety product backlog seem not ideal, as well as the interconnection with the functional product backlog, which make an in-time identification of the sprint backlog difficult. An interviewee said: It is difficult to determine the safety requirements when the development team has not planned the functional requirements for the coming sprint.

To face this challenge, we suggest and adapt an agile safety plan [21] in connection with the pre-planning meeting to increase the understanding of safety issues and enhance confidence. In our project, the results of STPA are part of the agile safety plan.

Challenge 5: The time to perform upfront planning is late. A team member said: the pre-planning meeting for safety issues should start before the sprint planning meeting. But the concrete time should be decided between the external safety expert, the internal safety expert and the product owner. Based on the experience of the previous sprints, it is better to start upfront planning one week before the sprint planning meeting (3 weeks/sprint). The time could be changed depending on the sprint length. More explanations are in challenge 4.

Challenge 6: The safety requirements lack well-defined completion criteria. In the normal Scrum, we have various testing methods to determine the completion of each feature such as unit testing, system testing, regression testing, and acceptance testing, which are promoted to be automated in an agile context. However, few agile testing methods are suitable for validating safety requirements, as the safety requirements are either from standard requirements or the safety analysis, which differentiates safety testing and functional testing. In S-Scrum, we use UAT (User Acceptance Testing) for validating safety requirements. Thus, a suitable safety criterion becomes important.

To face this challenge, we use a “Given-When-Then” format [23] as safety requirements’ criteria. The development team suggest that the external safety expert could decide the safety stories’ criteria and the internal safety expert could decide the safety tasks’ criteria. The whole development team could brainstorm both criteria. To this end, the product owner and safety expert perform the acceptance testing.

4.3 Case study - stage 2

After the optimizations described above, the objective of stage 2 is to validate the safety and agility of the optimized S-Scrum and discuss it in industry. We focus on answering the RQ 4 together with some discussion from industry. The general research strategy in stage 2 is shown in Table 2.

4.3.1 Optimized S-Scrum

To have a clear overview, we compare the optimized S-Scrum to the normal Scrum and the S-Scrum in our project respectively in Table 3. In the optimized S-Scrum, we differentiate between an internal safety expert and an external safety expert. A pre-planning meeting and weekly safety meetings are established between safety experts. We include the safety epics, to satisfy $<$the overall safety needs$>$, the system must $<$always be able to reach a safe state$>$ [22], in the story map. The safety product backlog is improved with optimized safety story: To keep $<$control action$>$ safe, the system must $<$achieve or avoid something$>$. An agile safety plan based on STPA technology is suggested for a clear overview. The safety culture is expected to be enhanced by the additional activities.

4.3.2 Data collection in stage 2

Stage 2 is from sprint 10 to sprint 13. The safety-related data, M16.1 to M16.3 and M17.1 to M17.3, were collected in the same way as in stage 1. The safety results were collected by both internal and external safety experts. The agility-related data, M1 to M15, were collected by the second round questionnaires666The questionnaire is available: https://zenodo.org/record/439696#.WODCovl96Uk. We further discussed the optimized S-Scrum by conducting 2 semi-structured interviews with one Scrum Master and one Product Owner from EPLAN GmbH, Germany. The interview lasted 2 hours. We formulated questions about the status of the Scrum development process in the company projects; the feasibility of the optimized S-Scrum in industry; and further suggestions from the industrial perspective. A project background illustration was provided before the interviews, together with the interview guidelines777The interview guideline is available: https://zenodo.org/record/439696#.WODCovl96Uk. The field notes, interview transcripts, and voice recordings were all preserved for backup.

4.3.3 Data analysis in stage 2

The quantitative data were compared with the numbers in stage 1. The interview results from the industry were text encoded with: status, challenges, possible solutions, and the feasibility of S-Scrum.

4.3.4 Results in stage 2 - RQ 4: What are the effects of the optimized S-Scrum on safety and agility?

As shown in Fig. 3, most of the evaluated agility aspects sustained a good level of satisfaction with little variance. However, the “technical design” is slightly reduced. Due to the new role, the collaborative part of design between safety work and development work fell on the internal safety expert. The personal capability is becoming important. To improve the technical design, cooperation shall increase between the external safety expert and the development team.

Regarding the safety of optimized S-Scrum, as we can see in Fig. 4, safety aspects improved (M16.1, M16.2, M16.3, M17.1, M17.2). We also rejected few safety requirements (M17.3): 1 (sprint 6), 1 (sprint 7), 0 (sprint 10), 2 (sprint 11). We can conclude that, in general, the optimized S-Scrum has better safety assurance capabilities. However, there are still some abnormal values in sprint 7. The number of safety requirements, the number of safety requirements traceable to hazards and the number of accepted safety requirements in sprint 7 are more than in sprint 10. This may be traced back to the fitting-in phase of the optimized S-Scrum. Since the training of STPA for the internal safety expert, we finished STPA in sprint 10 only once. In sprint 6, sprint 7, and sprint 11, we finished STPA twice. After the adaption of the new role, the safety data rose in sprint 11.

4.3.5 Results in stage 2 - Discussion

To strength the study further, we discussed our results preliminarily in industry. For Challenge 1, the conflict between functional requirements and non-functional requirements seems not obvious. As one interviewee mentioned: Since we have a relative small amount of non-functional requirements, the priorities are always determined by the product owner together with the discussion with some external experts. For Challenge 2, one interviewee mentioned: To enhance the communication between the team members and the experts, we have a technical meeting before each sprint planning meeting. The product owner sends the emails to the relevant experts depending on the goals of each sprint. The experts are welcomed to join the daily stand-up meetings. Thus, the experts have sufficient time to keep up with the development team, while the technical knowledge is deeply discussed in the technical meeting before the sprint planning meeting. The project has also a good knowledge sharing mechanism to support the communication during each sprint. One interviewee mentioned: We use pair programming, formal guidelines to teach new colleagues, chat clients, and screen sharing. When the team includes experts, the product owner will contact 2-3 colleagues to discuss technical stuff, who will inform other colleagues. A hierarchical communication mode is preferred for a multi-expert team. For Challenge 3, the industrial projects have also mentioned this problem: Internal user stories are used to record the non-functional requirements. The execution of internal user stories is up to the team. For Challenge 4, the two teams execute a sufficient planning. An interviewee mentioned: We have a refinement time slot to get all product backlog items approved (each team member has understood) and not so much discussion in the sprint planning meeting. The team members are beginning the refinement in the present sprint for the user stories in the next sprint. In Scrum, not all requirements have to be at the same level of detail at the same time [24]. The progressive refinement could be further extended for the safety planning and assessment to: (1) avoid a premature development decision from the high-level safety requirements; (2) reserve sufficient time for managing priorities between safety requirements and functional requirements; (3) increase the rework possibilities; (4) enhance the likelihood of using conversation to clarify safety requirements. That could also illustrate the Challenge 5. For Challenge 6, the refinement phase helps building a pre-understanding of each requirement and reaching a common criterion in the sprint planning meeting. The external expert is a regular member in industry. An interviewee mentioned: We prefer some experts with deep knowledge in the team, but the arrangement of an internal expert has to take more issues into account, such as training, responsibility, and even personal development. An external safety consultant to test the products and delivered trainings and an internal safety initiative [20] to promote safety practices across groups in industry could be align with our internal and external safety expert. Safety culture in industry is enhanced either by setting the regulations or by the established organization structure and activities. An agile safety plan is also required from some standards. They draw the safety plan either in the technical meeting or in parallel with the refinement. The technical meeting suggested in industry could also be considered as an extra (weekly) safety meeting. The pre-planning meeting seems to be a suitable form for realizing progressive refinement in industry. This alignment motivates more combinations between our optimizations and existing industrial practices. All the requirements and acceptance criteria are retrieved by brainstorming. An effective communication plays a vital role in executing acceptance testing.

5 Threats to validity

Construct validity: The first threat to construct validity is the general data analysis framework. To apply Scrum for safety-critical systems, we focus primarily on safety aspect and agility aspect in our exploratory study. In terms of agility, we referred to an official agility comparative survey [12] for ensuring the coverage of measurement. In terms of safety, S-Scrum was extended from Safe Scrum, which was originally developed in accordance with the general functional safety standard IEC 61508. Thus, the validation regarding to the consistency with IEC 61508 has not been included in the framework. Furthermore, in S-Scrum we mainly integrate STPA. We aim to validate the enhanced safety concerning the integrated safety analysis technique. Thus, the safety assurance technique’s capability and the deliverable products’ safety are set as two relevant goals. Yet, the goals and metrics seem not enough and the validation framework is possible to be extended. The second threat to construct validity is the validation periods for S-Scrum and optimized S-Scrum are shorter than our expectations. We executed the normal Scrum in the first five sprints to strengthen students’ background knowledge of agile techniques and prepare the detailed organization structure, which took us a lot of time.

Internal validity: The first threat to internal validity is the arrangement of team roles. One of the authors acted as the product owner and the safety expert concurrently in sprint 6 and sprint 7. To avoid this threat in alignment with the optimizations in sprint 10 and sprint 11, the product owner acted further as an external safety expert. An internal safety expert has been arranged in the development team. The second threat to internal validity exists in the qualitative data from the semi-structured interviews. The interviews have been performed by one of the authors together with the audio record. The language we used has also partial German. To avoid subjective and language bias, the audio recording has been transcribed independently by two researchers (one is a native German speaker) and compared to formulate a final result.

External validity: A student project is different from an industrial project. However, Höst et al. [30], Tichy, Kitchenham et al. [31] proposed that students could be acceptable. To consider this debatable issue, we mainly referred to an empirical study conducted by Falessi in 2017 [36]. 16 statements are provided by 65 empirical researchers. They mentioned: Conducting experiments with professionals as a first step should not be encouraged unless high sample sizes are guaranteed or performing replicas is cheap. In our research, there exists few industrial projects for developing safety-critical systems fully adopted a Scrum development process according to the preliminary research [35]. S-Scrum was also proposed in 2016 as a high-level process model. In addition, the long learning cycles and a new technology are two hesitations for using professionals. STPA was developed in 2012. In industry, there is still a lack of experts. Thus, we believe that in our research area, a student project is a relative suitable way to aggregate contributions. Even though, the generalizability is considered critical.

Reliability: The student project is a suitable way for a first validation. Yet, the results from the students are limited by their personal experience. Besides, the “grading power” of the researchers may influence the results. We separated our research work from the final examination of the product to mitigate this threat.

6 Conclusion

The main benefit of our research is that it provides a first empirical and practical insight into applying Scrum for safety-critical systems with the integration of STPA. Moreover, the presented challenges existing in priority management, communication, time pressure on determining safety requirements, safety planning, safety requirements’ acceptance criteria and solutions including the split of the safety expert, pre-planning meeting, regular safety meeting, improved safety epics, STPA-based safety stories and an agile safety plan could arouse interest in practitioners and show future research directions. The effects on safety and agility aspects indicate the feasibility to align STPA with a Scrum development process. The discussion in industry motivates the further step of transmitting the optimized S-Scrum from the academic environment towards industry environment. However, the execution of S-Scrum and optimized S-Scrum was in a specific context. We can rely our improvements on an academical project only. The generalization in industry of the optimizations remains subject to future work. Finally, regarding safety and security in agile development in today’s cyber-physical systems, even though special attention has to be paid to the respective norms and standards, problems’ exploration in practice seems also necessary.

7 Acknowledgements.

We want to thank Dr. A. Nguyen-Duc for proof reading and his valuable suggestions. We are grateful to all participants involved during the case study. Finally, we want to thank all the feedback on previous versions. The first author is supported by the LGFG (Stipendien nach dem Landesgraduiertenfördergesetz).