How do practitioners gain confidence in assurance cases?

TL;DR

Practitioners mainly rely on qualitative methods like peer review and checklists for confidence in assurance cases, but face barriers such as subjectivity and lack of guidance, highlighting a gap between research and practice.

Contribution

This study provides empirical insights into current confidence assessment practices and barriers faced by practitioners in applying CAMs in assurance cases.

Findings

Peer review and checklists are most common CAMs.

Practitioners prefer qualitative over quantitative CAMs.

Barriers include extra work, lack of guidance, and trust issues.

Abstract

CONTEXT: Assurance Cases (ACs) are prepared to argue that the system's desired quality attributes (e.g., safety or security) are satisfied. While there is strong adoption of ACs, practitioners are often left asking an important question: are we confident that the claims made by the case are true? While many confidence assessment methods (CAMs) exist, little is known about the use of these methods in practice OBJECTIVE: Develop an understanding of the current state of practice for AC confidence assessment: what methods are used in practice and what barriers exist for their use? METHOD: Structured interviews were performed with practitioners with experience contributing to real-world ACs. Open-coding was performed on transcripts. A description of the current state of AC practice and future considerations for researchers was synthesized from the results. RESULTS: A total of n = 19 practitioners were interviewed. The most common CAMs were (peer-)review of ACs, dialectic reasoning ("defeaters"), and comparing against checklists. Participants preferred qualitative methods and expressed concerns about quantitative CAMs. Barriers to using CAMs included additional work, inadequate guidance, subjectivity and interpretation of results, and trustworthiness of methods. CONCLUSION: While many CAMs are described in the literature there is a gap between the proposed methods and needs of practitioners. Researchers working in this area should consider the need to: connect CAMs to established practices, use CAMs to communicate with interest holders, crystallize the details of CAM application, curate accessible guidance, and confirm that methods are trustworthy.