Towards a More Systematic Approach to Secure Systems Design and Analysis

TL;DR

This paper investigates the variability among security experts' assessments of system security, aiming to improve decision-making consistency and reliability in secure system design.

Contribution

It introduces a systematic approach to measure expert opinion variability and demonstrates how aggregated expert assessments can produce a coherent security consensus.

Findings

Expert assessments vary significantly individually

Aggregation of opinions yields a consistent security view

Consensus assessments can inform better system design decisions

Abstract

The task of designing secure software systems is fraught with uncertainty, as data on uncommon attacks is limited, costs are difficult to estimate, and technology and tools are continually changing. Consequently, experts may interpret the security risks posed to a system in different ways, leading to variation in assessment. This paper presents research into measuring the variability in decision making between security professionals, with the ultimate goal of improving the quality of security advice given to software system designers. A set of thirty nine cyber-security experts took part in an exercise in which they independently assessed a realistic system scenario. This study quantifies agreement in the opinions of experts, examines methods of aggregating opinions, and produces an assessment of attacks from ratings of their components. We show that when aggregated, a coherent consensus view of security emerges which can be used to inform decisions made during systems design.