Expert Elicitation for Reliable System Design

TL;DR

This paper discusses how expert judgement can be effectively integrated into the reliability assessment process in systems engineering design, emphasizing the importance of problem structuring and bias mitigation.

Contribution

It introduces a holistic framework for incorporating expert judgement into reliability assessments throughout the system design lifecycle.

Findings

Expert judgement is crucial for problem structuring and failure mitigation.

Reliability assessments in design resemble a statistical control process.

Bias mitigation methods are necessary for rational consensus.

Abstract

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.