A consistent multi-user framework for assessing system performance

TL;DR

This paper introduces a comprehensive mathematical framework for evaluating sonar system performance across multiple users and criteria, accounting for uncertainties and conflicting requirements, to improve procurement and assessment processes.

Contribution

It presents a novel, general set-theoretic measure of effectiveness that integrates multiple performance aspects, user requirements, and uncertainties into a unified assessment framework.

Findings

Framework successfully assesses multi-user performance scenarios.

Handles conflicting requirements and multiple data sources.

Applied to real sonar system assessment examples.

Abstract

Agreeing suitability for purpose and procurement decisions depend on assessment of real or simulated performances of sonar systems against user requirements for particular scenarios. There may be multiple pertinent aspects of performance (e.g. detection, track estimation, identification/classification and cost) and multiple users (e.g. within picture compilation, threat assessment, resource allocation and intercept control tasks), each with different requirements. Further, the estimates of performances and the user requirements are likely to be uncertain. In such circumstances, how can we reliably assess and compare the effectiveness of candidate systems? This paper presents a general yet simple mathematical framework that achieves all of this. First, the general requirements of a satisfactory framework are outlined. Then, starting from a definition of a measure of effectiveness (MOE) based on set theory, the formulae for assessing performance in various applications are obtained. These include combined MOEs, multiple and possibly conflicting user requirements, multiple sources and types of performance data and different descriptions of uncertainty. Issues raised by implementation of the scheme within a simulator are discussed. Finally, it is shown how this approach to performance assessment is used to treat some challenging examples from sonar system assessment.