Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods

TL;DR

This paper provides comprehensive guidelines and strategies for conducting empirical research on formal methods and tools, addressing their unique challenges and maturity levels.

Contribution

It introduces nine distinct empirical study strategies tailored for formal methods and discusses their application difficulties and validity threats.

Findings

Nine study strategies outlined for empirical formal methods

Discussion of challenges and validity threats in applying these strategies

Guidelines linked to external standards and other fields

Abstract

Empirical studies on formal methods and tools are rare. In this paper, we provide guidelines for such studies. We mention their main ingredients and then define nine different study strategies (laboratory experiments with software and human subjects, usability testing, surveys, qualitative studies, judgment studies, case studies, systematic literature reviews, and systematic mapping studies) and discuss for each of them their crucial characteristics, the difficulties of applying them to formal methods and tools, typical threats to validity, their maturity in formal methods, pointers to external guidelines, and pointers to studies in other fields. We conclude with a number of challenges for empirical formal methods.