On the Importance of Reproducibility of Experimental Results Especially in the Domain of Security

TL;DR

This paper emphasizes the critical need for reproducibility in security research, highlighting discrepancies between datasheet specifications and real-world measurements, and advocates for standardized data reporting to enhance experimental reliability.

Contribution

It reports experimental verification of datasheet parameters, revealing significant deviations, and proposes verification and standardized reporting practices to improve reproducibility in security research.

Findings

Significant discrepancies between datasheet data and real-world measurements.

Verification of equipment datasheets is essential for reproducibility.

Standardized units should be used for measurement reporting.

Abstract

Security especially in the fields of IoT, industrial automation and critical infrastructure is paramount nowadays and a hot research topic. In order to ensure confidence in research results they need to be reproducible. In the past we reported [18] that in many publications important information such as details about the equipment used are missing. In this paper we report on our own experiments that we run to verify the parameters reported in the datasheets that came along with our experimental equipment. Our results show that there are significant discrepancies between the datasheets and the real world data. These deviations concern accuracy of positions, movements, duration of laser shots etc. In order to improve reproducibility of results we therefore argue on the one hand that research groups verify the data given in datasheets of equipment they use and on the other hand that they provide measurement set-up parameters in globally accepted units such as cm, seconds, etc.