A Method for the Characterisation of Observer Effects and its Application to OML

TL;DR

This paper introduces a method to evaluate observer effects of instrumentation tools, applies it to OML in wireless research, and finds that OML generally does not significantly bias measurements and can sometimes enhance performance.

Contribution

It presents a novel methodology for assessing observer effects and demonstrates its application to OML, providing guidelines for unbiased instrumentation in research.

Findings

OML has no significant impact on most instrumented applications.

Proper setup can prevent biases introduced by OML.

In some cases, OML improves application performance.

Abstract

In all measurement campaigns, one needs to assert that the instrumentation tools do not significantly impact the system being monitored. This is critical to future claims based on the collected data and is sometimes overseen in experimental studies. We propose a method to evaluate the potential "observer effect" of an instrumentation system, and apply it to the OMF Measurement Library (OML). OML allows the instrumentation of almost any software to collect any type of measurements. As it is increasingly being used in networking research, it is important to characterise possible biases it may introduce in the collected metrics. Thus, we study its effect on multiple types of reports from various applications commonly used in wireless research. To this end, we designed experiments comparing OML-instrumented software with their original flavours. Our analyses of the results from these experiments show that, with an appropriate reporting setup, OML has no significant impact on the instrumented applications, and may even improve some of their performances in specifics cases. We discuss our methodology and the implication of using OML, and provide guidelines on instrumenting off-the-shelf software.