A Methodology for Information Flow Experiments

TL;DR

This paper introduces a systematic methodology for information flow experiments, especially when analysts lack control or complete models, by framing the problem as causal inference and applying experimental science and statistical analysis.

Contribution

It formalizes limited information flow analysis, connects it to causal inference, and develops a general experimental methodology for detecting data usage by websites.

Findings

Established a causal inference framework for information flow detection

Provided practical guidelines for conducting experiments in this domain

Analyzed real-world website data to demonstrate the methodology

Abstract

Information flow analysis has largely ignored the setting where the analyst has neither control over nor a complete model of the analyzed system. We formalize such limited information flow analyses and study an instance of it: detecting the usage of data by websites. We prove that these problems are ones of causal inference. Leveraging this connection, we push beyond traditional information flow analysis to provide a systematic methodology based on experimental science and statistical analysis. Our methodology allows us to systematize prior works in the area viewing them as instances of a general approach. Our systematic study leads to practical advice for improving work on detecting data usage, a previously unformalized area. We illustrate these concepts with a series of experiments collecting data on the use of information by websites, which we statistically analyze.