Application of the hierarchical bootstrap to multi-level data in neuroscience

TL;DR

This paper advocates for using the hierarchical bootstrap in neuroscience to accurately analyze multi-level, non-independent data, demonstrating its advantages over traditional methods through simulations and real-world examples.

Contribution

It introduces the hierarchical bootstrap as a practical tool for neuroscience data analysis, showing its effectiveness in controlling false positives and maintaining statistical power.

Findings

Traditional tests can yield over 45% false positives in hierarchical data.

Summarizing data reduces power and may not control false positives.

Hierarchical bootstrap maintains error rates and improves power.

Abstract

A common feature in many neuroscience datasets is the presence of hierarchical data structures, most commonly recording the activity of multiple neurons in multiple animals across multiple trials. Accordingly, the measurements constituting the dataset are not independent, even though the traditional statistical analyses often applied in such cases (e.g., Students t-test) treat them as such. The hierarchical bootstrap has been shown to be an effective tool to accurately analyze such data and while it has been used extensively in the statistical literature, its use is not widespread in neuroscience - despite the ubiquity of hierarchical datasets. In this paper, we illustrate the intuitiveness and utility of this approach to analyze hierarchically nested datasets. We use simulated neural data to show that traditional statistical tests can result in a false positive rate of over 45%, even if the Type-I error rate is set at 5%. While summarizing data across non-independent points (or lower levels) can potentially fix this problem, this approach greatly reduces the statistical power of the analysis. The hierarchical bootstrap, when applied sequentially over the levels of the hierarchical structure, keeps the Type-I error rate within the intended bound and retains more statistical power than summarizing methods. We conclude by demonstrating the effectiveness of the method in two real-world examples, first analyzing singing data in male Bengalese finches (Lonchura striata var. domestica) and second quantifying changes in behavior under optogenetic control in flies (Drosophila melanogaster).