Outliers (typically) cannot cause type I errors in one-sample/paired t-tests

TL;DR

Outliers rarely cause false positives in t-tests unless specific conditions are met, such as small effect sizes and large sample sizes.

Contribution

The study derives mathematical bounds and empirical evidence showing when outliers can increase t-statistics.

Findings

Outliers can increase t-statistics only under narrow conditions like concordant direction and small effect sizes.

Monte-Carlo simulations and data set surveys support the derived mathematical bounds.

The risk of type I errors from outliers is low in small sample sizes.

Abstract

The presence of outlying data points can have a significant impact on statistical modeling and significance testing. In the specific context of one-sample t-tests, prior studies have shown (primarily through simulations) that outliers make it more likely for t-tests to fail to reject the null hypothesis. In this study, we investigate the opposite scenario: when an outlier can cause the rejection of the null hypothesis. While it may seem intuitive that outliers aligned with the direction of an effect strengthen that effect, prior studies have shown that this is not always the case. Towards this end, we introduce mathematical bounds on how large outliers can be while still increasing the t-statistic in a given sample. These bounds are validated and supported using Monte-Carlo simulations and a survey of available data sets. From these results, we find that although it is not impossible for outliers to cause significant results in paired or one-sample t-tests, it can only occur under rather narrow circumstances. Specifically, it requires a concordant outlier, a minimal sample size of (n≥15), and a sufficiently small effect size (μ^/σ^≤1/2). Based on these findings, we argue that the risk of isolated outliers causing type I errors is low in many practical situations, especially when sample sizes are small.

Figures

Curves displaying various measures of the t-statistic relative to the placement of the added sample xn+1 which is characterized by Δ (the number of standard deviations it falls above the mean).The left plot displays the modified t-statistic (t′) and the right plot plot displays the empirical influence function (EIF) of the t-statistic (which is a scaled measure of the difference between the modified and original t-statistic). Where appropriate the derived bounds Δ1* and Δ2* are displayed to show how they intersect with these curves.

Simulation results for each sample size n∈{10,25,100}, where each dot represents one iteration of the Monte-Carlo simulation.Each plot is displayed as outlier magnitude (Δ) vs. sample mean (μ^), with lines depicting the two bounds (Δ1*,Δ2*). Points colored by either case (top row), which indicates how the new observation affects the t-statistic for that (Δ,μ^) pair, or p-values (bottom row), which indicate whether the p-value falls below the 0.05 threshold both with and without the new observation. Note that cases in the top row are perfectly separated by the Δ1* and Δ2* lines supporting the validity of the proposed bounds.

Outlier magnitude that maximally increases the t-statistic (Δ2*) at the significance threshold (p < 0.05) as a function of sample size (n).Sample sizes where this value crosses notable 2-σ and 3-σ outlier thresholds are highlighted to indicate the approximate minimum sample size under which outliers can cause type I errors under each definition.

Scatter plot illustrating the relationship between t and t′ (left) and Δ and Δ2* (right) across the datasets examined in Experiment 3.Each dot represents one dataset and dots are color coded based on the p-value with and without the maximum concordant observation.

Equations

Full text

1 Introduction

The challenge of dealing with outliers is a pervasive methodological challenge which spans a range of scientific disciplines [1–5]. Although somewhat nebulous, the term outlier broadly refers to a sample of data that markedly differs from other members of its group. Despite often cast as bad data, outliers are not necessarily the result of experimental error and instead can simply be “an extreme manifestation of the random variability inherent in the data” [6, p. 1]. Regardless of the cause, outliers can disproportionately influence statistical estimates and bias statistical tests, and thus should be handled thoughtfully [7,8]. The best methods for handling outliers are a topic of significant debate across scientific disciplines. A common approach is to simply remove extreme values [9,10], though the criteria for identifying outliers can vary depending on the nature of the data. For a standard homogeneous sample, simple rules such as removing observations more than two or three standard deviations away from the mean can be employed [11]. However, in data sampled from different populations (or experimental conditions), more complicated questions arise as to whether the identification of outliers should be performed within each condition or across the entire dataset. For example, while [12] points out several issues with identifying outliers separately within conditions, [13] shows that identifying outliers across the entire dataset can also introduce problems. Note that all of these considerations become significantly more complicated in settings where data is multivariate [14,15] or the product of complex dynamic processes [16]. Because of the challenges associated with identifying outliers, and the risk they pose to many common tests, a range of statistical methods have been proposed with the aim of being robust to their presence [17–22]. The aim of this work is not to determine the best method of handling outliers, but to examine a very common statistical test (the paired/1-sample t-test) which is known to be somewhat sensitive to outliers, and to form a better understanding of the effect they can have on it.

In the standard null hypothesis significance testing framework, there are two types of errors that can occur. Type I errors refer to rejecting the null hypothesis when it is true (also known as false positives or false discoveries), and type II errors refer to failing to reject the null hypothesis when it is false (also called false negatives). A number of studies have examined the effects of outliers on common statistical tests and consistently found that their presence increases the risk of type II errors, whereas excluding outliers tends to increase the rate of type I errors [12,13,23,24]. In the case of the basic one-sample or paired Student’s t-test, the reason for this is quite clear: the presence of observations that deviate dramatically from the mean increases the sample standard deviation. Since the test statistic is calculated as the ratio of the mean to its associated standard error, an increase in variability inflates the denominator, thereby reducing the t-value and making it less likely that the test will reject the null hypothesis. Nevertheless, this seemingly obvious reasoning bears further scrutiny in certain contexts.

Imagine a study in which ten participants are each tested under two separate conditions, A and B. Suppose the first nine participants all perform slightly better in condition B than in condition A. If the tenth participant were found to perform dramatically better in condition B than condition A, it would be natural to assume that this new data point would only strengthen our confidence in the superiority of condition B over condition A. However, in practice, a paradox arises: if this tenth observation deviates too much from the rest, a paired t-test may be less likely to reject the null hypothesis (and suggest condition B is superior) with the outlier than without it. This counterintuitive effect—that concordant outliers can reduce the likelihood of rejecting the null hypothesis, even when they appear to support the main effect—has already been documented [25]. In a paired t-test, the test statistic is calculated using the mean of the within-subject difference scores, divided by the standard deviation of those differences. A concordant outlier increases both the difference in means across conditions and the standard deviation of the difference scores, and the resulting impact on the t-value depends on how these two changes interact. In some cases, the inflated variability dominates, shrinking the test statistic and reducing statistical power and in others the increase to the mean difference dominates and the t-statistic increases. However, the precise conditions under which outliers increase or decrease the t-statistic, and the corresponding likelihood of rejecting the null hypothesis, remain poorly understood.

Whereas much of the previous literature highlights the risk of outliers inflating type-II errors in t-tests, this work focuses on the risk they pose for type-I errors (or false positives). Specifically, we aim to develop a clearer understanding of the exact conditions under which outlying values can cause a paired t-test to reject the null hypothesis. For our purposes, we define an outlier as causing the rejection of the null if: 1) the null hypothesis is rejected when the outlier is included, and 2) the null hypothesis would not be rejected if the outlier were removed or replaced with any non-outlying value. As the definition for outlier is highly subjective, we explore the introduction of points between 0-10 standard deviations away from the mean and leave it to readers to decide which points meet their definition, although we will highlight some specific implications for 2-σ and 3-σ outlier thresholds. Within this framework, we use both mathematical derivations and empirical simulations to illustrate how different magnitudes of “outlier” can influence the t-statistic and corresponding statistical inferences. Finally, although the framework used in this paper is defined in terms of a generic one-sample t-test, we can easily think of these samples as differences and generalize these findings to paired t-tests as well. Thus, we will largely use the terms “one-sample” and “paired” t-test interchangeably.

The organization of this paper is laid out as follows. Sect 2 introduces the mathematical framework for this paper and presents two upper bounds on how large an outlier can be while still increasing the t-statistic. Sect 3 then presents three experiments designed to validate these bounds and develop a more concrete understanding of the scenarios under which concordant outliers can and cannot cause type I errors. The first two experiments use simulations to assess the effects of concordant outliers under different sample sizes and effect sizes. The third experiment looks at a set of fifty real-world datasets to see how frequently concordant outliers are present and whether or not they are driving any of the significant results in these data. Finally, Sect 4 presents a discussion of our findings to synthesize the main takeaways and relate these findings to the existing literature on this topic.

2 Framework and derivations

The methods for this paper will be laid out as follows. We will first introduce the basic framework and notation that will be used in this paper to describe a scenario in which an arbitrary set of data is influenced by the introduction of a new observation that is greater than the mean of the existing data. In general, we observe that when the new observation is only slightly greater than the mean its introduction will generally increase the t-statistic. However, when the new observation is several standard deviations above the mean it will often decrease the t-statistic. Thus, the second part of our methods derives closed form expressions for how many standard deviations above the mean the new observation should be in order to 1) increase the test statistic and 2) maximally increase the test statistic.

2.1 Problem description

Let us start by defining a set of data $[eqn]$ with sample mean and variance:

$$[eqn]$$

Now we extend the dataset with a new observation $[eqn]$ , where

$$[eqn]$$

For our analysis, the new observation xn + 1 serves the role of a concordant outlying data point that is added to an existing sample. Note that we will consider a wide range of Δ values in our analysis and describing xn + 1 as an outlier when Δ is small might be misleading. To avoid drawing an arbitrary line on what is and is not considered an outlier in our analysis, we keep this definition overly broad and leave the task of disambiguating which cases accurately describe outliers to our discussion.

The fundamental question we seek to address in this paper is whether or not it is possible for the introduction of an outlying point into the data to result in false discoveries for 1-sample student t-tests. More specifically, we ask the question: under what conditions of sample size (n), sample mean $[eqn]$ , and perturbation magnitude Δ, does the introduction of observation xn + 1 increase the t-statistic. Without loss of generality, we assume that:

$[eqn]$

$[eqn]$

While the first condition appears restrictive it merely assumes that the data has been scaled to unit variance prior to conducting the analysis. For example, suppose we have sample $[eqn]$ with mean $[eqn]$ and variance $[eqn]$ . We could then define $[eqn]$ , and our analysis would then hold for the scaled data. Importantly, this scaling has no effect on the t-statistic since it affects the numerator and denominator equally. This condition also makes interpretation of our results significantly easier since parameters like $[eqn]$ and Δ are now measured as number of standard deviations, rather than along the context-specific scale of the original data. So, if our criteria for outliers were defined as any points more than three standard deviations away from the mean, that would reflect cases where $[eqn]$ . Additionally, because $[eqn]$ we can think of the sample mean as representing the effect size in our data (since $[eqn]$ ), and will frequently refer to it as such going forward.

Regarding the second condition, since our results will be based on the relative magnitude of the t-statistic for the modified data, flipping the sign of both $[eqn]$ and Δ would not affect our findings (since the t-statistic depends on the magnitude of the effect, not its direction). Thus, this condition only limits generality in the sense that it requires the added outlying value to be in the same direction as the observed effect. This omission represents a rather uninteresting case, since it is generally assumed that outliers in the opposite direction of the observed effect will not result in false discoveries.

Using (1), we can derive the sample mean and variance of the modified data as (full derivation in S1 File):

$$[eqn]$$

and from this the t-statistic of the original and modified data are

$$[eqn]$$

and

$$[eqn]$$

One interesting observation we can make from these equations is that the both t and $[eqn]$ are dependent on only three parameters: the sample mean of the original data ( $[eqn]$ ), the sample standard deviation of the original data ( $[eqn]$ ), and the number of standard deviations the new sample is placed above the mean (Δ). This means that the effect of an outlier (or any new observation) on the t-statistic will be the same for any two datasets of equal mean and variance regardless of how they are distributed. This is an important point which will (in the subsequent analysis) allow us to make highly general claims about how outliers affect the t-statistic without relying on distributional assumptions.

2.2 Deriving bounds on outlier magnitude

Here we introduce theoretical limits on Δ that provide guarantees on how the new observation will affect the results of a one-sample t-test. The first observation we would like to highlight is that as the value of the new observation gets arbitrarily large ( $[eqn]$ ) the test statistic goes to one ( $[eqn]$ ). For a two-tailed t-test, this yields a p-value of $[eqn]$ . Thus, given a sufficiently large outlier, a t-test will always yield a null result regardless of the strength of evidence in the original (outlier-free) data. Note that these asymptotic characteristics have already been shown in [25]. The second observation is that there is an upper bound on Δ (which we will call $[eqn]$ ), defined in terms of only $[eqn]$ and n, beyond which the introduction of xn + 1 will reduce the t-statistic of the original data. This upper bound is expressed formally in Theorem 2.2 below.

Theorem 1. If $[eqn]$ and $[eqn]$ where

$$[eqn]$$

then $[eqn]$ .

The proof for Theorem 2.2 is provided in S2 File. The third and final observation is that there is a second upper bound on Δ (which we will call $[eqn]$ ) defined in terms of only $[eqn]$ and n, beyond which perturbing xn + 1 any further from the mean will reduce $[eqn]$ . This upper bound is expressed formally in Theorem 2.2 below.

Theorem 2. If $[eqn]$ where

$$[eqn]$$

then $[eqn]$ .

The proof for Theorem 2.2 is provided in S3 File. It is also fairly straightforward to show that $[eqn]$ under the predefined conditions (also included in S3 File). These two theorems provide general insights into the maximum values a new outlier can take while still exerting positive effects on the t-statistic. Relating this back to our theoretical framework Δ describes the number of standard deviations above the mean where the new observation (xn + 1) is located. The value of $[eqn]$ depicts the exact location where the new observation will maximally increase the t-statistic. Thus, if we imagine we can control the placement of this new observation, the further we move it away from that location the less it will increase the t-statistic. And if we move it far enough away from this point in the positive direction, $[eqn]$ describes the point at which its inclusion begins to lower the t-statistic (relative to its absence). Taken together with the asymptotic characteristics of Δ, this will eventually lead to failure to reject the null hypothesis. Based on the provided definitions for when an outlier causes rejection of the null hypothesis, $[eqn]$ provides a relatively clear indication of when this can occur. For any sample size (n) and sample mean ( $[eqn]$ ) we consider, if the value of $[eqn]$ would not meet our definition of outlier then no outlier can cause the rejection of the null, since no outlying observation can increase the t-statistic more than $[eqn]$  +  $[eqn]$ . For example, if n = 10 and $[eqn]$ , then $[eqn]$ , and any new observations added to the data cannot increase the t-statistic more than one exactly 0.9 standard deviations above the mean. Thus, where the addition of some outlying points might increase the t-statistic in this scenario its influence would necessarily be less than some alternative non-outlying sample.

To better understand what these results tell us about the relationship between the placement of the new data point xn + 1 and $[eqn]$ , Fig 1 displays the curves of (a) the new t-statistic ( $[eqn]$ ) and (b) the empirical influence function for the t-statistic across Δ for four different sample means $[eqn]$ . These plots also display where $[eqn]$ and $[eqn]$ fall on each of these curves. Going in order, the first plot shows t′, which starts at different values for the four means, but in each case ascends to a particular peak then begins to decline. The dotted purple line represents the $[eqn]$ pairs for a continuous range of $[eqn]$ values. As expected, this line intersects with the $[eqn]$ curves at the peaks of each $[eqn]$ -curve and these intersections occur at higher Δ values for lower $[eqn]$ curves. The second plot displays the empirical influence function (EIF) of the t-statistic, which we define as:

$$[eqn]$$

Curves displaying various measures of the t-statistic relative to the placement of the added sample xn+1 which is characterized by Δ (the number of standard deviations it falls above the mean).The left plot displays the modified t-statistic (t′) and the right plot plot displays the empirical influence function (EIF) of the t-statistic (which is a scaled measure of the difference between the modified and original t-statistic). Where appropriate the derived bounds Δ1 and Δ2* are displayed to show how they intersect with these curves.*

This plot displays how much the t-statistic changes as a result of the newly added sample. Note that while the EIF is more typically defined in terms of the location of the added sample, we define it terms of Δ (the location of the added sample relative to the mean) for consistency with other visualizations. Similar to the first plot, the purple line indicates the peak in the EIF for each curve, which occurs at the same Δ values as before. For this plot we also see a line for $[eqn]$ at $[eqn]$ , indicating the transition point between positive and negative EIF values, or alternatively the point at which the new t-statistic ( $[eqn]$ ) falls below the original t-statistic (t).

3 Experiments and results

In this section, we introduce three experiments to validate and support the previous derivations. The first experiment will use a Monte-Carlo simulation to empirically verify the accuracy of the bounds introduced in Sect 2, and to quantify what values for sample means ( $[eqn]$ ) and outlier magnitudes (Δ) result in changes to t-test results for three different sample sizes ( $[eqn]$ ). The second experiment mirrors the first, but considers all sample sizes between n = 2 and n = 100 to determine the maximum value of $[eqn]$ that occurs in cases where the significance result becomes significant after the introduction of the new data point. As $[eqn]$ describes the location of the new observation that maximally increases the t-statistic, this value provides an approximate upper bound on the outlier criteria under which an outlier can possibly cause a type I error at a given sample. Finally, experiment 3 applies this methodological framework to a survey of paired datasets to examine whether any cases of outliers causing type I errors can be identified.

3.1 Experiment 1: Comparison of bounds

To validate the previously described results we conduct a Monte-Carlo simulation. The Monte-Carlo simulation begins by randomly generating the mean of the original data and the magnitude of the outlier according to the following uniform distributions: $[eqn]$ , $[eqn]$ . From this, a raw version of the original dataset $[eqn]$ is generated by sampling n values from a Normal distribution with $[eqn]$ and $[eqn]$ :

$$[eqn]$$

This raw data is then normed to create

$$[eqn]$$

where $[eqn]$ is the sample standard deviation of the raw data. This ensures that $[eqn]$ as previously assumed. Note that while the data in this simulation is generated from a normal distribution, normality of the data is not required for our analysis to hold. Since t and $[eqn]$ can be expressed solely in terms of $[eqn]$ , n, and Δ the process generating the data has no effect on the results beyond its relation to these parameters. Thus, this experiment could be repeated using any non-normal distribution (with the same sample mean and variance) to achieve the same results. Alternative versions of this experiment demonstrating near identical results for non-normal cases are displayed in S4 File. From here, the modified dataset is defined just as in our model: $[eqn]$ where $[eqn]$  +  Δ. As we are interested in not only how the introduction of xn + 1 affects statistical testing, but also how further perturbation of xn + 1 would influence statistical outcomes, we also generate a third dataset

$$[eqn]$$

where $[eqn]$ is a very small perturbation constant meant to simulate calculation of a local derivative. Comparing $[eqn]$ with $[eqn]$ provides a simple way of determining whether pushing xn + 1 further from the mean will increase or decrease the t-statistic. For each of the three datasets (x, $[eqn]$ , and $[eqn]$ ), we calculate the t-statistics and associated p-values which we will call t, $[eqn]$ , $[eqn]$ and p, $[eqn]$ , $[eqn]$ respectively.

From here, within each run of the Monte-Carlo simulation, we will investigate several cases:

Case 0 ( $[eqn]$ and $[eqn]$ ) : Adding xn + 1 reduced the t-statisticCase 1 ( $[eqn]$ and $[eqn]$ ) : Adding xn + 1 increased the t-statistic, but less than if xn + 1 were closer to the meanCase 2 ( $[eqn]$ and $[eqn]$ ) : Adding xn + 1 increased the t-statistic more than any observation closer to the mean would have

If the results in Theorems 1 and 2 are valid and their conditions are met ( $[eqn]$ ), it should be the case that any observations with $[eqn]$ fall into Case 0, any observations with $[eqn]$ fall into Case 1, and any observations with $[eqn]$ fall into Case 2. The first goal of the simulation is to validate whether the theoretical bounds correctly categorize the empirical t-statistics. The second objective is to examine cases where either the original or modified dataset meets the standard statistical significance threshold $[eqn]$ .

Based on this setup, we run 10,000 iteration Monte-Carlo simulation across three different sample sizes ( $[eqn]$ ). Fig 2 displays the results of these simulations via a scatter plot of Δ vs. $[eqn]$ with lines for the two bounds $[eqn]$ and $[eqn]$ . Plots in the top row display individual iterations of the simulation color coded by the previously described cases. These plots clearly illustrate that the bounds accurately delineate these cases for all three sample sizes. As would be expected from the formula, $[eqn]$ is not noticeably affected by the sample size, while the $[eqn]$ line generally shifts leftwards as n gets larger. Plots in the bottom row of Fig 2 display the same results, but this time color coded based on whether p and $[eqn]$ are greater than or less than the significance threshold 0.05. Since the value of p is determined entirely by the values of $[eqn]$ and n (because $[eqn]$ is fixed), there is a clear separation between points for which $[eqn]$ and $[eqn]$ determined by a fixed value of $[eqn]$ for each n. We also see that this $[eqn]$ -threshold moves left as the sample size increases, and with this leftward shift we see increases in both $[eqn]$ and $[eqn]$ . Therefore, for smaller sample sizes there is a lower upper limit on the magnitude of outliers capable of producing false discoveries. In the n = 10 case, for example, we observe Δ values as high as 4.39 yielding a significant result in the modified data, despite no significance in the original data. However, it is important to point out that in these instances, the added data point could have been much smaller and still produced a significant p-value. If we examine the maximum value of $[eqn]$ in those trials, we see that it never exceeds 1.58. Since these experiments are conducted on normalized data, this means that a new observation need not be more than 1.58 standard deviations above the mean to maximize the t-statistic (and thus minimize the corresponding p-value). In the larger samples simulations n = 25 and n = 100 the maximum observed values for $[eqn]$ in trials where $[eqn]$ and $[eqn]$ was 2.77 and 5.83, respectively. Therefore, in larger sample sizes it is possible for outlying samples to meet our definition for causing the rejection of the null hypothesis, although these cases represent a relatively narrow proportion of the observed trials. The following subsection will further explore the relationship between sample size and the maximally influential outlier value $[eqn]$ at the significance threshold.

Simulation results for each sample size n∈{10,25,100}, where each dot represents one iteration of the Monte-Carlo simulation.Each plot is displayed as outlier magnitude (Δ) vs. sample mean (μ^), with lines depicting the two bounds (Δ1,Δ2*). Points colored by either case (top row), which indicates how the new observation affects the t-statistic for that (Δ,μ^) pair, or p-values (bottom row), which indicate whether the p-value falls below the 0.05 threshold both with and without the new observation. Note that cases in the top row are perfectly separated by the Δ1* and Δ2* lines supporting the validity of the proposed bounds.*

3.2 Experiment 2: Maximum outlier magnitude vs. sample size

In the previous simulation we observed that the values of Δ likely to change the outcome of the significance test increased for larger sample sizes. This was partially a result of the dependency of $[eqn]$ and $[eqn]$ on n, but primarily stems from the inverse relationship between $[eqn]$ , $[eqn]$ and $[eqn]$ combined with the fact that the effect size ( $[eqn]$ ) needed for statistical significance decreases for larger n. Our goal in this simulation is to develop a better understanding for the maximum value an outlier can take while still causing a statistically significant result in the modified sample. Remember, per our definitions, an outlier causing a significant result requires that 1) its inclusion leads to a significant result and 2) replacing it with any non-outlying value would not lead to a significant result. Per these definitions, it is only possible for outliers to cause false discoveries when $[eqn]$ meets our outlier criteria as otherwise any outlying sample that yields a significant result could be replaced with a non-outlying sample at $[eqn]$ to achieve the same result. Thus the goal of this simulation is to estimate the maximum value of $[eqn]$ in cases where $[eqn]$ and $[eqn]$ for a given sample size. To achieve this, we repeat the same 10,000 iteration Monte-Carlo simulation from before at every sample size $[eqn]$ and for each sample size measure the maximum value of $[eqn]$ corresponding to trials where $[eqn]$ and $[eqn]$ . The results of this simulation are presented in Fig 3.

Outlier magnitude that maximally increases the t-statistic (Δ2) at the significance threshold (p < 0.05) as a function of sample size (n).Sample sizes where this value crosses notable 2-σ and 3-σ outlier thresholds are highlighted to indicate the approximate minimum sample size under which outliers can cause type I errors under each definition.*

These results offer additional insights into the potential for concordant outliers to cause false discoveries across various sample sizes. If we define outliers as any point that is more than two standard deviations away from the mean, empirical results suggest that it is only possible for outliers to cause rejection of the null hypothesis at sample sizes greater than fourteen. If we impose as stricter definition and require outliers be more than three standard deviations from the mean, the minimum sample size increases to $[eqn]$ before concordant outliers can cause the null hypothesis to be rejected. Also note that these sample size requirements are necessary for concordant outliers to cause false positives, but not sufficient by themselves. Using the n = 100 simulation in Fig 2 as an example, we see that the cases where $[eqn]$ and $[eqn]$ still only occur for a very narrow range of $[eqn]$ -values.

3.3 Experiment 3: Examination of outlier effects in real paired data

In the final experiment, we will look at some real-world datasets containing paired observations to see how frequently concordant outliers occur in these datasets and how they influence paired t-test results. This experiment also provides the opportunity to illustrate how this approach can be applied to real world datasets to characterize the influence of existing data points. To attain a set of different datasets to examine which contain paired observations, we draw from two existing R packages that contain built in repositories of paired datasets, the PairedData package [26], and the BSDA package [27]. Together these libraries provide a total of 43 datasets. As some of these datasets contain more than 1 set of paired observations we then have 50 sets of paired samples for this analysis.

For each dataset, we go through the following procedure. Let us call the set of paired observations loaded from the data $[eqn]$ and $[eqn]$ . We can calculate the original difference pairs as $[eqn]$ − $[eqn]$ . If this doesn’t result in $[eqn]$ , then instead assign $[eqn]$ − $[eqn]$ . Now, we identify a candidate concordant outlier as the maximum value in these difference pairs and remove it from the sample.

$$[eqn]$$

where $[eqn]$ . The original data matching our framework (x) is then calculated as the rescaled version of this data

$$[eqn]$$

and the modified data ( $[eqn]$ ) is created by re-adding the scaled version of the candidate outlier back into this set

$$[eqn]$$

Using this data, we calculate the t-statistics both with and without the candidate outlier (t and $[eqn]$ respectively), their associated p-values (p and $[eqn]$ ). We can also calculate

$$[eqn]$$

and

$$[eqn]$$

After going through this process, four of the 50 samples showed negative sample means after the candidate outlier was removed and were excluded from subsequent analysis. The results of this analysis are displayed comprehensively in Table 1. The primary goal of this analysis is to determine how frequently concordant outliers are present in these data and whether instances of them causing rejection of the null hypotheses can be identified. Of the 46 paired samples, 23 contained a concordant outlier more than 2 standard deviations above the mean. Of these 23 cases there was only one dataset (the Vocab data) where the results of the test were altered by the concordant outlier. Although this dataset clearly meets the outlier criteria ( $[eqn]$ ) and the outlier affects the results of the test, this still does not qualify as causing the rejection since $[eqn]$ in this instance. Now, if we instead examine cases where the largest concordant observation did not meet the outlier threshold, we see that of those 23 cases in which the maximum observation (x^*^) fell below the 2-σ threshold eight datasets showed a significant result when the candidate outlier was included. Consequently in these datasets it is far more common for a non-outlying observation to yield a rejection that didn’t exist before than for and outlying one.

Table 1: Summary of results for each of the datasets analyzed in Experiment 3. Note that (in line with the described experiment) all statistics (except T′ and p′) are calculated after the maximum concordant observation has been removed. Rows in which the p-value falls below the 0.05 threshold when the maximum concordant observation is introduced (p > 0.05 & p′<0.05) are highlighted in gray.Note that as Theorem 2.2 requires that μ^>1/n, Δ1 is not reported for datasets where this assumption is violated.*

Visualizations of these results are displayed in Fig 4 where each dot represents the results for a single dataset. The first scatter plot displays the updated t-statistic with the maximum concordant observation relative to the t-statistic without it. Although this plot shows the t-statistic increases in the vast majority of cases (37 of 46 according to Table 1) we know from our previous discussion that the majority of these cases the concordant observation increasing the t-statistic does not meet the 2-σ outlier threshold. In contrast, all nine of the cases where the t-statistic decreases the concordant observation exceeds this threshold. Note that by definition cases where the concordant observation decreases the t-statistic are cases where $[eqn]$ . Looking at these nine cases in Table 1 we see that this tends to occur for datasets with larger effect sizes ( $[eqn]$ ). This makes sense as $[eqn]$ is inversely related to $[eqn]$ . Interestingly, for three of these datasets (Blink, Grain2, and Oxytocin) the $[eqn]$ is less than two, meaning that new observations in these samples don’t have to be that much greater than the mean in order to negatively affect the t-statistic. The second set of plots displays the point where the concordant observation will maximally increase the t-statistic ( $[eqn]$ ) relative to the maximum observation (Δ). There are a few interesting observations we can make from this plot. First, in the majority of datasets $[eqn]$ , meaning that for these datasets no outlying point can increase the t-statistic more than a point within two standard deviations of the mean. We also observe that in the majority of these cases $[eqn]$ , which indicates that the test statistics in these data would be greater if the maximum observation were actually smaller. Thus while some of these data show statistically significant test results and contain concordant outliers, the strength of the t-statistic generally comes in spite of the magnitude of these outliers not as a product of it. In contrast, if we look at the ten cases where $[eqn]$ , we see that none of these data show statistically significant test results either with or without the concordant observation present. This is not surprising since (by definition) $[eqn]$ is only large when $[eqn]$ is small. So, in summary, of the 50 datasets analyzed in this experiment, we could not find a single case where a statistically significant result could be attributed to an outlying sample by the definitions proposed in this paper.

Scatter plot illustrating the relationship between t and t′ (left) and Δ and Δ2 (right) across the datasets examined in Experiment 3.Each dot represents one dataset and dots are color coded based on the p-value with and without the maximum concordant observation.*

3.4 Code availability

Code for the subsequent simulations can be found on Open Science Framework at https://osf.io/yfju9/. Simulations were conducted in R version 4.4.0 [28] and used dplyr for data manipulation [29], ggplot and patchwork for visualization[30,31], and BSDA and PairedData for data examples [26,27].

4 Discussion

The question we sought to investigate in this manuscript is whether it is possible for outliers, specifically concordant ones to cause false discoveries in one-sample or paired t-tests. While it is undoubtedly true under the definitions provided that outliers can lead to false discoveries, the conditions under which this occurs are surprisingly limited. Because the number of standard deviations above the mean that an outlying point maximally increases the t-statistic is shown to be inversely proportional to the effect size, outliers can only increase the t-statistic more than non-outlying values when the effect size is small-to-moderate (approximately $[eqn]$ for a $[eqn]$ outlier threshold). As smaller sample sizes require larger effect sizes to achieve statistical significance and the ability of single observations to increase the t-statistic is limited, we can generally rule out the possibility of outliers causing type I errors in small sample sizes ( $[eqn]$ for a $[eqn]$ outlier threshold). When considering smaller effect sizes, as the initial effect size approaches zero, the number of standard deviations away from the mean at which a point maximally increases the t-statistic approaches infinity. However, in practice, the extent to which such points can increase the t-statistic is limited and frequently insufficient to result in rejection of the null hypothesis. Using a 2-σ outlier threshold, the highest observed p-value for which an outlier caused a significant result (i.e. both $[eqn]$ and $[eqn]$ ) was p = 0.096. Therefore, even in cases where outliers meet our criteria for causing a rejection of the null hypothesis, some initial indication of an effect must already be present in the original data.

To summarize, the following criteria must all be met for an outlier to cause a false discovery. First, the observed effect in the original data must be on the border of statistical significance. Based on our empirical findings, this alone probably occurs less than 5% of the time when the null hypothesis is true. Second, the dataset must contain a sufficient number of samples (n > 15) in order for effects near the significance threshold to be small enough to be positively affected by outliers. Finally, if the previous two criteria are met, the outlier still needs to be both: 1) concordant with the observed effect, and 2) not so great in magnitude that it starts to reduce the t-statistic. The narrowness of these conditions explains why the analysis of sample datasets did not find a single instance in which a significant effect could be attributed to an outlying sample, and justifies the titular claim that outliers typically cannot cause type I errors. These conditions also contrast starkly with the relatively broad conditions necessary for outliers to cause type II errors, which require only: 1) the original data exhibits a significant effect, and 2) the outlier is sufficiently large (in either direction) to nullify that effect. One part of this result that feels counter intuitive is the conclusion that outliers pose greater risk of causing false discoveries in large samples than small samples. Since a single corrupt observation constitutes a greater proportion of a small sample than a large one, it is natural to assume it would carry greater impact. This idea does not conflict with our findings. Examining Fig 2 shows that the number of cases where the p-value falls below the significance threshold as a result of the new observation (red dots) appears to at least as high in the smaller sample sizes ( $[eqn]$ ). However, in the smaller sample sizes these cases tend to occur more frequently for smaller Δ values. Thus it is not the case that isolated observations cannot inflate the t-statistic in smaller samples, just that the ones doing so tend not to be far enough from the mean to be generally considered outliers.

Relating these findings to prior research, this is not the first study to show that outliers pose a greater risk for type II errors than type I errors in one-sample or paired t-tests [12,13,23,24]. However, few studies have focused specifically on the effects of concordant outliers. Our work builds naturally on the work of [25], who showed through simulation that increasing the magnitude of a concordant outlier will eventually result in increasing p-values and decreases to the null-hypothesis rejection rate. Building on this work, this study provides closed-form expressions for the point at which: 1) further increases to the outlier will begin to decrease the t-statistic ( $[eqn]$ ), and 2) outliers large enough to to decrease the t-statistic by their introduction ( $[eqn]$ ). In doing this, we have also established a clearer understanding of how the effect of concordant outliers on the t-statistic depends on the relationship between the outlier’s magnitude, the sample size, and the effect size of the original data. To our knowledge, this is the first study to propose that outliers pose no risk of causing false-discoveries in one-sample or paired t-tests with small-sample designs.

This work also aligns well with existing theory within the robust statistics literature, which has made efforts to quantify that sensitivity of the t-statistic (and other measures) to corrupt data. One common measure of robustness that is closely related to the work in this paper is the breakdown point. The breakdown point of an estimator refers to the maximum percentage of data that can be corrupted before an estimator becomes unreliable (or exhibits arbitrarily large error) [32]. In the context of statistical inference the breakdown point can be separated into the level breakdown point (robustness of validity), which measures level of contamination needed to guarantee rejection of the null hypothesis, and the power breakdown point (robustness of efficiency), which measures the level of contamination needed to guarantee failure to reject the null hypothesis. Work that has examined these properties for the t-statistic have found it exhibits some robustness of validity, but little robustness of efficiency [17,33,34]. For the breakdown point specifically, the t-statistic has a power breakdown point of zero and a level breakdown point that is generally non-zero and proportional to $[eqn]$ [33].

One important point to note about this analysis is that it makes no assumptions on the distribution from which the data is drawn. This means that this analysis, and the associated conclusions, are applicable regardless of whether the data is normally distributed or if violations of other standard assumptions such as heteroskedasticity or non-independence are present. As a result the conclusions we draw about whether or not an outlier is capable of causing an outlier for a given sample size, outlier magnitude, and effect size are not dependent on any of these distributional assumptions. So, for example, the claim that outliers cannot cause significant t-test results when n < 15 is true even when any or all of these assumptions are violated. Violation of these assumptions may yield inflated type I or type II error rates in the statistical test, but it does not affect our conclusions about the ability of isolated outlying values to disproportionately impact a t-test towards a significant result. Note that the lack of distributional assumptions made in our analysis is a double edged sword. While it means that the conclusions in this paper are not reliant on properties of the underlying distribution, it also means that we cannot make claims about type I and type II error rates.

It is important to emphasize that these findings in no way diminish the concern that outliers, or more broadly corrupt data and the violation of distributional assumptions, may pose in one-sample or paired t-tests. While we make the case that isolated outliers cannot cause type I errors in small samples, an alternative interpretation of these results is that the erroneous data most likely to cause type I errors cannot be detected based on its deviation from the mean. Rather than diminishing the concern outliers present to 1-sample t-tests, this work reinforces the well established understanding that the primary concern outliers present in this context is diminished efficiency and reduced statistical power. Thus, in cases of known contamination or non-normality alternative robust or non-parametric methods may still be appropriate. For this, there are a range of robust methods that attempt to maintain efficiency close to that of parametric methods while reducing sensitivity to violations of model assumptions. These include strategies based on trimmed means [18], bootstrapping methods [19,20], and M-estimators [18,21,22]. Non-parametric methods can also provide a compelling option in some cases, but are not advisable for every application [35–37]. This work is not meant to discourage the use of such alternatives, but to facilitate proper interpretation of the t-test in cases where it is used with outliers present.

Perhaps the most restrictive limitation imposed by our analysis is that only isolated outliers are considered. Some study of the effects of multiple concordant outliers can be found in the analysis presented in [34], which examines the effects of asymmetric contamination and finds that large contaminator rates (15%) can lead to significantly inflated type I error rates. To better understand the effects of multiple outliers, this framework could be used to test the iterative addition of points at $[eqn]$ and assess what the minimal number of optimally placed corrupt observations is necessary to produce a significant result in different scenarios. However, considering the joint impact of multiple concordant outliers remains a topic for future work. One final point to note regarding the application of these ideas to other settings is that the framework used here is an inversion of how outliers are usually characterized. Very often outliers are considered as part of the original sample and considered for removal, whereas our framework considers the addition of a possibly outlying data point to a clean original sample. So when we say, for example, that $[eqn]$ is the number of standard deviations above the mean at which a new observation maximally increases the test statistic, the mean and standard deviations values referred to here are both measured without the new observation. As a result, these formulas do not apply exactly to the context of outlier removal, but can easily be implemented in that context by re-measuring these terms sans outlier, as shown in Experiment 3. As a final note, it is worth mentioning that the σ-based outlier criteria used in this paper are chosen for convenience, and are generally not recommended as a method for identifying outliers [11,38].

5 Conclusion

This study investigates how outlying values influence the results of one-sample or paired t-tests, specifically in cases where the outlier is concordant with the direction of the effect observed in the rest of the data. We show that, even in these cases, outliers frequently decrease rather than increase the test statistic. To formalize this, we introduced bounds that provide upper limits on how large an outlier can be before it begins to reduce the test statistic. These bounds show that for small sample sizes (n < 15), adding an outlier more than two standard deviations above the mean cannot produce a significant result that could not already be achieved by a non-outlier. In larger samples, outliers can cause significant results that non-outlying values cannot, but only under relatively constrained circumstances.

Supporting information

S1 Appendix ABasic derivations.(PDF)

S2 Appendix BProof of Theorem 1.(PDF)

S3 Appendix CProof of Theorem 2.(PDF)

S4 Appendix DNon-normal simulations.(DOCX)