Does Learning Require Memorization? A Short Tale about a Long Tail
Vitaly Feldman
TL;DR
This paper provides a theoretical explanation for why memorization is necessary in over-parameterized learning, especially for natural, long-tailed data distributions, linking empirical observations to formal models.
Contribution
It introduces a novel theoretical model showing memorization's necessity for optimal generalization in natural data, supported by empirical evidence and applicable to privacy and model compression.
Findings
Memorization is necessary for near-optimal generalization on natural data.
Long-tailed subpopulation distributions make memorization essential.
Limiting memorization impacts different subgroups unequally.
Abstract
State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning. We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving close-to-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Does Learning Require Memorization?
A Short Tale about a Long Tail
Vitaly Feldman
Google Research, Brain Team Now at Apple. Part of this work was done while the author was visiting the Simons Institute for the Theory of Computing.
Abstract
State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning.
We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving close-to-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and supported by the results of several recent empirical works. In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed. Image and text data is known to be long-tailed and therefore our results establish a formal link between these empirical phenomena. Our results allow to quantify the cost of limiting memorization in learning and explain the disparate effects that privacy and model compression have on different subgroups.
1 Introduction
Understanding the generalization properties of learning systems based on deep neural networks (DNNs) is an area of great practical importance and significant theoretical interest. The models used in deep learning are famously overparameterized, that is, contain many more tunable parameters than available data points. This makes it is easy to find models that “overfit” to the data by effectively memorizing the labels of all the training examples. The standard theoretical approach to understanding of how learning algorithms avoid such overfitting is based on the idea of regularization. Learning algorithms are designed to either explicitly or implicitly balance the level of the model’s complexity (and, more generally, its ability to fit arbitrary data) and the empirical error on the training dataset. Fitting each mislabeled point or an outlier requires increasing the level of model’s complexity and therefore, by tuning this balance, the learning algorithm can find the patterns in the data without overfitting.
A variety of regularization techniques are widely used in practice and have been analyzed theoretically. Yet, the accepted view of regularization contradicts the empirical evidence from most modern image and text classification datasets. Deep learning algorithms tend to produce models that fit the training data very well, typically achieving - accuracy, even when the accuracy on the test dataset is much more modest (often in the - range). Such (near) perfect fitting requires memorization111In this work we will formalize and quantify this notion of memorization. Informally, we say that a learning algorithm memorizes the label of some example in its dataset if the model output on predicts on whereas the model obtained by training on without is unlikely to predict on . of mislabeled data and outliers which are inevitably present in large datasets. Further, it is known that the same learning algorithms achieve training accuracy of over on the large ImageNet dataset [Den+09] that is labeled completely randomly [Zha+17]. It is therefore apparent that these algorithms are not using regularization that is sufficiently strong to prevent memorization of (the labels of) mislabeled examples and outliers.
This captivating disconnect between the classical theory and modern ML practice has attracted significant amount of research and broad interest in recent years (see Sec. 1.3 for an overview). At the same time the phenomenon is far from new. Random forests [Bre01] and Adaboost [FS97] are known to achieve their optimal generalization error on many learning problems while fitting the training data perfectly [Sch+98, Sch13, Wyn+17]. There is also recent evidence that this holds for kernel methods in certain regimes as well [Zha+17, BMM18, LR18].
Understanding this disconnect is also of significant importance in the context of privacy-preserving machine learning. Privacy is a natural concern when the training data contains sensitive information about individuals such as medical records or private communication. The propensity of deep learning algorithms to memorize training data is known to pose privacy risks when the resulting model is deployed [Sho+17]. This leads to the question of whether such memorization is necessary for learning with high accuracy or is merely an artifact of the current learning methods.
1.1 Our contribution
We propose a conceptually simple explanation and supporting theory for why memorization of seemingly useless labels may be necessary to achieve close-to-optimal generalization error. It is based on the view that the primary hurdle to learning an accurate model is not the noise inherent in the labels but rather an insufficient amount of data to predict accurately on rare and atypical instances. Such instances are usually referred in practice as the “long tail” of the data distribution. It has been widely observed that modern datasets used for visual object recognition and text labeling follow the classical long-tailed distributions such as Zipf distribution (or more general power law distributions).
To formalize the notion of having a “long tail” we will model the data distribution of each class (in a multiclass prediction problem) as a mixture of distinct subpopulations. For example, images of birds include numerous different species photographed from different perspectives and under different conditions (such as close-ups, in foliage and in the sky) [VHP17]. Naturally, the subpopulations may have different frequencies (which correspond to mixture coefficients). We model the informal notion of long-tailed data distributions as distributions in which the frequencies of subpopulations are long-tailed. The long-tailed nature of subpopulation frequencies is known in datasets for which additional human annotations are available. A detailed discussion of this phenomenon in the SUN object detection benchmark [Xia+10] can be found in the work of [ZAR14]. In Fig. 1 we include a plot from the work that demonstrates the long tail of the frequency distribution.
Additional evidence that classes can be viewed as long-tailed mixtures of subpopulations comes from extreme multiclass problems. Specifically, these problems often have more than fine-grained labels and the number of examples per class is long-tailed [BS17, WRH17, Kri+17, VHP17, Cui+18, BS19]. Observe that fine-grained labels in such problems correspond to subcategories of coarser classes (for example, different species of birds all correspond to the “bird” label in a coarse classification problem). We also remark that subpopulations do not have to directly correspond to human-definable categories. They are the artifacts of the representation used by the learning algorithm which are often relatively low-level.
It is natural to presume that before seeing the dataset the learning algorithm does not know the frequencies of subpopulations. The second key observation underlying our explanation is that the algorithm may not be able to predict accurately on a subpopulation until at least one example from the subpopulations is observed. Alternatively, the accuracy of the algorithm on a subpopulation is likely to increase noticeably once a representative example from that subpopulation is observed. A dataset of samples from a long-tailed mixture distribution will have some subpopulations from which just a single example was observed (and some subpopulations from which none at all). To predict more accurately on a subpopulation from which only a single example was observed (and to fit the example) the learning algorithm needs to memorize the label of the example. The question is whether this is necessary for achieving close-to-optimal generalization error. The answer depends on the frequency of the subpopulation. If the unique example from a subpopulation (or singleton) comes from an extremely rare (or “outlier”) subpopulation then memorizing it has no significant benefits. At the same time, if the singleton comes from an “atypical” subpopulation with frequency on the order of , then memorizing such an example is likely to improve the accuracy on the entire subpopulation and thereby reduce the generalization error by .
The key point of this work is that based on observing a single sample from a subpopulation, it is impossible to distinguish samples from “atypical” subpopulations from those in the “outlier” ones. Therefore an algorithm can only avoid the risk of missing “atypical” subpopulations by also memorizing the labels of singletons from the “outlier” subpopulations. Importantly, in a long-tailed distribution of frequencies, the total weight of frequencies on the order of is significant enough that ignoring these subpopulations will hurt the generalization error substantially. Thus, for such distributions, an algorithm needs to memorize the labels of outliers in order to achieve close-to-optimal generalization.
The long tail effect also explains why memorizing mislabeled examples can be necessary. As discussed, a learning algorithm may be unable to infer the label of a singleton example accurately based on the rest of the dataset. Thus as long as the observed label is the most likely to be true and the singleton comes from an “atypical” subpopulation, the algorithm needs to memorize the label. In contrast, if the mislabeled example comes from a subpopulation with many other examples in the dataset, the correct label can be inferred from the other labels and thus memorization is not necessary (and can even be harmful). In most datasets used in machine learning benchmarks only relatively atypical examples are mislabeled and the noise rate is low. Thus learning algorithms for such datasets are tuned to memorize the labels quite aggressively.
1.1.1 Overview
On a technical level our primary contribution is turning this intuitive but informal explanation into a formal model that allows to quantify the trade-offs involved. This model also allows to quantify the cost of limiting memorization (for example, via regularization or ensuring differential privacy) when learning from natural data distributions.
We start by explaining why achieving close-to-optimal generalization error requires fitting outliers and (some) mislabeled examples since this is the phenomenon observed in practice. We then formalize the claim that such fitting requires label memorization. Our explanation is based on a simple model for classification problems that incorporates the long tail of frequencies in the data distribution. The goal of the model is to isolate the discussion of the effect of memorization on the accuracy from other aspects of modeling subpopulations. More formally, in our model the domain is unstructured and has size (each point will correspond to a subpopulation in the more general model). In the base model the true labeling function belongs to some class of functions known to the learning algorithm. We will be primarily interested in the setting where is rich (or computationally hard) enough that for a significant fraction of the points the learning algorithm cannot predict the label of a point well without observing it in the dataset. In particular, fitting some of the examples will require memorizing their labels.
Nothing is known a priori about the frequency of any individual point aside from a prior distribution over the frequencies described by a list of frequencies . Our results are easiest to express when the objective of the learning algorithm is to minimize the expectation of the error over a random choice of the marginal distribution over from some meta-distribution (instead of the more usual worst-case error). In addition, for convenience of notation we will also measure the error with respect to a random choice of the labeling function from some distribution over . That is, the objective of a learning algorithm is defined as:
[TABLE]
Specifically, we consider the following meta-distribution over marginal distributions on : the frequency of each point in the domain is chosen randomly and independently from the prior of individual frequencies and then normalized to 1. This process results in a meta-distribution over marginal distributions that is similar to choosing the frequencies of the elements to be a random permutation of the elements of . Models measuring the worst-case error over all the permutations of a list of frequencies underlie the recent breakthroughs in the analysis of density estimation algorithms [OS15, VV16]. We believe that results similar to ours can be obtained in this worst-case model as well and leave such an extension for future work222The extension to measuring the worst-case error over the choice of , on the other hand, is straightforward..
Our main result (Thm. 2.3) directly relates the number of points that an algorithm does not fit to the sub-optimality (or excess error) of the algorithm via a quantity that depends only on the frequency prior and . Importantly, excess error is measured relative to the optimal algorithm and not relative to the best model in some class. Formally, we denote by the number of examples that appear once in the dataset and are mislabeled by the classifier that outputs on . A special case of our theorem states:
[TABLE]
Here refers to the expected generalization error of and is the minimum achievable error by any algorithm (expectations are with respect to the meta-distribution over learning problems resulting from the process we described, randomness of the learning algorithm and also sampling of the dataset). The important quantity here is
[TABLE]
where is the actual marginal distribution over frequencies that results from our process and is, basically, a slightly smoothed version of . We note that the optimal algorithm in this case does not depend on and thus our modeling does not require the learning algorithm to know to achieve near-optimal generalization error.
The quantity is easy to compute given . As a quick numerical example, for the prototypical long-tailed Zipf distribution (where the frequency of the -th most frequent item is proportional to ) over the universe of size and samples, one gets the expected loss of at least per every example the learner does not fit. For comparison, the worst-case loss (per point) in this setting is determined by the least frequent element and is . Given that the expected fraction of samples that appear once is , an algorithm that does not fit well will be suboptimal by (with the optimal top- error for balanced classes being in this case). More generally, we show that can be lower bounded by the total weight of the part of the prior which has frequency on the order of and also that the absence of frequencies on this order will imply negligible (see Sec. 2.5 for more details).
In our basic model the data is labeled correctly and fitting all the training examples (also referred to as interpolation) is the optimal strategy. We extend our model to a more general setting in which examples can be mislabeled. Under the assumption that the learning algorithm’s prior makes the observed label the most likely to be correct by some margin we demonstrate that memorization of labels is necessary for singleton examples. The cost of not fitting given in eq. (1) is now multiplied by (see Sec. 2.4 for details). Note that in the presence of noise, interpolation may no longer be the optimal strategy, and in particular, memorization of noisy labels can be necessary even in the non-interpolating regime.
Continuous data distributions:
Naturally, our simple setting in which individual points have significant probability does not capture the continuous and high-dimensional ML problems where each individual point has an exponentially small (in the dimension) probability. In this more general setting the prediction on the example itself has negligible effect on the generalization error. To show how the effects we demonstrated in the simple discrete setting extend to continuous distributions, we consider mixture models of subpopulations. In our model, the frequencies of subpopulations (or mixture coefficients) are selected randomly according to the prior as before. The labeling function is also chosen as before and is assumed to be constant over every subpopulation.
The discussion of the relationship between fitting the dataset and generalization makes sense only if one assumes that the prediction on the data point in the dataset will affect the predictions on related points. In our setting it is natural to assume that (with high probability) the learning algorithm’s prediction on a single point from a subpopulation will be correlated with the prediction on a random example from the same subpopulation. We refer to this condition as coupling (Defn. 3.1) and show that eq. (1) still holds up to the adjustment for the strength of the coupling.
Intuitively, it is clear that this form of “coupling” is likely to apply to “local” learning rules such as the nearest neighbors algorithm. Indeed, our assumption can be seen as a more abstract version of geometric smoothness conditions on the marginal distribution of the label used in analysis of such methods (e.g. [CD14]). We also show that it applies to linear predictors/SVMs in high dimension provided that distinct subpopulations are sufficiently uncorrelated (see Sec. 3.1). Deep neural networks are known to have some of the properties of both nearest neighbor rules and linear classifiers in the last-hidden-layer representation (e.g. [CSG18]). Thus DNNs are likely to exhibit this type of coupling as well.
From fitting to memorization and privacy:
The results we described so far demonstrate that an algorithm that does not fit the training data well will be suboptimal on long-tailed data distributions. Fitting of training data was not previously explained only when the learning algorithm fits the training labels much better than the test data, in other words, when the generalization gap is large (often ). Such fitting suggests that the training algorithm memorized a large fraction of the training labels. To make this intuition formal we give a simple definition of what memorizing a label of a point in the dataset means (we are not aware of a prior formal definition of this notion). Formally, for a dataset and define
[TABLE]
where denotes the dataset that is with removed. This value is typically non-negative and we think of label as memorized when this value is larger than some fixed positive constant (such as ). Namely, the label of an example is memorized if it is fit well by the algorithm despite being hard to predict based on the rest of the dataset.
This definition is closely related to the classical leave-one-out notion of stability [DW79, BE02] but focuses on the change in the label and not in the incurred loss. As in the case of stability, our notion of label memorization is directly related to the expected generalization gap. Indeed, the expectation over the choice of dataset of the average memorization value is equal to the expectation of the generalization gap. Thus a large generalization gap implies that a significant fraction of labels is memorized.
An immediate corollary of this definition is that an algorithm with a limited ability to memorize labels will not fit the singleton data points well whenever the algorithm cannot predict their labels based on the rest of the dataset. Two natural situations in which the algorithm will not be able to predict these labels are learning a complex labeling function (e.g. having large VC dimension) and computational hardness of finding a simple model of the data. In addition, the labels are also hard to predict in the presence of noise. A direct corollary of our results is that limiting memorization (for example via regularization or model compression) and differential privacy has costs in terms of achievable generalization error. The sharp quantitative nature of these results allows us to explain recent empirical findings demonstrating that these costs can disproportionably higher for less frequent subgroups in the population (see Section 4.4 for details).
1.2 Known empirical evidence
The best results (that we are aware of) on modern benchmarks that are achieved without interpolation are those for differentially private (DP) training algorithms [Aba+16, Pap+16, Pap+17, McM+18]. While not interpolating is not the goal, the properties of DP imply that a DP algorithm with the privacy parameter cannot memorize individual labels (see Sec.4.3 for more details on why). Moreover, they result in remarkably low gap between the training and test error that is formally explained by the generalization properties of DP [Dwo+14]. However, the test error results achieved in these works are well below the state-of-the-art using similar models and training algorithms. For example, [Pap+17] report accuracy of and on MNIST and SVHN as opposed to and , respectively when training the same models without privacy.
The motivation and inspiration for this work comes in part from attempts to understand why DP algorithms fall short of their non-private counterparts and which examples are they more likely to misclassify. A thorough and recent exploration related to this question can be found in the work of [CEP19]. They consider different ways to measure how “prototypical” each of the data points is according to several natural metrics and across MNIST, CIFAR-10, Fashion-MNIST and ImageNet datasets and compare between these metrics. One of those metrics is the highest level of privacy that a DP training algorithm can achieve while still correctly classifying an example that is correctly classified by a non-private model. As argued in that work and is clear from their comprehensive visualization, the examples on which a DP model errs are either outliers or atypical ones. To illustrate this point, we include the examples for MNIST digit “3” and CIFAR-10 “plane” class from their work as Fig. 2. In addition, the metric based on DP is well correlated with other metrics of being prototypical such as relative confidence of the (non-private) model and human annotation. Their concepts of most and least prototypical map naturally to the frequency of subpopulation in our model. Thus their work supports the view that the reason why learning with DP cannot achieve the same accuracy as non-private learning is that it cannot memorize the tail of the mixture distribution. This view also explains the recent empirical results showing that the decrease in accuracy is larger for less well represented subpopulations [BS19a].
Another empirical work that provides indirect support for our theory is [Arp+17]. It examines the relationship between memorization of random labels and performance of the network for different types of regularization techniques. The work demonstrates that for some regularization techniques it is possible to reduce the ability of the network to fit random labels without significantly impacting their performance on true labels. The explanation proposed for this finding is that memorization is not necessary for learning. While it may appear to contradict our theory, a closer look at the result suggests the opposite conclusion. On the true labels almost all their regularization techniques still reach near perfect train accuracy with test accuracy of at most . The only two techniques that do not quite interpolate (though still reaching around train accuracy) are exactly the ones that do exhibit clear correlation between ability to fit random labels and test accuracy (see “input binary mask” and “input gaussian” in their Figs. 10 and 11). We remark (and elaborate in Section 5) that fitting random examples or even interpolation are not necessary conditions for the application of our approach and for memorization being beneficial.
In a subsequent work with Chiyuan Zhang [FZ20a] we investigate label memorization and test the predictions of our theory directly. In particular, using an efficiently computable proxy for the memorization score, we discover examples whose labels are memorized in MNIST, CIFAR-10/100, and ImageNet datasets. Visual inspection of these examples confirms that these examples are a mix of outlier/mislabeled examples and correctly labeled but atypical examples. We then demonstrate that memorized examples are important for learning as removing them from the training set decreases the accuracy of the resulting model significantly. Further, the long-tail theory in this work predicts that there is a significant fraction of examples whose memorization is necessary for predicting accurately on examples from the same subpopulation in the test set. More formally, there exist examples in the training set such that for each of them the label is memorized by the learning algorithm in the sense defined above; there exists a dependent example in the test set in the following sense: the accuracy of the model on the dependent test example drops significantly when the corresponding example from the training set is removed (with no significant effect on the accuracy on the other test examples). We design an algorithm for testing this prediction efficiently. The results of this algorithm on MNIST, CIFAR-100, and ImageNet datasets reveal numerous visually similar pairs of relatively atypical examples [FZ20a, FZ20].
1.3 Related work
One line of research motivated by the empirical phenomena we discuss here studies implicit regularization in the overparameterized regime (namely, when the parameter space is large enough that the learning algorithm can perfectly fit the dataset). For example, the classical margin theory [Vap82, CV95, Sch+98] for SVMs and boosting suggests that, while the ambient dimension is large, the learning algorithm implicitly maximizes the margin. The generalization gap can then be upper bounded in terms of the margin. Examples of this approach in the context of DNNs can be found in [NTS15, Ney+17, BFT17, Ney+17a, LMZ18] (and references therein). These notions imply that it is beneficial to overparameterize and suffice for explaining why the training algorithm will select the best model among those that do fit the training set. However implicit regularization does not explain why, despite the regularization, the training error is near zero even when the generalization error is large.
Another line of research studies generalization properties of learning algorithms that fit the training data perfectly, often referred to as interpolating [BHM18, BMM18]. For example, a classical work of [CH67] gives bounds on the generalization error of the 1-nearest neighbor algorithm. Recent wave of interest in such methods has lead to new analyses of existing interpolating methods as well as new algorithmic techniques [Wyn+17, BRT18, BHM18, LR18, BMM18, RZ19, Bar+19, BHX19, Has+19, MVS19]. These works bypass the classical approach to generalization outlined above and demonstrate that interpolating methods can generalize while tolerating some amount of noise. In particular, they show that interpolation can be “harmless” in the sense that interpolating methods can in some cases achieve asymptotically optimal generalization error. At the same time, for the problems studied in these works there also exist non-interpolating algorithms with the same (or better) generalization guarantees. Thus these works do not explain why on many datasets (such as MNIST, CIFAR-10/100, SVHN) state-of-the-art classifiers interpolate the training data. We also remark that while interpolating the training set (with high generalization error) requires memorization, memorization also occurs without interpolation. For example, experiments of [Zha+17] show that training error is achieved by a standard deep learning algorithm on completely randomly labeled 1000-class ImageNet dataset (with generalization error being ).
It is known that in the convex setting SGD converges faster when all the loss functions have a joint minimizer [SST10, NWS14] and therefore it has been suggested that interpolation is the result of computational benefits of optimization via SGD [MBB18]. However this hypothesis is not well supported by empirical evidence since interpolation does not appear to significantly affect the speed with which the neural networks are trained [Zha+17]. In addition, methods like nearest neighbors, boosting, and bagging are not trained via SGD but tend to interpolate the data as well.
Algorithmic stability [BE02, Sha+09, HRS16, FV19] is essentially the only general approach that is known to imply generalization bounds beyond those achievable via uniform convergence [Sha+09, Fel16]. However it runs into exactly the same conceptual issue as capacity-based bounds: average stability needs to be increased by at least to fit an arbitrary label. In fact, an interpolating learning algorithm does not satisfy any non-trivial uniform stability (but may still be on-average stable).
We focus on interpolation and the importance of label memorization in learning as this is the phenomenon that had no prior explanation. However neural networks are known to memorize much more than just labels [Car+19, Car+20]. Such memorization presents even higher privacy risks and thus requires a more fundamental understanding. Building on the ideas in this work, recent work shows that for some natural data distributions, memorization of information about the entire sample can be necessary for achieving close-to-optimal generalization [Bro+20]
2 Fitting the Training Data in Unstructured Classification
In this section we describe a simple learning setting over an unstructured discrete domain that incorporates a prior over the distribution of frequencies. We demonstrate that in the noiseless setting, a learning algorithm that does not fit the training examples will be suboptimal and express the excess error in terms of the properties of the prior over frequencies. We show that this result also holds in the presence of label noise (although only for the singleton examples). We then show that the excess error is significant if and only if the distribution of frequencies is long-tailed. Finally, we compare the conclusions of our analysis with those of the standard approaches in our setting.
2.1 Preliminaries
For a natural number , we use to denote the set . For a condition (which defines a subset of some domain ) we use to denote the indicator function of the condition (from to ). A dataset is specified by an ordered -tuple of examples but we will also treat it as the multi-set of examples it includes. Let denote the set of all points that appear in .
For a probability distribution over , denotes choosing by sampling it randomly from . For subset (or condition) and function over , we denote by the probability distribution of , when and is conditioned on . For two probability distributions over the same domain we use to denote the total variation distance between them.
The goal of the learning algorithm is to predict the labels given a dataset consisting of i.i.d. samples from some unknown distribution over . For any function and distribution over , we denote . As usual, for a randomized learning algorithm we denote its expected generalization error on a dataset by
[TABLE]
where refers to being the output of a (possibly) randomized algorithm. We also denote by the expectation of the generalization error of when examples are drawn randomly from .
2.2 Problem setup
To capture the main phenomenon we are interested in, we start by considering a simple and general prediction problem in which the domain does not have any underlying structure (such as the notion of distance). The domains and are discrete, and (for concreteness one can think of and ).
The prior information about the labels is encoded using a distribution over functions from to . The key assumption is that nothing is known a priori about the frequency of any individual point aside from a prior distribution over the individual frequencies. One natural approach to capturing this assumption is to assume that the frequencies of the elements in are known up to a permutation. That is, a distribution over is defined by picking a random permutation of elements of the prior . Exact knowledge of the entire frequency prior is also a rather strong assumption in most learning problems. We therefore use a related but different way to model the frequencies (which we have not encountered in prior work). In our model the frequency of each point in is chosen randomly and independently from the list of possible frequencies and then normalized to sum up to .
More formally, let denote the distribution over probability mass functions on defined as follows. For every , sample randomly, independently and uniformly from the elements of . Define the corresponding probability mass function on as . This definition can be naturally generalized to sampling from a general distribution over frequencies (instead of just the uniform over a list of frequencies). We also denote by the resulting marginal distribution over the frequency of any single element in . That is,
[TABLE]
Note that, while is used to define the process, the actual distribution over individual frequencies the process results in is and our bounds will be stated in terms of properties of . At the same time, this distinction is not particularly significant for applications of our result since, as we will show later, is essentially a slightly smoothed version of .
The key property of this way to generate the frequency distribution is that it allows us to easily express the expected frequency of a sample conditioned on observing it in the dataset (or, equivalently, the mean of the posterior on the frequency). Specifically, in Appendix A we prove the following lemma:
Lemma 2.1**.**
For any frequency prior , and a sequence of points that includes exactly times, we have
[TABLE]
An instance of our learning problem is generated by picking a marginal distribution randomly from and picking the true labeling function randomly according to . We refer to the distribution over obtain by picking and outputting by . We abbreviate as whenever the prior and are clear from the context.
We are interested in evaluating the generalization error of a classification algorithm on instances of our learning problem. Our results apply (via a simple adaption) to the more common setup in statistical learning theory where is a set of functions and worst case error with respect to a choice of is considered. However for simplicity of notation and consistency with the random choice of , we focus on the expectation of the generalization error on a randomly chosen learning problem:
[TABLE]
2.3 The cost of not fitting
We will now demonstrate that for our simple problem there exists a precise relationship between how well an algorithm fits the labels of the points it observed and the excess generalization error of the algorithm. This relationship will be determined by the prior and . Importantly, this relationship will hold even when optimal achievable generalization error is high, a regime not covered by the usual analysis in the “realizable” setting.
In our results the effect of not fitting an example depends on the number of times it occurs in the dataset and therefore we count examples that does not fit separately for each possible multiplicity. More formally,
Definition 2.2**.**
For a dataset and , let denote the set of points that appear exactly times in . For a function let
[TABLE]
and let
[TABLE]
It is not hard to see (and we show this below) that in this noiseless setting the optimal expected generalization error is achieved by memorizing the dataset. Namely, by the algorithm that outputs the function that on the points in the dataset predicts the observed label and on points outside the dataset predicts the most likely label according to the posterior distribution on . We will now quantify the excess error of any algorithm that does not fit the labels of all the observed data points. Our result holds for every single dataset (and not just in expectation). To make this formal, we define to be the probability distribution over triples where , and . For any dataset , let denote the marginal distribution over distribution-function pairs conditioned on . That is:
[TABLE]
We then define the expected error of conditioned on dataset being equal to as
[TABLE]
We will also define to be the minimum of over all algorithms .
Theorem 2.3**.**
Let be a frequency prior with a corresponding marginal frequency distribution , and be a distribution over . Then for every learning algorithm and every dataset :
[TABLE]
where
[TABLE]
In particular,
[TABLE]
Proof.
We denote the marginal distribution of over by and the marginal distribution over by . We begin by noting that for every consistent with the examples in , the distribution of conditioned on is still , since is chosen independently of any labeling. Therefore we can conclude that is equal to the product distribution .
To prove the claim we will prove that
[TABLE]
where . This will imply the claim since the right-hand expression is minimized when for all , and for all ,
[TABLE]
Moreover, this minimum is achieved by the algorithm that fits the examples in and predicts the label that minimizes on all the points in . Namely,
[TABLE]
Plugging this into eq. (2) gives the first claim.
We now prove eq. (2).
[TABLE]
Using the fact that , for every we get
[TABLE]
Hence we obtain that the term in line (4) is exactly equal to the second term on the right hand side of eq. (2).
For the term in line (3), we pick an arbitrary for some . We can decompose
[TABLE]
since additional conditioning on does not affect the distribution of (as mentioned, is a product distribution). Let denote the sequence of points in the dataset . The labels of these points do not affect the conditioning of and therefore by Lemma 2.1,
[TABLE]
For a point , we denote by the label of in . This label is unique in our setting and is equal to for every in the support of . Therefore, by combining the above two equalities we obtain that, as claimed in eq.(2), line (3) is equal to
[TABLE]
To obtain the second part of the theorem we denote by the marginal distribution of over . Observe that
[TABLE]
since the optimal algorithm is given as an input. The second claim now follows by taking the expectation over the marginal distribution over :
[TABLE]
∎
2.4 Extension to label noise
A more general way to view Theorem 2.3 is that it translates excess error on points in into excess generalization error (excess error on points in is the difference between the total error of on and the error of the optimal algorithm on ). This view holds even if we allow noise in the labels. In the presence of noise the observed labels are not necessarily correct and therefore the error of on may no longer be equal to the empirical error . At the same time, if for a singleton example , the posterior probability of label on is higher than that of other labels, then fitting label on is still the optimal strategy. In this case any algorithm that does not do that will be suboptimal by at least by (for every such example). When the noise level is relatively low and affects primarily hard examples (which is the case in most standard benchmark datasets), the observed label is much more likely to be the correct one than the other labels. Thus on such datasets it is optimal to fit even noisy labels.
To make this argument formal we consider a more general setting in which for every true labeling function the examples are labeled by some . Formally, we assume that there is a possibly randomized mapping from the support of to and sampling of from also includes . In particular, in the conditional probability we include the randomness with respect to generation of (that labeled ) from . Further, it is natural to assume that for a singleton example its label given by is the most likely to be correct by some margin even conditioned on the rest of the dataset. Formally, we denote the confidence margin in the given label for the given prior as
[TABLE]
Theorem 2.4**.**
Using the notation in (the proof of) Theorem 2.3, we have that
[TABLE]
In particular,
[TABLE]
Proof.
As in the proof of Theorem 2.3, the fact that implies that
[TABLE]
where, as before, . This implies that
[TABLE]
By our definition in eq. (5), for every , if then
[TABLE]
Substituting this into eq. (7), we obtain the claimed result. ∎
2.5 From tails to bounds
Given a frequency prior , Theorem 2.3 gives a general and easy way to compute the effect of not fitting an example in the dataset. We now spell out some simple and easier to interpret corollaries of this general result and show that the effect can be very significant. The primary case of interest is , namely examples that appear only once in , which we refer to as singleton examples. In order to fit those, an algorithm needs to memorize their labels whenever is hard to learn (see Section 4.1 for a more detailed discussion). We first note that the expected number of singleton examples is determined by the weight of the entire tail of frequencies below in . Specifically, the expected fraction of the distribution contributed by frequencies in the range is defined as:
[TABLE]
At the same time the expected number of singleton points is:
[TABLE]
For every we have that (for sufficiently large ). Therefore:
[TABLE]
We will now show that the expected cost of not fitting any of the singleton examples is lower bounded by the weight contributed by frequencies on the order of . Our bounds will be stated in terms of the properties of (as opposed to itself) and therefore, before proceeding, we briefly explain the relationship between these two.
Relationship between and :
Before the normalization step, for every , is distributed exactly according to (that is uniform over . Therefore, it is sufficient to understand the distribution of the normalization factor conditioned on for some . Under this condition the normalization factor is distributed as the sum of independent samples from plus . The mean of each sample is exactly and thus standard concentration results can be used to obtain that is concentrated around . Tightness of this concentration depends on the properties of , most importantly, the largest value and . For , can be effectively seen as convolving each multiplicatively by a factor whose inverse is a Gaussian-like distribution of mean and variance . More formally, using Bernstein’s (or Bennett’s) concentration inequality (e.g. [Sri02]) we can easily relate the total weight in a certain range of frequencies under to the weight in a similar range under .
Lemma 2.5**.**
Let be a frequency prior and be the corresponding marginal distribution over frequencies. For any Then for and any ,
[TABLE]
where , and .
Note that
[TABLE]
By taking , we can ensure that the boundaries of the frequency interval change by a factor of at most (roughly) . For such we will obtain and in particular will suffice for making the correction at least (which is insignificant for our purposes).
Bounds for :
We now show a simple lower bound on in terms of (similar results hold for other choices of the interval ). We also do not optimize the constants in the bounds as our goal is to demonstrate the qualitative behavior.
Lemma 2.6**.**
For every frequency prior and sufficiently large ,
[TABLE]
If, in addition, , then
[TABLE]
Proof.
We first observe that the denominator of satisfies
[TABLE]
Now, by simple calculus, for every and sufficiently large ,
[TABLE]
Therefore
[TABLE]
To obtain the second part of the claim we apply Lemma 2.5 for (as discussed above). To verify, observe that for sufficiently large and , and , and . ∎
The value of corresponds to paying on the order of in generalization error for every example that is not fit by the algorithm. Hence if the total weight of frequencies in the range of is at least some then the algorithm that does not fit them will be suboptimal by times the fraction of such examples in the dataset. By eq. (8), the fraction of such examples themselves is determined by the weight of the entire tail . For example, if is the Zipf distribution and then and . Thus an algorithm that does not fit most of the singleton examples will be suboptimal by . Numerically, for an algorithm that in a binary prediction problem does no better than random on the singletons will have excess error of (relative to the optimum which is in this case).
We can contrast this situation with the case where there are no frequencies that are on the order of . Even when the data distribution has no elements with such frequency, the total weight of the frequencies in the tail and as a result the fraction of singleton points might be large. Still, as we show, in such case the cost of not fitting singleton examples will be negligible.
Lemma 2.7**.**
Let be a frequency prior such that for some , , where for . Then .
Proof.
We first observe that the numerator of is at most:
[TABLE]
By Markov’s inequality, implies
[TABLE]
In addition, by our definition of ,
[TABLE]
Therefore the first term in the numerator is upper bounded by . At the same time the second term in the numerator satisfies:
[TABLE]
Therefore the second term is at least as large as the first term and we obtain that:
[TABLE]
Thus as claimed. ∎
For , under the conditions of Lemma 2.7 we will obtain that the suboptimality of the algorithm that does not fit any of the singleton examples is at most .
2.6 Comparison with standard approaches to generalization
We now briefly demonstrate that standard approaches for analysis of generalization error cannot be used to derive the conclusions of this section and do not capture our simple problem whenever . For concreteness, we will use with the uniform prior over all labelings. We will also think of that consists of frequencies and frequencies (thus ). Without any structure in the labels, a natural class of algorithms for the problem are algorithms that pick a subset of points whose labels are memorized and predict randomly on the other points in the domain.
First of all, it is clear that any approach that does not make any assumption on the marginal distribution cannot adequately capture the generalization error of such algorithms. A distribution-independent generalization bound needs to apply to the uniform distribution over . For this distribution the expected generalization error for a randomly chosen labeling function will be at least . In particular, for sufficiently large , the differences in the generalization error of different algorithms will be insignificant and therefore such notion will not be useful for guiding the choice of the algorithm.
Notions that are based on the algorithm knowing the input distribution are not applicable to our setting. Indeed the main difficulty is that the algorithm does not know the exact frequencies of the singleton elements. An algorithm that knows would not need to fit the points whose frequency is . Thus the algorithm would be able to achieve excess generalization error of at most without fitting the dataset. In contrast, our analysis shows that an algorithm that only knows the prior and fits only of the dataset will be suboptimal by .
Fairly tight data-dependent bounds on the generalization error can be obtained via the notion of empirical Rademacher complexity [Kol01, BM02]. Empirical Rademacher complexity for a dataset and the class of all Boolean functions on that memorize points is . Similar bound can also be obtained via weak notions of stability such as average leave-one-out stability [BE02, RMP05, Muk+06, Sha+10]
[TABLE]
where refers to with -th example removed. If we were to use either of these notions to pick (the number of points to memorize), we would end up not fitting any of the singleton points. The simple reason for this is that, just like a learning algorithm cannot distinguish between “outlier” and “atypical” points given in this setting, neither will any bound. Therefore any true upper bound on the generalization error that is not aware of the prior on the frequencies needs to be correct when all the points that occur once are “outliers”. Fitting any of the outliers does not improve the generalization error at all and therefore such upper bounds on the generalization error cannot be used to correctly guide the choice of .
An additional issue with the standard approaches to analysis of the generalization error is that they bound the excess error of an algorithm relative to the best function in some class of functions or relative to the Bayes optimal predictor (which is the the optimal predictor for the true data distribution). In our model this would mean comparing with the perfect predictor which has generalization error of [math]. For the prior we consider, the optimal algorithm has generalization error of over . Thus theoretical analysis that is not close-to-perfectly tight will not lead to a meaningful bound. For example, standard bounds based on Rademacher complexity are suboptimal by a factor of at least two and thus lead to vacuous bounds. In contrast, our analysis can give a meaningful bound on the generalization error even when used with a relatively crude bound on the excess error.
3 General Mixture Models
Our problem setting in Section 2 considers discrete domains without any structure on . The results also focus on elements of the domain whose frequency is on the order of . Naturally, practical prediction problems are often high-dimensional with each individual point having an exponentially small (in the dimension) probability. Therefore direct application of our analysis from Section 2 for the unstructured case makes little sense. Indeed, any learning algorithm can be modified to a learning algorithm that does not fit any of the points in the dataset and achieves basically the same generalization error as simply by modifying ’s predictions on the training data to different labels and vice versa (any algorithm can be made to fit the dataset without any effect on its generalization).
At the same time in high dimensional settings the points have additional structure that can be exploited by a learning algorithm. Most machine learning algorithms are very likely to produce the same prediction on points that are sufficiently “close” in some representation. The representation itself may be designed based on domain knowledge or derived from data. This is clearly true about -NN, SVMs/linear predictors and has been empirically observed for neural networks once the trained representation in the last hidden layer is considered.
The second important aspect of natural image and text data is that it can be viewed as a mixture of numerous subpopulations. As we have discussed in the introduction, the relative frequency of these subpopulations has been observed to have a long-tailed distribution most obvious when considering the label distribution in extreme multiclass problems [ZAR14, BS17, WRH17, Kri+17, VHP17, Cui+18, VH+18, BS19] (see also Fig. 1). A natural way to think of and a common way to model subpopulations (or mixture components) is as consisting of points that are similar to each other yet sufficiently different from other points in the domain.
We capture the essence of these two properties using the following model that applies the ideas we developed in Section 2 to mixture models. To keep the main points clear we keep the model relatively simple by making relatively strong assumptions on the structure. (We discuss several ways in which the model’s assumptions can be relaxed or generalized later).
We model the unlabeled data distribution as a mixture of a large number of fixed distributions . For simplicity, we assume that these distributions have disjoint support, namely is supported over and for (without loss of generality ). For we denote to be the index of the sub-domain of and by (or ) the sub-domain (or subpopulation, respectively) itself.
The unknown marginal distribution is defined as for some vector of mixture coefficients that sums up to . We describe it as a distribution over (that is ). As in our unstructured model, we assume that nothing is known a priori about the mixture coefficients aside from (possibly) a prior described by a list of frequencies. The mixture coefficients are generated, as before, by sampling from . We denote by the distribution over defined as .
We assume that the entire subpopulation is labeled by the same label and the label prior is captured via an arbitrary distribution over functions from to . Note that such prior can be used to reflect a common situation where a subpopulation that is “close” to subpopulations and is likely to have the same label as either or . The labeling function for the entire domain is sampled by first sampling and defining .
To model the properties of the learning algorithm we assume that for every point in a dataset the distribution over predictions for a random predictor output by is close to (or at least not too different) from the distribution over predictions that produces over the entire subpopulation of . This follows the intuition that labeling will have a measurable effect on the prediction over the entire subpopulation. This effect may depend on the number of other points from the same subpopulation and therefore our assumption will be parameterized by parameters.
Definition 3.1**.**
Let be a domain partitioned into sub-domains with subpopulations over the sub-domains. For a dataset , let denote the union of subpopulations such that points from appear exactly times in . For , we say that an algorithm is -subpopulation-coupled if for every , ,
[TABLE]
Note that we do not restrict the algorithm to be coupled in this sense over subpopulations that are not represented in the data. This distinction is important since predictors output by most natural algorithms vary over regions from which no examples were observed. As a result the setting here cannot be derived by simply collapsing points in the sub-domain into a single point and applying the results from the unstructured case. However, the analysis and the results in Sec. 2 still apply essentially verbatim to this more general setup. All we need is to extend the definition of to look at the multiplicity of sub-domains and not points themselves and count mistakes just once per sub-domain. For a function let
[TABLE]
As before, . With this definition we get the following generalization of Theorem 2 (we only state the version for the total expectation of the error but the per-dataset version holds as well):
Theorem 3.2**.**
Let be subpopulations over sub-domains and let and be some frequency and label priors. Then for every -subpopulation-coupled learning algorithm :
[TABLE]
where is defined in Thm. 2.3.
We now briefly discuss how the modeling assumptions can be relaxed. We first note that it suffices for subpopulation coupling to hold with high probability over the choice of dataset from the marginal distribution over the datasets . Namely, if the property in Definition 3.1 holds with probability over the choice of (where, is the marginal distribution over the datasets) then the conclusion of the theorem holds up to an additional . This follows immediately from the fact that Theorem 3.2 holds for every dataset separately.
The assumption that the components of the mixture are supported on disjoint subdomains is potentially quite restrictive as it does not allow for ambiguous data points (for which Bayes optimal error is ). Subpopulations are also often modeled as Gaussians (or other distributions with unbounded support). If the probability of the overlap between the subpopulations is sufficiently small, then one can reduce this case to the disjoint one by modifying the components to have disjoint supports while changing the marginal distribution over by at most in the TV distance (and then appealing to the same argument as above). Dealing with a more general case allowing general overlap is significantly messier but the basic insight still applies: observing a single point sampled from some subpopulation increases the expectation of the frequency of the subpopulation under the posterior distribution. That increase can make this expectation significant making it necessary to memorize the label of the point.
3.1 Examples
We will now provide some intuition on why one would expect the -subpopulation-coupling to hold for some natural classes of algorithms. Our goal here is not to propose or justify specific models of data but rather to relate properties of known learning systems (and corresponding properties of data) to subpopulation coupling. Importantly, we aim to demonstrate that the coupling emerges from the interaction between the algorithm and the geometric properties of the data distribution and not from any explicit knowledge of subpopulations.
Local algorithms:
A simple example of a class of algorithms that will exhibit subpopulation coupling is -NN-like algorithms and other algorithms that are in some sense locally smooth. If subpopulations are sufficiently “clustered” so that including the example in the predictor will affect the prediction in the neighborhood of and the total weight of affected neighborhood is some fraction of the subpopulation, then we will obtain subpopulation coupling with . In the more concrete (and extreme case), when for every point , the most distant point in is closer than the closest point from the other subpopulations we will get that any example from a subpopulation will cause a -NN classifier to predict in the same way over the entire subpopulation. In particular, it would make it -subpopulation-coupled for .
Linear classifiers:
A more interesting case to understand is that of linear classifiers and by extension SVMs and (in a limited sense) neural networks. We will examine a high-dimensional setting, where . We will assume that points within each subpopulation are likely to have relatively large inner product whereas for every subpopulation most points will, with high probability have, a substantially large component that is orthogonal to the span of random samples from other populations. These conditions are impossible to satisfy when but are easy to satisfy when is sufficiently large. Formally, we assume that points in most datasets sampled from the data distribution satisfy the following condition:
Definition 3.3**.**
Let be a domain partitioned into subdomains . We say that a sequence of points is -independent if it holds that
- •
for all such that for some , and
- •
for all such that , and any , .
We consider the performance of linear classifiers that approximately maximize the margin. Here, by “approximately” we will simply assume that they output classifiers that achieve at least of the optimal margin achievable when separating the same points in the given dataset. Note that algorithms with this property are easy to implement efficiently via SGD on the cross-entropy loss [Sou+18] and also via simple regularization of the Perceptron algorithm [SSS05]. We will also assume that the linear classifiers output by the algorithm lie in the span of the points in the dataset333A linear classifier can always be projected to the span of the points without affecting the margins. This assumption allows us to avoid having to separately deal with spurious correlations between unseen parts of subpopulations and the produced classifiers. Formally, we define approximately margin-maximizing algorithms in this multi-class setting (for convenience, restricted to the homogeneous case) as follows:
Definition 3.4**.**
An algorithm is an approximately margin maximizing -class linear classifier if given a dataset it outputs linear classifiers satisfying:
- •
for every , lies in the span of ;
- •
for every , the prediction of on depends only on the predictions of the classifiers and;
- •
for every , let and . If can be linearly separated from by a homogeneous linear separator with margin then for all , .
We now show that linear classifiers over distributions that produce datasets independent in the sense of Definition 3.3 will have high subpopulation coupling. In order to guarantee strong coupling, we will assume that the set of points in a random dataset together with the set of points that consists of additional samples from every mixture present in (namely, ) satisfy the independence condition with high probability. Formally, we establish the following result (the proof can be found in Appendix B).
Theorem 3.5**.**
Let be a domain partitioned into sub-domains with subpopulations over the sub-domains. Let be any approximately margin maximizing -class linear classifier and be a frequency prior. Assume that for and , , with probability at least , is -independent for some . Then for any labeling prior , is -subpopulation-coupled with probability and .
As a simple example of subpopulations that will produce sets of points that are -independent with high probability we pick each to be a spherically-symmetric distribution supported on a ball of radius around some center of norm . We also pick the centers randomly and independently from the uniform distribution on the unit sphere. It is not hard to see that, by the standard concentration properties of spherically-symmetric distributions, a set of samples from an arbitrary mixture of such distributions will be -independent with high probability for and . Thus for , suffices to ensure that .
4 The Memorization, Privacy and Stability
So far we have discussed memorization by learning algorithms informally. In this section we give a simple definition of label memorization and demonstrate that fitting the training data in the setting we consider requires label memorization whenever there is enough (statistical or computational) uncertainty in the labels. This allows us to show that limits on the memorization ability of an algorithm translate into a loss of accuracy (on long-tailed distributions). This result explains a recent empirical finding [BPS19, Hoo+20, Hoo+20a] that in a dataset that is a mixture of several groups the loss in accuracy due to limited memorization will be higher on less frequent subgroups. Finally, we show that (even relatively weak forms of) differential privacy imply that the algorithm cannot memorize well.
To keep the notation cleaner we will discuss these results in the context of our simpler model from Sec.2 but they can be easily adapted to our mixture model setting. For simplicity of notation, we will also focus on memorization of singleton elements.
4.1 Memorization
To measure the ability of an algorithm to memorize labels we will look at how much the labeled example affects the prediction of the model on . This notion will be defined per specific dataset and example but in our applications we will use the expectation of this value when the dataset is drawn randomly.
Definition 4.1**.**
For a dataset and define
[TABLE]
where denotes the dataset that is with removed.
In this definition we measure the effect simply as the total variation distance444Strictly speaking, the memorization value can be negative (in which case it is equal to the negation of the TV distance) but for most practical algorithms we expect this value to be non-negative. between the distributions of the indicator of the label being , but other notions of distance could be appropriate in other applications. For this notion of distance our definition of memorization is closely related to the leave-one-out stability of the algorithm (see eq. (9)). Indeed, it is easy to see from this definition that LOO stability upper bounds the expected memorization:
[TABLE]
As in the case of stability label memorization can be related to the generalization gap in the following way (the proof follows immediately from taking the expectation over ).
Lemma 4.2**.**
For every distribution over and any learning algorithm we have that
[TABLE]
where is the expected empirical error of on :
[TABLE]
Note that the term is not exactly equal to the expectation of the generalization error , but in practice the difference between those is typically negligible (less than ). The immediate implication of Lemma 4.2 is that a large generalization gap indicates that many labels are memorized and vice versa.
An immediate corollary of our definition of memorization is that if cannot predict the label of without observing it then it needs to memorize it to fit it. More formally,
Lemma 4.3**.**
For every dataset , learning algorithm and index ,
[TABLE]
In particular,
[TABLE]
There can be several reasons why an algorithm cannot predict the label on without observing it. The simplest one is that if there is statistical uncertainty in the label. To measure the uncertainty in a distribution over labels we will simply use 1 minus the maximum probability of any specific label:
[TABLE]
Note that is exactly the error of the Bayes optimal predictor given that the posterior distribution on the label is .
Significant statistical uncertainty conditioned on knowing all the other labeled examples exists only when the labeling prior has high entropy (such as being uniform over a class of functions of VC dimension larger than ). In practice, there might exist a relatively simple model that explains the data well yet the learning algorithm cannot find (or even approximate) this model due to computational limitations. This can be modeled by considering the best accuracy in predicting the label of given for the restricted class of algorithms to which belongs. For example, the uniform prior can be achieved for all polynomial-time algorithms by using a pseudo-random labeling function [GGM86]. More generally, Lemma 4.3 implies that any upper bound on the expected accuracy of a learning algorithm on an unseen singleton example implies the need to memorize the label in order to fit it. Thus the results in the remainder of this section extend directly to computational notions of uncertainty in place of . We now spell out the properties of this simple statistical notion of uncertainty.
Lemma 4.4**.**
Let be an arbitrary distribution over . For a dataset , and , let denote the dataset with in place of example . Then we have:
[TABLE]
In particular, for every distribution and labeling prior that also generates the noisy labeling function for every (as in Sec. 2.4)
[TABLE]
where denotes the conditional distribution over the label of after observing all the other examples:
[TABLE]
Proof.
By Definition 4.1, for every ,
[TABLE]
Thus,
[TABLE]
giving the first claim.
The second claim follows from the definition of and observing that an expectation is taken on that ensures that for every point the error will be averaged over all labelings of the point according to conditional distribution of the corresponding label.
∎
4.2 The cost of limited memorization
We will now translate Lemma 4.4 into bounds on the excess error of algorithms that cannot memorize the labels well. For this purpose we will use the following definition.
Definition 4.5**.**
We say that a learning algorithm is -memorization limited if for all and all we have .
Bounds on memorization ability result directly from a variety of techniques, such as implicit and explicit regularization and model compression. Somewhat simplistically, one can think of these techniques as minimizing the sum some notion of capacity scaled by a regularization parameter and the empirical error. Fitting a label that is not predicted correctly based on the rest of the dataset typically requires increasing the capacity. Therefore a regularized algorithm will not fit the example if the increase in the capacity (scaled by ) does outweigh the decrease in the empirical error. These decisions are randomized and thus correspond to a bounded probability that the algorithm will memorize a label.
Using the definitions and Lemma 4.4, we immediately obtain the following example corollary on the excess error of any -memorization limited algorithm.
Corollary 4.6**.**
In the setting of Thm. 2.4, let be any -memorization limited algorithm. Then
[TABLE]
The bound in this corollary depends on the expectation of the uncertainty in the label . While, in general, this quantity might be hard to estimate it might be relatively easy to get a sufficiently strong upper bound. For example, if for the labeling is uniform and -wise independent for that upper-bounds the typical number of distinct points (or subpopulations in the general case) then, with high probability, it will hold that . As discussed in Section 2.5, for Zipf prior distribution and , any -memorization limited algorithm with being a constant will have excess error of . Equivalently, any algorithm that achieves the optimal generalization error will need to memorize labels. In particular, it will have a generalization gap of . These conclusions hold even in the presence of random noise. Consider, for example, the random classification noise model in which is defined by replacing the correct label with a random and uniformly chosen one with probability . For this model we will have that for singleton examples . Thus we obtain that even noisy labels need to be memorized as long as .
4.3 Cost of privacy
Memorization of the training data can be undesirable in a variety of settings. For example, in the context of user data privacy, memorization is known to lead to ability to mount black-box membership inference attacks (that discover the presence of a specific data point in the dataset) [Sho+17, LBG17, Lon+18, Tru+18] as well as ability to extract planted secrets from language models [Car+19]. The most common approaches toward defending such attacks are based on the notion of differential privacy [Dwo+06] that are formally known to limit the probability of membership inference by requiring that the output distribution of the learning algorithm is not too sensitive to individual data points. Despite significant recent progress in training deep learning networks with differential privacy, they still lag substantially behind the state-of-the-art results trained without differential privacy [SS15, Aba+16, Pap+16, Wu+17, Pap+17, McM+18]. While some of this lag is likely to be closed by improved techniques, our results imply that the some of this gap is inherent due to the data being long-tailed. More formally, we will show that the requirements differential privacy imply a lower bound on the value of (for simplicity just for ). We will prove that this limitation applies even to algorithms that satisfy a very weak form of privacy: label privacy for predictions. It protects only the privacy of the label as in [CH11] and also with respect to algorithms that only output a prediction on an (arbitrary) fixed point [DF18]. Formally, we define:
Definition 4.7**.**
Let be an algorithm that given a dataset outputs a random predictor . We say that is -differentially label-private prediction algorithm if for every and datasets that only differ in a label of a single element we have for any subset of labels ,
[TABLE]
It is easy to see that any algorithm that satisfies this notion of privacy is -memorization limited. A slightly more careful analysis in this case gives the following analogues of Lemma 4.4 and Corollary 4.6.
Theorem 4.8**.**
Let be an -differentially label-private prediction algorithm and let be an arbitrary distribution over . For a dataset , and , we have:
[TABLE]
In particular, in the setting of Thm. 2.4, for every distribution and labeling prior ,
[TABLE]
and, consequently,
[TABLE]
Proof.
By the definition of -differential label privacy for predictions, for every ,
[TABLE]
Thus,
[TABLE]
The rest of the claim follows as before. ∎
This theorem is easy to extend to any subpopulation from which only examples have been observed using the group privacy property of differential privacy. This property implies that if labels are changed then the resulting distributions are -close (in the same sense) [DR14]. The total weight of subpopulations that have at most examples for a small value of is likely to be significant in most modern datasets. Thus this may formally explain at least some of the gap in the results currently achieved using differentially private training algorithms and those achievable without the privacy constraint.
Uniform stability:
A related notion of stability is uniform prediction stability [BE02, DF18] that, in the context of prediction, requires that changing any point in the dataset does not change the label distribution on any point by more than in total variation distance. This notion is useful in ensuring generalization [BE02, FV19] and as a way to ensure robustness of predictions against data poisoning. In this context, -uniform stability implies that the algorithm is -memorization limited (and also is -differentially private for predictions). Therefore Corollary 4.6 implies limitations of such algorithms.
4.4 Disparate effect of limited memorization
Corollary 4.6 and Theorem 4.8 imply that limiting memorization increases the generalization error of an algorithm on long-tailed (and sufficiently hard) learning problems. Moreover, the excess error due to limited memorization depends on the prior , hardness of the problem and the number of samples . This implies that if the data distribution consists of several subgroups with different properties, then the cost of limiting memorization can be different for these subgroups. In particular, the cost can be higher for smaller subgroups or those with more distinct subpopulations. These are not hypothetical scenarios. For differential privacy these effects were observed in a concurrent work of [BS19a]. For model compression the differences in the costs have been confirmed and investigated in a subsequent work of Hooker et al. [Hoo+20, Hoo+20a]. In addition to disparate effects, these works empirically demonstrate that the increase in error is most pronounced on atypical examples.
As a concrete example of why our long-tail theory explains the different costs we consider a -class classification problem over subpopulations, Zipf prior , and samples. We will also assume for simplicity, that the labeling prior is uniform and independent over all subpopulations and there is no noise. Let be a -memorization limited learning algorithm for . The choice of does not affect the comparison as it will scale the excess error for all subgroups in the same way. The labels of all the subpopulations that have not been observed in the sample are completely unpredictable and therefore the expected error of the optimal algorithm in this setting is equal to
[TABLE]
To compute this value in our setting of parameters we will use instead of as those are very close for large and it is easier to perform (and verify) computations on . This gives us . Applying Corollary 4.6, we obtain that cost of limiting memorization to is .
Now, consider the same question but for a sample that only has examples. Then and the cost of limited memorization . Finally, consider the same question but with the number of subpopulations and (corresponding to a harder learning problem). Then and the cost of limited memorization is .
Next, assume that we are given a learning problem that is a mixture of the first and second settings, namely, the population is and we are given examples. Then in each subgroup we still have the same optimums and the same cost of limited memorization. The cost of limited memorization is more than twice higher for the smaller subgroup in this mixture problem. Similarly, in the mixture of the first and third settings ( and ) the cost of limited memorization is twice higher for the subgroup with a harder prediction problem.
The cost of memorization with 10 classes and is the same as the cost of (label) differential privacy for predictions with and so the same conclusions follow from Theorem 4.8.
Understanding of the causes of such disparate effects can be used to design mitigation strategies. For example, by using different levels of regularization (or compression) on different subgroups the costs can be balanced. Similarly, a different privacy parameter can be used for different subgroups (assuming that the additional risk of privacy violations is justified by the increase in the accuracy).
5 Discussion
Our work provides a natural and simple learning model in which memorization of labels and, in some cases interpolation, are necessary for achieving nearly optimal generalization when learning from a long-tailed data distribution. It suggests that the reason why many modern ML methods reach their best accuracy while (nearly) perfectly fitting the data is that these methods are (implicitly) tuned to handle the long tail of natural data distributions. Our model explicitly incorporates the prior distribution on the frequencies of subpopulations in the data and we argue that such modeling is necessary to avoid the disconnect between the classical view of generalization and the practice of ML. We hope that the insights derived from our approach will serve as the basis for future theoretical analyses of generalization that more faithfully reflect modern datasets and learning techniques. A recent example that such modeling has practical benefits can be found in [Cao+19].
Acknowledgements
Part of the inspiration and motivation for this work comes from empirical observations that differentially private algorithms have poor accuracy on atypical examples. I’m grateful to Nicholas Carlini, Ulfar Erlingsson and Nicolas Papernot for numerous illuminating discussions of experimental work on this topic [CEP19] and to Vitaly Shmatikov for sharing his insights on this phenomenon in the context of language models. I would like to thank my great colleagues Peter Bartlett, Misha Belkin, Olivier Bousquet, Edith Cohen, Roy Frostig, Daniel Hsu, Phil Long, Yishay Mansour, Mehryar Mohri, Tomer Koren, Sasha Rakhlin, Adam Smith, Kunal Talwar, Greg Valiant, and Chiyuan Zhang for insightful feedback and suggestions on this work. I thank the authors of [ZAR14] for the permission to include Figure 1 from their work.
Appendix A Proof Lemma 2.1
The key property of our problem definition is that it allows to decompose the probability of a dataset (under the entire generative process) into a probability of seeing one of the points in the dataset and the probability of seeing the rest of the dataset under a similar generative process. Specifically, we prove the following lemma.
Lemma A.1**.**
For , a sequence of points that includes exactly times, let be equal to with all the elements equal to omitted. Then for any frequency prior and in the support of , we have
[TABLE]
In particular:
[TABLE]
Proof.
We consider the distribution of conditioned on (which, by our assumption, is an event with positive probability). We denote this distribution by . From the definition of we get that a random sample from can generated by setting , then for all , sampling from and normalizing the results to sum to . That is, defining
[TABLE]
From here we obtain that an equivalent way to generate a random sample from is to sample from and then multiply the resulting p.m.f. by (with as before). Naturally, for any ,
[TABLE]
Now we denote by the subset of indices of elements of that are different from : . We can now conclude:
[TABLE]
The second part of the claim follows directly from the fact that, by definition of ,
[TABLE]
∎
We can now prove Lemma 2.1 which we restate here for convenience.
Lemma A.2** (Lemma 2.1 restated).**
For any frequency prior , and a sequence of points that includes exactly times, we have
[TABLE]
Proof.
We first observe that by the Bayes rule and Lemma A.1:
[TABLE]
This leads to the claim:
[TABLE]
∎
Appendix B Proof of Theorem 3.5
Theorem B.1** (Thm. 3.5 restated).**
Let be a domain partitioned into sub-domains with subpopulations over the sub-domains. Let be any approximately margin maximizing -class linear classifier and be a frequency prior. Assume that for and , , with probability at least , is -independent for some . Then for any labeling prior , is -subpopulation-coupled with probability and .
Proof.
For the given priors and , let be a dataset sampled from for and . Let . Let and let be sampled from , that is, consists of additional independent samples from every subpopulation with a single sample.
We will show that for any that is -independent, the output of any approximately margin maximizing -class linear classifier gives predictions on that are consistent with those on (which are defined by ): if for then for every ,
[TABLE]
By Defn. 3.4, this implies that the prediction of the classifier on is identical to that on . By our assumption, is not -independent with probability at most . By Markov’s inequality, probability over the choice of such that, the probability over the choice of that is not -independent is more than , is at most . By our definition of , the marginal distribution of is exactly . This implies that, with probability at least over the choice of the dataset , for every , and we have
[TABLE]
as required by Defn. 3.1 (for ).
To prove the stated consistency property for that is -independent, we will first show that every subset of points in can be separated from its complement with margin of . We will then use the properties of approximately margin maximizing classifiers and, again, independence to obtain consistency.
For any vector , we denote . To show that the margin is large, we define the weights explicitly by using one representative point from every subpopulation in . Without loss of generality, we can assume that these representatives are for some . Let be an arbitrary partition of these representatives into positively and negatively labeled ones. We define and consider the linear separator given by .
To evaluate the margin we first observe that . This follows via induction on :
[TABLE]
Now for , assume that and (without loss of generality) that is the representative of subdomain . Then
[TABLE]
Thus we obtain that is labeled in the same way as its representative and with margin of at least . This holds for all and therefore shows that the desired separation can be achieved with margin of at least .
Let be the linear separators returned by . Let be one of them. By our assumptions on , separates with margin of at least and further it lies in the span on . Namely, there exist such that .
We now pick an arbitrary singleton point from . Without loss of generality we assume that it is , and let be the point from the same subdomain . Let be the part of that excludes . By our assumption, is a singleton and therefore the points in are from other subdomains. By the independence of , this implies that and .
Now we need to show that the margin condition implies that is sufficiently large. Specifically,
[TABLE]
and thus
[TABLE]
where we used the fact that, by the triangle inequality, . This implies that . We can now bound
[TABLE]
where the last inequality assumes that . Thus we obtain that for every , every point in will be classified by in the same way as the point from the same subpopulation in . ∎
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[Aba+16] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan Mc Mahan, Ilya Mironov, Kunal Talwar and Li Zhang “Deep learning with differential privacy” In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , 2016, pp. 308–318 ACM
- 2[Arp+17] Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville and Yoshua Bengio “A closer look at memorization in deep networks” In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , 2017, pp. 233–242 JMLR. org
- 3[Bar+19] Peter L Bartlett, Philip M Long, Gábor Lugosi and Alexander Tsigler “Benign Overfitting in Linear Regression” In ar Xiv preprint ar Xiv:1906.11300 , 2019
- 4[BE 02] Olivier Bousquet and André Elisseeff “Stability and generalization” In JMLR 2 JMLR. org, 2002, pp. 499–526
- 5[BFT 17] Peter L Bartlett, Dylan J Foster and Matus J Telgarsky “Spectrally-normalized margin bounds for neural networks” In Advances in Neural Information Processing Systems , 2017, pp. 6240–6249
- 6[BHM 18] Mikhail Belkin, Daniel J Hsu and Partha Mitra “Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate” In Advances in Neural Information Processing Systems , 2018, pp. 2300–2311
- 7[BHX 19] Mikhail Belkin, Daniel Hsu and Ji Xu “Two models of double descent for weak features” In ar Xiv preprint ar Xiv:1903.07571 , 2019
- 8[BM 02] P. Bartlett and S. Mendelson “Rademacher and Gaussian Complexities: Risk Bounds and Structural Results” In Journal of Machine Learning Research 3 , 2002, pp. 463–482