Upper tails for arithmetic progressions revisited

TL;DR

This paper provides precise estimates for the probability that the number of k-term arithmetic progressions in a random subset exceeds its expectation by a large amount, revealing three distinct probabilistic regimes.

Contribution

It introduces a new combinatorial result on entropic stability, enhancing understanding of upper-tail probabilities in arithmetic progression counts.

Findings

Identifies three regimes of upper-tail behavior: Gaussian, Poisson, and a small set dominance.

Provides asymptotically sharp estimates for the upper-tail probabilities across parameter ranges.

Employs novel combinatorial techniques to analyze sets with rich arithmetic structure.

Abstract

Let $X$ be the number of $k$-term arithmetic progressions contained in the $p$-biased random subset of the first $N$ positive integers. We give asymptotically sharp estimates on the logarithmic upper-tail probability $\log \Pr(X \ge E[X] + t)$ for all $\Omega(N^{-2/k}) \le p \ll 1$ and all $t \gg \sqrt{Var(X)}$, excluding only a few boundary cases. In particular, we show that the space of parameters $(p,t)$ is partitioned into three phenomenologically distinct regions, where the upper-tail probabilities either resemble those of Gaussian or Poisson random variables, or are naturally described by the probability of appearance of a small set that contains nearly all of the excess $t$ progressions. We employ a variety of tools from probability theory, including classical tilting arguments and martingale concentration inequalities. However, the main technical innovation is a combinatorial result that establishes a stronger version of `entropic stability' for sets with rich arithmetic structure.