Mean square in the prime geodesic theorem

TL;DR

This paper establishes improved average bounds for the remainder in the prime geodesic theorem across cofinite Fuchsian groups, utilizing trace formulas to enhance understanding of geodesic distribution.

Contribution

It introduces new average bounds for the prime geodesic theorem's remainder, applying the Selberg and Kuznetsov trace formulas for broader classes of groups.

Findings

Improved mean square bounds for the prime geodesic theorem's remainder.

Enhanced bounds specifically for the modular group using Kuznetsov trace formula.

Results surpass previous pointwise bounds on average.

Abstract

We prove upper bounds for the mean square of the remainder in the prime geodesic theorem, for every cofinite Fuchsian group, which improve on average on the best known pointwise bounds. The proof relies on the Selberg trace formula. For the modular group we prove a refined upper bound by using the Kuznetsov trace formula.

Equations

$$\left(\begin{array}{cc}{\lambda }^{1/2}& 0\\ 0& {\lambda }^{-1/2}\end{array}\right),\lambda \in \mathbb{R},\lambda >1.$$

$${\pi }_{\Gamma }(X)=♯\{P_0:N{P}_0\le X\}.$$

$${\pi }_{\Gamma }(X)\sim \mathrm{li}(X),$$

$$\pi (X)=♯\{p\le X:p\text{ prime}\}\sim \mathrm{li}(X)\text{as }X\to \infty .$$

$$Z_{\Gamma}(s)=\prod_{P_{0}}\prod_{\nu=0}^{\infty}\big{(}1-(NP_{0})^{-\nu-s}\big{)},$$

$$\pi (X)-\mathrm{li}(X)\text{and}{\pi }_{\Gamma }(X)-\mathrm{li}(X).$$

$${\psi }_{\Gamma }(X)=\sum _{NP\le X}{\Lambda }_{\Gamma }(NP),$$

$$M_{\Gamma }(X)=\sum _{1/2<s_j\le 1}\frac{X^{s_j}}{s_j},$$

$$P_{\Gamma }(X)={\psi }_{\Gamma }(X)-M_{\Gamma }(X).$$

$$\frac{\psi (e^y)-e^y}{e^{y/2}}$$

$$\frac{1}{A}{\int }_A^{2A}|P_{\Gamma }(X){|}^{2}dX\ll A^{4/3}.$$

$$P_{\Gamma }(X)\ll X^{1/2+\epsilon }\forall \epsilon >0,$$

$$P_{\Gamma }(X)\ll X^{3/4}.$$

$$P_{\Gamma }(X)={\Omega }_{\pm }(X^{1/2-\delta })\forall \delta >0,$$

$$P_{\Gamma }(X)\ll X^{\eta +\epsilon }\forall \epsilon >0,$$

$$B=\{X\ge 1:|P_{\Gamma }(X)|\ge X^{2/3}(\log X{)}^{1+\epsilon }\}$$

$$\frac{1}{A}{\int }_A^{2A}{\left|P_G(X)\right|}^{2}dX\ll A^{5/4+\epsilon }.$$

$$P_G(X)\ll X^{\eta +\epsilon },\text{with}\eta =\frac{25}{36}\text{and for all}\epsilon >0,$$

$$R(X,T)=\sum _{t_j\le T}{X}^{i{t}_j}.$$

$$R(X,T)\ll T^{1+\epsilon }{X}^{\epsilon }$$

$$\frac{1}{A}{\int }_A^{2A}|R(X,T){|}^{2}dX\ll T^{2+\epsilon }{A}^{1/4+\epsilon },$$

$$\sum_{r,s\atop r\neq s}\frac{a_{r}a_{s}}{|r-s|^{\lambda}}\ll_{\lambda}\Big{(}\sum_{r}a_{r}^{p}\Big{)}^{2/p}.$$

$$\underset{A\to \infty }{\lim }\frac{1}{\log A}{\int }_1^A\left(\frac{P_{\Gamma }(X)}{X^{1/2}}\right)\frac{dX}{X}=L_{\Gamma }.$$

$$h(t)\text{ is even},$$

$$h(t)\text{ is holomorphic in the strip }|\Im (t)|\le 1/2+\epsilon ,$$

$$h(t)\ll (1+|t|{)}^{-2-\epsilon }\text{ in the strip.}$$

$$\sum _P\frac{g(\log NP)}{2\sinh ((\log NP)/2)}{\Lambda }_{\Gamma }(NP)=IE+EE+PE+DS+CS+AL,$$

$$IE=-\frac{\mathrm{vol}(\Gamma \backslash \mathbb{H})}{4\pi }{\int }_{-\infty }^{+\infty }th(t)\tanh (\pi t)dt.$$

$$EE=-\sum_{\mathcal{R}}\sum_{1\leq\ell<m}\big{(}2m\sin(\pi\ell/m)\big{)}^{-1}\int_{-\infty}^{+\infty}h(t)\frac{\cosh(\pi(1-2\ell/m)t)}{\cosh(\pi t)}dt.$$

$$PE=\frac{\mathfrak{C}}{2\pi }{\int }_{-\infty }^{+\infty }h(t)\psi (1+it)dt+\mathfrak{C}g(0)\log 2.$$

$$DS=\sum _{t_j}h(t_j),$$

Full text

This work was supported by the Max-Planck-Institut für Mathematik. We would like to record our thanks to the Institute for the excellent working conditions.

1. Introduction

There is a striking similarity between the distribution of lengths of primitive closed geodesics on the modular surface and prime numbers. If we consider the asymptotic of the associated counting function, the problem of finding optimal upper bounds on the remainder is intriguing, especially because the relevant zeta function is known to satisfy the corresponding Riemann hypothesis.

We start with reviewing briefly the framework of the problem (for a more detailed introduction we refer to [24], [14, §10.9]), and then state our results. Since the definitions extend to every finite volume Riemann surface, we work in this generality, and only at a later point we specialize to the modular surface.

Every cofinite Fuchsian group $\Gamma$ acts on the hyperbolic plane $\mathbb{H}$ by linear fractional transformations, and every hyperbolic element $g\in\Gamma$ is conjugated over $\operatorname{SL}_{2}(\mathbb{R})$ to a matrix of the form

[TABLE]

The trace of $g$ is therefore $\operatorname{Tr}(g)=\lambda^{1/2}+\lambda^{-1/2}$, and we define its norm to be $N(g)=\lambda$. Since the trace and the norm are constant on the conjugacy class of $g$, if we set $P=\{\gamma g\gamma^{-1},\gamma\in\Gamma\}$, we can define the trace and the norm of $P$ by $\operatorname{Tr}(P)=\operatorname{Tr}(g)$ and $NP=N(g)$. Moreover, we say that $g$ (resp. $P$) is primitive if $g$ cannot be expressed as a positive power (greater than one) of another element $g_{0}\in\Gamma$. Every class $P$ can then be written as $P=P_{0}^{\nu}$ for some primitive conjugacy class $P_{0}$ and some positive integer $\nu\geq 1$.

For $X>0$ define the counting function

[TABLE]

A hyperbolic conjugacy class $P$ determines a closed geodesic on the Riemann surface $\Gamma\backslash\mathbb{H}$, and its length is given exactly by $\log NP$. For this reason we say that $\pi_{\Gamma}(X)$ in (1.1) counts primitive hyperbolic conjugacy classes in $\Gamma$, lengths of primitive geodesics on $\Gamma\backslash\mathbb{H}$, or, by abusing language, “prime” geodesics.

It is a result from the middle of the last century (see e.g. Huber [9, 10], although the result was already known to Selberg) that as $X\to\infty$ we have the asymptotic formula

[TABLE]

where $\operatorname{li}(X)$ is the logarithmic integral. The asymptotic coincides with that of the prime counting function, since we have

[TABLE]

The structure of the two problems is also similar. The study of primes is strictly related to the study of the zeros of the Riemann zeta function $\zeta(s)$; on the other hand, for primitive geodesics we need to study the zeros of the Selberg zeta function $Z_{\Gamma}(s)$, which is defined for $\Re(s)>1$ by

[TABLE]

the outer product ranging over primitive hyperbolic classes in $\Gamma$, and extends to a meromorphic function on the complex plane.

It is an interesting question to determine the correct rate of the approximation in (1.2) and (1.3), namely to prove optimal upper bounds on the differences

[TABLE]

In the case of primes, the conjectural estimate $|\pi(X)-\operatorname{li}(X)|\ll X^{1/2}\log X$ is equivalent to the Riemann hypothesis (see e.g. [11, Th. 12.3]).

To understand what type of control we should expect on the difference on the right in (1.4), we introduce the weighted counting function

[TABLE]

where $\Lambda_{\Gamma}(NP)=\log(NP_{0})$ if $P$ is a power of the primitive conjugacy class $P_{0}$, and $\Lambda_{\Gamma}(x)=0$ otherwise. The function $\psi_{\Gamma}(X)$ is the analogous of the classical summatory von Mangoldt function $\psi(X)$ in the theory of primes, and studying $\pi_{\Gamma}(X)$ is equivalent to study $\psi_{\Gamma}(X)$, but it is easier to work with the latter.

The asymptotic (1.2) for $\pi_{\Gamma}(X)$ translates into $\psi_{\Gamma}(X)\sim X$ as $X$ tends to infinity. The spectral theory of automorphic forms provides a finite number of additional secondary terms, and we define the complete main term $M_{\Gamma}(X)$ to be the function

[TABLE]

where $s_{j}$ is defined by $\lambda_{j}=s_{j}(1-s_{j})$, $\lambda_{j}$ are the eigenvalues of the Laplace operator $\Delta$ acting on $L^{2}(\Gamma\backslash\mathbb{H})$, and the sum is restricted to the small eigenvalues, that is, those satisfying $0\leq\lambda_{j}<1/4$. Since the eigenvalues of $\Delta$ form a discrete set with no accumulation points, the sum in (1.5) is finite. The remainder $P_{\Gamma}(X)$ is then defined as

[TABLE]

Various types of pointwise upper bounds have been proved in the past years on $P_{\Gamma}(X)$, and we review them in Remark 1.5 below. In a different direction, it is possible to study the moments of $P_{\Gamma}(X)$, and this has been neglected so far.

In the case of the prime number theorem, it is a classical result (assuming the Riemann hypothesis) that the normalized remainder

[TABLE]

admits moments of every order and a limiting distribution with exponentially small tails (see [28, p. 242] and [23, Th. 1.2]). The aim of this paper is to prove the following estimate on the second moment of $P_{\Gamma}(X)$.

Theorem 1.1**.**

Let $\Gamma$ be a cofinite Fuchsian group. For $A\gg 1$ we have

[TABLE]

Remark 1.2**.**

By construction the zeros of the Selberg zeta function correspond to the eigenvalues of $\Delta$. The zeros corresponding to small eigenvalues give rise to the secondary terms in the definition of $M_{\Gamma}(X)$, and there are no other zeros in the half plane $\Re(s)>1/2$. In this sense $Z_{\Gamma}(s)$ satisfies an analogous of the Riemann hypothesis, and the fact suggests that we might have

[TABLE]

since this is the case for the primes under RH. However, the same method of proof fails, due to the abundance of zeros of $Z_{\Gamma}(s)$ (which in turn is related to the fact that $Z_{\Gamma}(s)$ is a function of order two, while $\zeta(s)$ is of order one), and only gives

[TABLE]

For a proof we refer to [14, Th. 10.5]. On the other hand, it is possible to prove that (1.6), if true, is optimal, since we have the Omega result by Hejhal [8, Th. 3.8, p. 477, and note 18, p. 503]

[TABLE]

which can be strengthened to $\Omega_{\pm}(X^{1/2}(\log\log X)^{1/2})$ for cocompact groups and congruence groups.

Remark 1.3**.**

Recently Koyama [16] and Avdispahić [1, 2] have tried an approach to find upper bounds on $P_{\Gamma}(X)$ using a lemma of Gallagher [6, Lemma 1]. Their results are of the type

[TABLE]

where $\eta=7/10$ (resp. $\eta=3/4$) for $\Gamma$ cocompact (resp. ordinary cofinite), and with $X$ outside a set $B\subseteq[1,\infty)$ of finite logarithmic measure, that is, such that $\int_{B}X^{-1}dX<\infty.$ Theorem 1.1 improves on this to the extent that $\eta$ can be taken to be any number greater than $2/3$. Indeed, a direct consequence of Theorem 1.1 is that for every cofinite Fuchsian group, and for every $\varepsilon>0$, the set

[TABLE]

has finite logarithmic measure. Iwaniec claims without proof in [12, p. 187] that “it is easy to prove that $P_{\Gamma}(X)\ll X^{2/3}$ for almost all $X$”. Theorem 1.1 proves his claim (up to $\varepsilon$) for $X$ outside a set of finite logarithmic measure.

In the case of the modular group we can prove a stronger bound than for the general cofinite case. In this case we can exploit the Kuznetsov trace formula, obtaining the following.

Theorem 1.4**.**

Let $G=\operatorname{PSL}(2,\mathbb{Z})$. For $A\gg 1$ and every $\varepsilon>0$,

[TABLE]

Remark 1.5**.**

The estimate in (1.7) is currently the best known pointwise upper bound in the case of general cofinite Fuchsian groups. For the modular group and congruence groups it is possible to prove sharper estimates. Iwaniec [13, Th. 2], Luo and Sarnak [19, Th. 1.4], Cai [3, p. 62], and Soundararajan and Young [26, Th. 1.1] have worked on reducing the exponent for the modular group $G=\operatorname{PSL}(2,\mathbb{Z})$. The currently best known result is

[TABLE]

due to Soundararajan and Young. The exponent $7/10$, proved by Luo and Sarnak, holds also for congruence groups, see [20, Cor. 1.2], and for cocompact groups coming from quaternion algebras, see [15].

Observe that not only the exponent $\eta=1/2$ is out of reach, but it seems to be a hard problem reaching and, afterwards, breaking the barrier $\eta=2/3$. Iwaniec suggested ([12, p. 188], [13, eq. (12)]) that this follows from the assumption of the Lindelöf hypothesis for Rankin L-functions. In Theorem 1.1 we prove the bound with the exponent $2/3$ on average, unconditionally, for every cofinite Fuchsian group. Theorem 1.4 is saying that $P_{G}(X)\ll X^{5/8+\varepsilon}$ on average. This goes halfway between the trivial bound (1.7) and the conjectural estimate (1.6).

In the proof of Theorem 1.4 we use a truncated formula for $\psi_{\Gamma}(X)$, proved by Iwaniec [13, Lemma 1], condensing the analysis of the Selberg trace formula, and relating $P_{\Gamma}(X)$ to the spectral exponential sum

[TABLE]

The trivial bound for this sum, in view of Weyl’s law (see e.g. [14, Cor. 11.2] or [27, Th. 7.3]), is $|R(X,T)|\ll T^{2}$. Reducing the exponent of $T$ is possible at the cost of some power of $X$ (see [13, 19]), and Petridis and Risager conjectured [21, Conj. 2.2] that we should have square root cancellation, that is,

[TABLE]

for every $\varepsilon>0$. This would give the conjectural bound (1.6) for the modular group. In the appendix to [21] Laaksonen provides numerics that support this conjecture. In the proof of Theorem 1.4 (see Proposition 4.5) we prove that we have

[TABLE]

and hence $|R(X,T)|\ll T^{1+\varepsilon}X^{1/8+\varepsilon}$ on average.

The mean square estimate in Theorem 1.4 reduces to proving a similar estimate for certain weighted sums of Kloosterman sums. We obtain the desired bounds in a relatively clean way by applying the Hardy-Littlewood-Pólya inequality [7, Th. 381, p. 288] in a special case: for $0<\lambda<1$, $\lambda=2(1-p^{-1})$, and $\{a_{r}\}$ a sequence of non-negative numbers, then we have

[TABLE]

With more work it is possible to sharpen Theorem 1.4 by replacing $A^{\varepsilon}$ by some power of $\log A$. To do this, one can use a version of (1.8) with explicit implied constant proved by Carneiro and Vaaler [4, Cor. 7.2 eq. (7.20)], or the extremal case of (1.8) with $(\lambda,p)=(1,2)$, and a logarithmic correction, proved by Li and Villavert [18].

We also note that we simply use the Weil bound when estimating the weighted sums of Kloosterman sums and we do not exploit any cancellation amongst the Kloosterman sums. Exploring this phenomenon could lead to a power-saving improvement of Theorem 1.4.

For the proof of Theorem 1.1 we use the Selberg trace formula with a suitably chosen test function that fullfills our needs. In the general case of $\Gamma$ cofinite, unlike in the modular group case, Iwaniec’s truncated formula for $\psi_{\Gamma}(X)$ is not available. It is probably possible to prove an analogous result, but we preferred to work directly with the trace formula. In fact, in order to prove his formula, Iwaniec uses some arithmetic information on the structure of the lengths of closed geodesics on the modular surface [13, Lemma 4], namely their connection with primitive binary quadratic forms (proved by Sarnak [25, §1]), and it is unclear whether a similar argument works for the general cofinite case.

Remark 1.6**.**

We do not discuss the first moment of $P_{\Gamma}(X)$. Following the argument of [22, Th. 1.1 and Th. 4.1] it should be possible to prove that for every cofinite Fuchsian group $\Gamma$ there exists a constant $L_{\Gamma}$ such that we have

[TABLE]

2. Proof of Theorem 1.1

In this section we explain the structure of the proof of Theorem 1.1. We decided to keep this section to a more colloquial tone, and to relegate the technical computations to the next section. Hence the proof of the theorem is completed in section 3.

The starting point of our analysis is the Selberg trace formula, that we recall from the book of Iwaniec (see [14, Th. 10.2]). An even function $g$ is said to be an admissible test function in the trace formula if its Fourier transform $h(t)$ (see (2.4) for the convention used to define the Fourier transform) satisfies the conditions ([14, eq. (1.63)])

[TABLE]

For an admissible test function $g$, the Selberg trace formula is the identity

[TABLE]

where the sum on the left runs over hyperbolic conjugacy classes in $\Gamma$, and the terms appearing on the right are explained as follows.

The term $IE$ denotes a contribution coming from the identity element, which forms a conjugacy class on its own. We have

[TABLE]

The term $EE$ denotes a contribution from the elliptic conjugacy classes in $\Gamma$ (there are only finitely many such classes). Denote by $\mathcal{R}$ a primitive elliptic conjugacy class, and let $m=m_{\mathcal{R}}>1$ be the order of $\mathcal{R}$. We have

[TABLE]

The term $PE$ denotes a contribution from the parabolic conjugacy classes, which are associated to the cusps of $\Gamma$. Let $\mathfrak{C}$ denote the total number of inequivalent cusps. Then we have

[TABLE]

Here $\psi(z)$ is the digamma function, i.e. the logarithmic derivative of the Gamma function, $\psi(z)=(\Gamma^{\prime}/\Gamma)(z)$. The term $DS$ corresponds to the discrete spectrum, and is given by

[TABLE]

where the sum runs over the spectral parameter $t_{j}$, associated to the eigenvalue $\lambda_{j}$ of the Laplace operator via the identity $\lambda_{j}=1/4+t_{j}^{2}$ (here and in the following we assume that $t_{j}\in[0,i/2]$ for $\lambda_{j}\in[0,1/4]$, and $t_{j}>0$ for $\lambda_{j}>1/4$). The term $CS$ comes from the continuous spectrum, and is given by

[TABLE]

where $\varphi(s)$ is the scattering determinant for the group $\Gamma$ (see [14, p. 140]). Finally, the term $AL$ is a single term that comes from a combination of the spectral part and the geometric part (see [14, eq. (10.11) and (10.17)]), and it is defined by $AL=h(0)\operatorname{Tr}(\Phi(1/2)-\mathrm{I})/4$, where $\Phi(s)$ is the scattering matrix associated to $\Gamma$.

A first naive choice of test function in the Selberg trace formula (2.2) is

[TABLE]

where $\rho=\log X$. If we choose $g(x)$ in this way, then the left hand side of (2.2) reduces exactly to the function $\psi_{\Gamma}(X)$. Unfortunately the function $g_{\rho}(x)$ in (2.3) is not an admissible test function, and we need therefore to take a suitable approximation of it. An analysis of the right hand side in (2.2) then leads to the desired results. To see that $g_{\rho}$ is not admissible in the trace formula, consider its Fourier transform $h_{\rho}(t)$. We have

[TABLE]

The integral can be computed directly, which gives, for $t=\pm i/2$,

[TABLE]

and, for $t\neq\pm i/2$,

[TABLE]

The last integral in (2.4) shows that $h_{\rho}(t)$ is even and entire in $t$, but from (2.6) we see that we only have $h_{\rho}(t)\ll t^{-1}$ as $t$ tends to infinity, and so we do not have sufficient decay as required in (2.1).

We construct two functions $g_{\pm}(x)$ that approximate from above and below the function $g_{\rho}$, and are admissible in the Selberg trace formula. Let $q(x)$ be an even, smooth, non-negative function on $\mathbb{R}$ with compact support contained in $[-1,1]$ and unit mass (i.e. $\|q\|_{1}=1$). Let $0<\delta<1/4$ and define

[TABLE]

For $\rho>1$ define $g_{\pm}$ to be the convolution product of the shifted function $g_{\rho\pm\delta}$ with the function $q_{\delta}$, namely

[TABLE]

Taking convolution products has the advantage that the Fourier transform of the convolution is the pointwise product of the Fourier transforms of the two factors. Hence if we denote by $\widehat{q}_{\delta}(t)$ the Fourier transform of $q_{\delta}$, that is,

[TABLE]

and by $h_{\pm}$ the Fourier transform of $g_{\pm}$, then we obtain

[TABLE]

Since the function $\widehat{q}_{\delta}(t)$ is entire and satisfies, for $|\Im(t)|\leq M<\infty$,

[TABLE]

(see Lemma 3.1), we conclude that $h_{\pm}(t)$ is an entire function and satisfies $h_{\pm}(t)\ll(1+|t|)^{-2-\varepsilon}$ for some $\varepsilon>0$ in the strip $|\Im(t)|\leq 1/2+\varepsilon$, as required in (2.1). This shows that the function $g_{\pm}$ is an admissible test function in the trace formula. By construction, the function $g_{\pm}(x)$ is supported on $|x|\in[0,\rho+\delta\pm\delta]$. Moreover we have the inequalities (see Lemma 3.2), for $x\geq 0$,

[TABLE]

If we set

[TABLE]

then using the asymptotic $\psi_{\Gamma}(X)\sim X$ we conclude that we have the inequalities

[TABLE]

From this we deduce that we have

[TABLE]

and in order to prove Theorem 1.1 we give bounds on the right hand side in (2.10). We found convenient to pass to the logarithmic variable $\rho=\log X$, since this simplifies slightly the computations. Moreover, we insert a weight function in the integral to pass from the sharp mean square to a smooth one. Consider a smooth, non-negative real function $w(\rho)$, compactly supported in $[-\varepsilon,1+\varepsilon]$, for $0<\varepsilon<1/4$. In addition, assume that $0\leq w(\rho)\leq 1$, and that $w(\rho)=1$ for $\rho\in[0,1]$. Let

[TABLE]

In view of the inequality

[TABLE]

where $R=\log A$, we see that in order to estimate the last integral in (2.10) it suffices to upper bound the integral on the right in (2.12). In the following, abusing notation, we write $\psi_{\pm}(\rho)$ in place of $\psi_{\pm}(e^{\rho})$, and $M_{\Gamma}(\rho)$ in place of $M_{\Gamma}(e^{\rho})$.

At this point we can exploit the Selberg trace formula to analyze $\psi_{\pm}$. The terms of the discrete spectrum $DS$ associated to the small eigenvalues $\lambda_{j}\in[0,1/4]$ (corresponding to the spectral parameter $t_{j}$ in the interval $[0,i/2]$, and for technical convenience we include the eigenvalue $\lambda_{j}=1/4$) need particular care. We define $M_{\pm}(\rho)$ to be the sum of such terms, namely

[TABLE]

In view of the definition of the complete main term in (1.5) it is easy to prove (see Lemma 3.3) that we have

[TABLE]

Hence we have

[TABLE]

where the term $DS_{\pm}^{\prime}$ denotes now the contribution from the discrete spectrum restricted to the eigenvalues $\lambda_{j}>1/4$. The first three terms and the term $AL_{\pm}$ in (2.13) can be bounded pointwise (see Lemma 3.5 – 3.7) by

[TABLE]

Squaring and integrating in (2.13) we obtain therefore

[TABLE]

In Proposition 3.10 and Proposition 3.11 we prove that the following estimate holds:

[TABLE]

Combining (2.14) and (2.15) we obtain

[TABLE]

and choosing $\delta=e^{-R/3}$ to optimize the first and the last term, we arrive at the bound

[TABLE]

Recalling (2.10) and (2.12), and setting $R=\log A$, we conclude that we have

[TABLE]

This proves Theorem 1.1.

3. Technical lemmata

In this section we prove the auxiliary results needed in section 2 to prove Theorem 1.1. We start with a simple computation to bound the function $\widehat{q}_{\delta}(t)$ and its derivatives.

Lemma 3.1**.**

Let $0<\delta<1/4$, $\widehat{q}_{\delta}(t)$ as in (2.8), and let $t\in\mathbb{R}$. Let $j,k\in\mathbb{N}$, $j,k\geq 0$. We have

[TABLE]

and the implied constant depends on $j$ and $k$.

Proof.

From the definition of $\widehat{q}_{\delta}(t)$ we have

[TABLE]

Bounding in absolute value we get

[TABLE]

Integrating by parts in the last integral in (3.1), and using that $q(x)$ is smooth and has compact support, we obtain instead

[TABLE]

Combining (3.2) and (3.3) we obtain

[TABLE]

This proves the lemma. ∎

The function $\widehat{q}_{\delta}(t)$ is used to construct the convolution product $g_{\pm}=g_{\rho\pm\delta}\ast q_{\delta}$ that approximate the function $g_{\rho}$. In the next lemma we show that $g_{\rho}$ is bounded from above and below by $g_{+}$ and $g_{-}$ respectively, up to a small error.

Lemma 3.2**.**

Let $\rho>1$ and $0<\delta<1/4$. Let $g_{\rho}(x)$ and $g_{\pm}(x)$ as in (2.3) and (2.7) respectively. For $x\geq 0$ we have

[TABLE]

Proof.

Observe that $g_{\pm}=g_{\rho}\ast q_{\delta}$, and since both factors in the convolution product are non-negative, we conclude that $g_{\pm}(x)\geq 0$ for every $x\in\mathbb{R}$. Observe also that $g_{\pm}$ is supported on the compact set $\{|x|\leq\rho+\delta\pm\delta\}$. By definition of $g_{\pm}$ we can therefore write, for $x\geq 0$,

[TABLE]

where $Q=[\max\{-\delta,x-(\rho\pm\delta)\},\delta]$. For $0\leq x\leq\delta$ we have

[TABLE]

so that (3.4) holds trivially.

For $x>\delta$ we have $x-y>0$ for every $y\in Q$, and using the addition formula for the hyperbolic sine we can write

[TABLE]

Now for $\delta<x<\rho\pm\delta-\delta$ we have $Q=[-\delta,\delta]$, so that (3.5) reduces to

[TABLE]

For $\rho-2\delta\leq x\leq\rho$ we can bound by positivity in (3.5)

[TABLE]

so that the first inequality in (3.4) holds in this case. Finally for $\rho\leq x\leq\rho+2\delta$ we observe that $g_{\rho}(x)=0$ and $g_{+}(x)\geq 0$, and so we conclude that the second inequality in (3.4) holds in this case. This proves the lemma. ∎

The function $h_{\pm}(t)$ associated to $g_{\pm}(x)$ gives, for $t$ corresponding to small eigenvalues, the terms appearing in the definition of the main term $M_{\Gamma}(X)$ in (1.5), up to small error. We prove this in the next lemma.

Lemma 3.3**.**

Let $\rho>1$, $0<\delta<1/4$, and let $h_{\pm}(t)$ as in (2.9). Let $t_{j}\in(0,i/2]$ be the spectral parameter associated to the eigenvalue $\lambda_{j}\in[0,1/4)$. There exists $0<\varepsilon_{\Gamma}<1/4$ such that

[TABLE]

Proof.

Recall that there are only finitely many eigenvalues $\lambda_{j}$ in $(0,1/4)$, and therefore there exists $0<\varepsilon_{\Gamma}<1/4$ such that $|t_{j}|\geq\varepsilon_{\Gamma}$ for every $\lambda_{j}\in(0,1/4)$. Observe that for $|t|\leq 1/2$ we have

[TABLE]

The claim then follows from (2.5) and (2.6), and from the fact that $h_{\pm}(t)=h_{\rho\pm\delta}(t)\widehat{q}_{\delta}(t)$. ∎

Remark 3.4**.**

Equation (3.6) also holds for $\lambda_{j}=1/4$, except that we have to multiply the first term by a factor of $2$.

The next three lemmata show that in the problem of estimating $P_{\Gamma}(X)$ using the Selberg trace formula we can neglect the terms coming from the identity class, the elliptic classes, and the parabolic classes, as they contribute a small quantity.

Lemma 3.5**.**

Let $\rho>1$, $0<\delta<1/4$, and let $g_{\pm}$ and $h_{\pm}$ as in (2.7) and (2.9) respectively. Then we have

[TABLE]

Proof.

Recall that we have $h_{\pm}(t)=h_{\rho}(t)\widehat{q}_{\delta}(t)$, and that by Lemma 3.1 we have $\widehat{q}_{\delta}(t)\ll(1+|\delta t|^{k})^{-1}$ for every $k\geq 0$. Using (2.6) and the definition of the hyperbolic cosine we can write

[TABLE]

Bounding in absolute value the integrand associated to the last term in (3.7), we obtain

[TABLE]

Consider now the integrand associated to the first term in (3.7) We integrate by parts twice and obtain

[TABLE]

By Lemma 3.1 we see that the boundary terms vanish, and we can bound

[TABLE]

for every $k\geq 0$, with implied constant depending on $k$. Fixing $k>1$ we obtain

[TABLE]

The other terms in (3.7) are treated similarly, and are bounded by the same quantity. Adding (3.8) and (3.9) we obtain the desired estimate. ∎

Lemma 3.6**.**

Let $\rho>1$, $0<\delta<1/4$, and $h_{\pm}$ as in (2.9). Let $m,\ell\in\mathbb{N}$, with $m\geq 2$ and $1\leq\ell<m$. Then

[TABLE]

with implied constant that depends on $m$ and $\ell$.

Proof.

Recall that $h_{\pm}(t)=h_{\rho\pm\delta}(t)\widehat{q}_{\delta}(t)$, and use (3.7) to express $h_{\rho\pm\delta}(t)$. The integrand associated to the last term in (3.7) is bounded by

[TABLE]

Consider now the term $\exp((\rho\pm\delta)(1/2+it))$ in (3.7). The corresponding integral contributes

[TABLE]

The boundary terms vanish, and by Lemma 3.1 we can bound

[TABLE]

where the implied constant depends on $m$ and $\ell$. Hence we get

[TABLE]

The terms associated to the other exponentials in (3.7) are treated similarly, and are bounded by the same quantity. ∎

Lemma 3.7**.**

Let $\rho>1$, $0<\delta<1/4$, and let $g_{\pm}$ and $h_{\pm}$ as in (2.7) and (2.9). Then

[TABLE]

Proof.

We use here the formula given in [14, eq. (10.17)] to get back to an integral involving $g_{\pm}$. We have

[TABLE]

By (3.7) we know that

[TABLE]

and by definition of $g_{\pm}(x)$ we have $g_{\pm}(x)=O(\delta)$ for $|x|\leq\delta$. The last integral in (3.10) is analyzed as follows. For $x\in[0,\delta]$ we bound

[TABLE]

so that we have, uniformly in $\delta$,

[TABLE]

For $x>\delta$ we integrate by parts obtaining

[TABLE]

Since $g_{\pm}(\delta)\ll\delta$, the boundary term is bounded by $O(\delta\log(\delta^{-1}))$. If we write $\cosh(x/2)=\sinh(x/2)+e^{-x/2}$, the integral associated to $e^{-x/2}$ can be bounded by

[TABLE]

Finally we have

[TABLE]

The exponential cancels with the first term in (3.11), and combining the other estimates we obtain the claim. ∎

We turn now our attention to finding upper bounds for the mean square of the spectral side in the Selberg trace formula. We start with an estimate for the integral of $h_{\pm}(t_{1})h_{\pm}(t_{2})$.

Lemma 3.8**.**

Let $0<\delta<1/4$, let $h_{\rho\pm\delta}(t)$ be as in (2.9), and let $w_{R}(\rho)$ be as in (2.11). Let $R>1$, and $t_{1},t_{2}\in\mathbb{R}$. Then we have

[TABLE]

where $v(t)=(1+|t|)^{-1}$, and the implied constant does not depend on $\delta$.

Proof.

In order to express $h_{\rho\pm\delta}$ we consider again equation (3.7).

Multiplying $h_{\rho\pm\delta}(t_{1})$ with $h_{\rho\pm\delta}(t_{2})$, the product of the first exponential in (3.7) (from the factor $h_{\rho\pm\delta}(t_{1})$) and the last term from the factor $h_{\rho\pm\delta}(t_{2})$ contributes

[TABLE]

Consider now the product of the first exponential in (3.7) for $h_{\rho\pm\delta}(t_{1})$ and the same term for $h_{\rho\pm\delta}(t_{2})$. This contributes

[TABLE]

The other terms in the product $h_{\rho\pm\delta}(t_{1})h_{\rho\pm\delta}(t_{2})$ are bounded similarly by (3.12) and (3.13), except that we need to replace $|t_{1}+t_{2}|$ by $|t_{1}-t_{2}|$ when we integrate the product $\exp((\rho\pm\delta)(\pm 1\pm i(t_{1}-t_{2}))$. This concludes the proof. ∎

In order to exploit at best the bound proved in the previous lemma, we estimate the size of the spectrum on unit intervals.

Lemma 3.9**.**

Let $\Gamma$ be a cofinite Fuchsian group, let $\varphi(s)$ be the scattering determinant associated to $\Gamma$, and let $T>1$. We have

[TABLE]

Proof.

Recall Weyl’s law in its strong form (see [27, Th. 7.3])

[TABLE]

where we recall that $\mathfrak{C}$ is the number of inequivalent cusps of $\Gamma$. Consider the above equation at the point $T+1$, and subtract from it the same quantity for $T$. In order to shorten notation we write $f(t)=-\varphi^{\prime}/\varphi(1/2+it)$. We get

[TABLE]

The function $f(t)$ is bounded from below by a constant $k$ that depends on the group (this follows from the Maass-Selberg relations, see [14, eq. (10.9)]). Hence we have

[TABLE]

Since the number $\sharp\{T\leq t_{j}\leq T+1\}$ is non-negative, we can write

[TABLE]

Again from the fact that $f(t)$ is bounded from below by a constant, we infer that in fact we have

[TABLE]

Similarly we find

[TABLE]

and from (3.14) we conclude that we also have $\sharp\{T\leq t_{j}\leq T+1\}\ll T$. This proves the lemma. ∎

At this point we can prove bounds on the mean square of the discrete and continuous spectrum in the Selberg trace formula. We discuss first the discrete spectrum.

Proposition 3.10**.**

Let $0<\delta<1/4$, let $h_{\pm}$ as in (2.9), and let $R>1$. We have

[TABLE]

Proof.

Recall that $h_{\pm}(t)=h_{\rho\pm\delta(t)}\widehat{q}_{\delta}(t)$, and that $\widehat{q}_{\delta}(t)\ll(1+|\delta t|^{k})^{-1}$ for every $k\geq 0$. Using Lemma 3.9 this implies that the series

[TABLE]

is absolutely convergent, and we can write

[TABLE]

By Lemma 3.8 we can estimate the integral and bound the double sum by

[TABLE]

where $v(t)=(1+|t|)^{-1}$. Using Lemma 3.1 to bound $\widehat{q}_{\delta}(t)\ll(1+|\delta t|^{k})^{-1}$ for every $k\geq 0$, we can estimate the second sum in (3.15) by

[TABLE]

Consider now the first sum in (3.15). By symmetry and positivity, we can consider only the sum over $t_{\ell}\geq t_{j}$. Moreover we split the sum in order to optimize the bounds available. Consider a unit neighbourhood of the diagonal $t_{\ell}=t_{j}$. Using Lemma 3.9 we can estimate

[TABLE]

The tail of the double sum, that is, the range $t_{j}>\delta^{-1}$ and $t_{\ell}>t_{j}+1$, can be analyzed (we follow here the same method as in Cramér [5]) by using a unit interval decomposition for the sum over $t_{\ell}$, together with Lemma 3.9, to get

[TABLE]

Finally, the range $t_{j}\leq\delta^{-1}$ and $t_{\ell}>t_{j}+1$ is bounded by

[TABLE]

We conclude that we have

[TABLE]

as claimed. ∎

The analysis of the continuous spectrum is similar, and we obtain the same bounds. With the proposition below we conclude the list of auxiliary results needed to prove Theorem 1.1.

Proposition 3.11**.**

Let $0<\delta<1/4$ and $h_{\pm}$ be as in (2.9). Let $R\geq 1$. Then

[TABLE]

Proof.

Recall that we have $h_{\pm}(t)=h_{\rho\pm\delta}(t)\widehat{q}_{\delta}(t)$, and that $\widehat{q}_{\delta}(t)\ll(1+|\delta t|^{k})^{-1}$ for every $k\geq 0$. For simplicity we write

[TABLE]

Let $J$ denote the integral in (3.16). Due to the decay properties of $\widehat{q}_{\delta}$ and to Lemma 3.9, $J$ is absolutely convergent, and so we can write

[TABLE]

The innermost integral is bounded using Lemma 3.8. This gives

[TABLE]

The second integral in (3.17) is bounded by

[TABLE]

where in the first inequality we have used a unit interval decomposition of the domain of integration, and Lemma 3.9 to bound the integral of $|f(t)|$ in unit intervals. Consider now the first integral in (3.17). By symmetry and positivity we can consider only the integral over $t_{2}\geq t_{1}\geq 0$. A unit neighbourhood of the diagonal $t_{1}=t_{2}$ gives

[TABLE]

The tail of the double integral, that is, the range $t_{1}\geq\delta^{-1}$ and $t_{2}\geq t_{1}+1$, can be bounded as follows:

[TABLE]

The range $t_{1}\leq\delta^{-1}$ and $t_{2}\geq t_{1}+1$ contributes

[TABLE]

Summarizing, we have showed that we have the bound

[TABLE]

which is what we wanted. This proves the proposition. ∎

4. Modular group

This section is devoted to the proof of Theorem 1.4, concerning the case $G=\text{PSL}_{2}(\mathbb{Z})$. Our approach starts with a lemma of Iwaniec [13, Lemma 1], which gives

[TABLE]

where $1\leq T\leq X^{1/2}(\log X)^{-2}$ (and it is understood that the sum runs over $t_{j}>0$). Note that in this case the only small eigenvalue of $\Delta$ is $\lambda=0$, so that $M_{G}(X)=X$ and $P_{G}(X)=\psi_{G}(X)-X$. From the equation above we deduce that

[TABLE]

We now describe the outline of the proof of Theorem 1.4. The main idea is to use the Kuznetsov trace formula to estimate the mean square of

[TABLE]

or, equivalently by partial summation, the mean square of

[TABLE]

Therefore, we start by setting up the Kuznetsov trace formula with a test function that gives us a a smooth version of the sum in (4.2), and we estimate the mean square of the weighted sum of Kloosterman sums that show up in its geometric side (Lemma 4.1). This allows us to bound the spectral side of the trace formula (Lemma 4.2). In order to control the behavior of the Fourier coefficients in the spectral sums of the trace formula we use a smoothed average of these sums. This way we obtain (Lemma 4.4) a mean square estimate of a smoothed version of $R(X,T)$, from which we extract a mean square estimate for the sharp sum (Lemma 4.5).

We now start by setting up the Kuznetsov trace formula. Let $\phi(x)$ be a smooth function on $[0,\infty]$ such that

[TABLE]

for $l=0,1,2,3$. Define

[TABLE]

where $J_{\nu}$ is the Bessel function of the first kind and order $\nu$. By the properties of the Hankel transform we have

[TABLE]

We now choose $\phi$ as in Luo-Sarnak [19]. For $X,T>1$ we set

[TABLE]

and apply the Kuznetsov trace formula [17, Th. 1] with $\phi_{X,T}$ as the test function. Let $\{f_{i}\}_{i=1}^{\infty}$ be an orthonormal basis of Maass cusp forms for $\text{SL}_{2}(\mathbb{Z})$, with eigenvalues $\lambda_{j}=\frac{1}{4}+t_{j}^{2}$. These cusp forms have Fourier expansions [17, eq. (2.10)]

[TABLE]

where $K_{\nu}(x)$ is the $K$-Bessel function. For $l_{1},l_{2}\geq 1$ the trace formula reads

[TABLE]

where $\rho_{j}(l)=\nu_{j}(l)\cosh(\pi t_{j})^{1/2}$, $d_{it}(l)=\sum_{d_{1}d_{2}=l}(d_{1}/d_{2})^{it}$, and $S(l_{1},l_{2},c)$ is the classical Kloostermann sum. By [19, p. 234] we have

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where

[TABLE]

Analyzing the right hand side of the Kuznetsov trace formula, we prove a bound for the mean square of $S_{n}(\phi_{X,T})$ as follows.

Lemma 4.1**.**

Let $A,T>2$ and let $n$ be a positive integer. Then, for any $\varepsilon>0$,

[TABLE]

We postpone the proof of Lemma 4.1 to the end of the section, and we show here how to recover a similar bound on the spectral side of the Kuznetsov trace formula.

Lemma 4.2**.**

Let $A,T>2$ and let $n$ be a positive integer. Then, for any $\varepsilon>0$,

[TABLE]

Proof.

By the trace formula and the bounds in equations (4.4)–(4.6) we deduce

[TABLE]

Observing that $d^{4}(n)\ll n^{\varepsilon}$, the claim follows from Lemma 4.1. ∎

Once we have bounds for a fixed $n$, we average over $n\in[N,2N]$. Let $h(\xi)$ be a smooth function supported in $[N,2N]$, whose derivatives satisfy

[TABLE]

and such that

[TABLE]

The next lemma is a result of Luo-Sarnak [19].

Lemma 4.3**.**

Let $h$ be as above. Then

[TABLE]

Proof.

See [19, pg.233]. ∎

Lemma 4.4**.**

Let $A,T>2$. Then, for any $\varepsilon>0$,

[TABLE]

Proof.

By the previous lemma

[TABLE]

where the last step can be obtained by $|\hat{\phi}(t_{j})|\ll e^{-t_{j}/T}$ and partial summation. Note now that from equation (4.3) it follows that

[TABLE]

Combining (4.8) and (4.9) we deduce that

[TABLE]

Taking absolute value, squaring, and integrating over $X$, we infer that

[TABLE]

Moreover, by Cauchy-Schwarz and the properties of $h$, we have

[TABLE]

Finally, we obtain

[TABLE]

where the last inequality follows from Lemma 4.2. Choose $N=A^{-1/4}T^{2}$ to complete the proof. ∎

The next step is to replace the smoothed sum $\sum X^{it_{j}}e^{-t_{j}/T}$ with the truncated one. This gives us a corresponding mean square estimate for $R(X,T)$.

Proposition 4.5**.**

Let $A,T>2$, and let $R(X,T)$ be as in (4.2). Then, for any $\varepsilon>0$,

[TABLE]

Proof.

We start by choosing a smooth function $g$ that approximates the characteristic function of $[1,T]$. Let $g$ be a smooth function supported on $[1/2,T+1/2]$ such that $0\leq g(\xi)\leq 1$, and $g(\xi)=1$ when $\xi\in[1,T]$. By the strong Weyl’s law we know that $|\{t_{j}:U\leq t_{j}\leq U+1\}|\ll U$, and so we have

[TABLE]

Define $\hat{g}(\xi)$ to be the Fourier transform of $g(\xi)\exp(\xi/T)$. By [19, p. 235-236] we have that

[TABLE]

where the error term can be bounded by $O(T\log T)$. Also note the estimate

[TABLE]

Defining

[TABLE]

we deduce that

[TABLE]

The first term in the sum above we bound, using Lemma 4.4 and the first bound in (4.10), by

[TABLE]

and the second term, using again Lemma 4.4 and the second bound in (4.10), by

[TABLE]

On taking $\delta=T^{-2}$ we obtain the claim. ∎

We are now able to conclude the proof of Theorem 1.4. Let $2<T\leq A^{1/2}(\log A)^{-2}$. By partial summation,

[TABLE]

Therefore,

[TABLE]

The first term on the right hand side is bounded, in view of Proposition 4.5, by $O(A^{5/4+\varepsilon})$. The second term can be bounded by using Cauchy-Schwarz inequality and again Proposition 4.5, giving

[TABLE]

Inserting these bounds in (4.1) we obtain

[TABLE]

Choosing $T=A^{3/8}$ above concludes the proof of Theorem 1.4.

4.1. Weighted sums of Kloosterman sums

In the remainder of the section we prove Lemma 4.1. Recall that we want to estimate, on average, the following sum of Kloostermann sums

[TABLE]

where $S_{n}(\psi)$ is the weighted sum of Kloosterman sums defined in (4.7), i.e.

[TABLE]

with $n$ a (large) positive integer. The function $\phi_{X,T}$ carries some oscillation in $X$, that we exploit when integrating over $X\in[A,2A]$.

Lemma 4.6**.**

Let $A,T>2$, and let $z_{1},z_{2}$ be positive real numbers. Then

[TABLE]

and

[TABLE]

For the last bound we assume $z_{1}\neq z_{2}$.

Proof.

Inserting the definition of $\phi_{X,T}$ we can write the integral as

[TABLE]

Bounding the integrand uniformly using

[TABLE]

we obtain the first bound

[TABLE]

To obtain the second bound we use integration by parts in equation (4.11) to get

[TABLE]

where

[TABLE]

We can bound these terms (up to a constant) by

[TABLE]

which gives us the second bound

[TABLE]

This proves the lemma. ∎

The lemma above will provide the necessary bounds to establish Lemma 4.1.

Proof of Lemma 4.1.

We expand the square in the integrand and exchange the integral with the sums, so that we can rewrite the integral as

[TABLE]

We now split the sum in equation (4.12) into two sums $\Sigma_{d}$ and $\Sigma_{nd}$, where $\Sigma_{d}$ is the sum over the diagonal terms $c_{1}=c_{2}$, and $\Sigma_{nd}$ is the sum over the terms $c_{1}\neq c_{2}$. We shall make use of the Weil bound on Kloosterman sums throughout the proof, namely

[TABLE]

Moreover, we have

[TABLE]

We bound the diagonal terms using the first bound in Lemma 4.6 (with $z_{1}=z_{2}=(4\pi n)/c$), obtaining

[TABLE]

To bound the non-diagonal terms we interpolate the two bounds in Lemma 4.6 to get, for $0<\lambda<1$,

[TABLE]

Therefore,

[TABLE]

where the last inequality follows from the Hardy-Littlewood-Pólya inequality (1.8). Applying the Weil bound (4.13) we obtain

[TABLE]

Pick $\lambda=1-\varepsilon$ and note that $d(n)\ll_{\varepsilon}n^{\epsilon}$ to finish the proof. ∎