Computing nonsurjective primes associated to Galois representations of genus $2$ curves

TL;DR

This paper presents a practical algorithm to compute the finite set of primes where the Galois action on the Jacobian of genus 2 curves over Q is not maximal, leveraging classification, sampling, and modularity theorems.

Contribution

It introduces a new algorithm for computing nonsurjective primes for Galois representations of genus 2 curves, integrating theoretical classification with computational techniques.

Findings

Algorithm successfully implemented in Sage

Applied to all genus 2 curves with trivial endomorphism ring in LMFDB

Results are publicly available on curve homepages

Abstract

For a genus $2$ curve $C$ over $\mathbb{Q}$ whose Jacobian $A$ admits only trivial geometric endomorphisms, Serre's open image theorem for abelian surfaces asserts that there are only finitely many primes $\ell$ for which the Galois action on $\ell$-torsion points of $A$ is not maximal. Building on work of Dieulefait, we give a practical algorithm to compute this finite set. The key inputs are Mitchell's classification of maximal subgroups of $\mathrm{PSp_4}(\mathbb{F}_\ell)$, sampling of the characteristic polynomials of Frobenius, and the Khare--Wintenberger modularity theorem. The algorithm has been submitted for integration into Sage, executed on all of the genus~$2$ curves with trivial endomorphism ring in the LMFDB, and the results incorporated into the homepage of each such curve.