Vafa-Witten invariants for projective surfaces I: stable case

TL;DR

This paper develops Vafa-Witten invariants for projective surfaces in the stable case, using virtual localisation and symmetric obstruction theory, connecting to modular forms and instanton moduli spaces.

Contribution

It introduces a new framework for defining Vafa-Witten invariants on surfaces with stable Higgs pairs, incorporating rational contributions beyond the vanishing theorem case.

Findings

Invariants are constant under deformations.

Calculations recover modular form predictions.

Results extend understanding of instanton moduli spaces.

Abstract

On a polarised surface, solutions of the Vafa-Witten equations correspond to certain polystable Higgs pairs. When stability and semistability coincide, the moduli space admits a symmetric obstruction theory and a $\mathbb C^*$ action with compact fixed locus. Applying virtual localisation we define invariants constant under deformations. When the vanishing theorem of Vafa-Witten holds, the result is the (signed) Euler characteristic of the moduli space of instantons. In general there are other, rational, contributions. Calculations of these on surfaces with positive canonical bundle recover the first terms of modular forms predicted by Vafa and Witten.

Equations

$${\mathcal{A}}_P\times {\Omega }^{+}({\mathfrak{g}}_P)\times {\Omega }^0({\mathfrak{g}}_P)⟶$$

$$(d_A,B,\Gamma )\mid \text{joinrel}\text{relbar}\text{joinrel}\to$$

$$F_A^{0,2}$$

$$F_A^{1,1}\wedge \omega +[\varphi ,\overline{\varphi }]$$

$${\overline{\partial }}_A\varphi$$

$$(E,\varphi ),\varphi \in \mathrm{Hom}(E,E\otimes K_S),$$

$$\frac{c_1(F)\cdot h}{\mathrm{rank}(F)}<\frac{c_1(E)\cdot h}{\mathrm{rank}(E)}$$

$${\mathrm{Higgs}}_{K_S}(S)\cong {\mathrm{Coh}}_c(X);$$

$$\widetilde{\mathsf{VW}}_{r,c_{1},c_{2}}(S)\ =\ \int_{\big{[}\mathcal{N}_{r,c_{1},c_{2}}\big{]}^{\operatorname{vir}}}\frac{1}{e(N^{\operatorname{vir}})}\ \in\ \mathbb{Q},$$

$$\cdots ⟶\mathrm{Hom}(E,E\otimes K_S)⟶{\mathrm{Ext}}^1({\mathcal{E}}_{\varphi },{\mathcal{E}}_{\varphi })⟶{\mathrm{Ext}}^1(E,E)⟶\cdots$$

$$\mathcal{N}_{L}^{\perp}\ =\ \big{\{}(E,\phi)\colon\det E\cong L,\ \operatorname{tr}\phi=0\big{\}}.$$

$${\mathrm{Ext}}_X^{\ast }({\mathcal{E}}_{\varphi },{\mathcal{E}}_{\varphi })\cong H^{\ast -1}(K_S)\oplus H^{\ast }({\mathcal{O}}_S)\oplus {\mathrm{Ext}}_X^{\ast }({\mathcal{E}}_{\varphi },{\mathcal{E}}_{\varphi }{)}_{\perp }.$$

$$\mathsf{VW}_{r,c_{1},c_{2}}\ :=\ \int_{\big{[}(\mathcal{N}^{\perp}_{r,L,c_{2}})^{\mathbb{C}^{*}}\big{]}^{\operatorname{vir}\ }}\frac{1}{e(N^{\operatorname{vir}})}\ \in\ \mathbb{Q}.$$

$${\mathcal{N}}_L^{\perp }\times T^{\ast }[-1]B{\mathbb{C}}^{\ast }\subset T^{\ast }[-1]\text{curly}{M}_L.$$

$$E\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$$T_{E^{\ast }}^{\ast }\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$$\pm e({\mathcal{M}}_{\mathrm{asd}}).$$

$$\mathcal{N}\subset T^{\ast }[-1]\text{curly}M.$$

$${\mathcal{N}}_L^{\perp }\subset T^{\ast }[-1]\text{curly}{M}_L,$$

$${\chi }^B:\mathcal{N}⟶\mathbb{Z}$$

$$\int_{[\mathcal{N}]^{\operatorname{vir}}}1\ =\ e\big{(}\mathcal{N},\chi^{B}\big{)}\ :=\ \sum_{i\in\mathbb{Z}}i\cdot e\big{(}(\chi^{B})^{-1}(i)\big{)}.$$

$$e(\mathcal{N},\chi^{B})\ =\ e\Big{(}\mathcal{N}^{\mathbb{C}^{*}},\chi^{B}\big{|}_{\mathcal{N}^{\mathbb{C}^{*}}}\Big{)}.$$

$$\mathsf{vw}_{r,c_{1},c_{2}}\ :=\ e\Big{(}(\mathcal{N}_{L}^{\perp})^{\mathbb{C}^{*}},\chi^{B}\big{|}_{(\mathcal{N}_{L}^{\perp})^{\mathbb{C}^{*}}}\Big{)}\ \in\ \mathbb{Z}.$$

$${\mathrm{Ob}}_{\mathcal{N}}⟶{\mathcal{O}}_{\mathcal{N}}\mathrm{and}{\mathrm{Ob}}_{{\mathcal{N}}_L^{\perp }}⟶{\mathcal{O}}_{{\mathcal{N}}_L^{\perp }}.$$

$$\big{[}\mathcal{N}\big{]}^{\mathrm{loc}}\,\in\,A_{0}\big{(}\mathcal{N}^{\mathbb{C}^{*}}\big{)}\qquad\mathrm{and}\qquad\big{[}\mathcal{N}^{\perp}_{L}\big{]}^{\mathrm{loc}}\,\in\,A_{0}\big{(}(\mathcal{N}^{\perp}_{L})^{\mathbb{C}^{*}}\big{)}.$$

$$\mathsf{vw}^{\prime}_{r,c_{1},c_{2}}\ :=\ \int_{\big{[}\mathcal{N}^{\perp}_{L}\big{]}^{\mathrm{loc}}}1.$$

$$\mathsf{vw}={\mathsf{vw}}^{\prime },$$

$$\mathsf{vw}\ =\ \mathsf{vw}^{\prime}\ =\ (-1)^{\operatorname{vd}}e\Big{(}(\mathcal{N}^{\perp}_{L})^{\mathbb{C}^{*}}\Big{)}.$$

$$Z_r(S,L):=q^{-s}\sum _{n\in \mathbb{Z}}{\mathsf{VW}}_{r,c_1(L),n}(S)q^n$$

$$c(1-q{)}^{g-1}{\left(1+\frac{1-3q}{\sqrt{(1-q)(1-9q)}}\right)}^{1-g},$$

$$\begin{gathered} \sum _{n\in \mathbb{Z}}{\mathsf{VW}}_{2,K_S,n}^{\varphi \ne 0}(S)q^n= \\ \frac{1}{2^{p_g(S)+g}}\prod _{n=1}^{\infty }(1-q^{2n}{)}^{-12(p_g(S)+1)}(1-q^n{)}^{2g-2}{\left(\sum _{j=0}^{\infty }{q}^{j^2+j}\right)}^{1-g}(\mathrm{mod}{q}^4). \end{gathered}$$

$$({\mathcal{E}}_{\varphi },s)$$

Full text

When the vanishing theorem of Vafa-Witten holds, the result is the (signed) Euler characteristic of the moduli space of instantons. In general there are other, rational, contributions. Calculations of these on surfaces with positive canonical bundle recover the first terms of modular forms predicted by Vafa and Witten.

1. 1 Introduction

2. 1.1 The Vafa-Witten equations

3. 1.2 The equations on a Kähler surface

4. 1.3 Hitchin-Kobayashi correspondence

5. 1.4 Spectral construction

6. 1.5 Localisation and a $U(r)$ Vafa-Witten invariant

7. 1.6 \fortocVirtual technicalities and an $SU(r)$ VW invariant\excepttocVirtual technicalities and an $SU(r)$ Vafa-Witten invariant

8. 1.7 Digression: derived algebraic geometry

9. 1.8 Cotangent field theories

10. 1.9 The complex case is more linear

11. 1.10 Other localisations

12. 1.11 Calculations and modularity

13. 1.12 Semistable case

14. 1.13 Kapustin-Witten equations on projective surfaces

15. 1.14 Notation

16. 2 Spectral construction

17. 2.1 Gieseker stability

18. 2.2 Resolution

19. 2.3 Deformation theory

20. 2.4 In families

21. 3 Perfect obstruction theory for $U(r)$ theory

22. 3.1 Atiyah classes

23. 4 The $U(r)$ Vafa-Witten invariant

24. 4.1 Localisation

25. 5 Deformation theory of the Higgs field

26. 5.1 Deformations of quotients

27. 5.2 Deformations of Higgs fields

28. 5.3 Matrices

29. 5.4 Higgs bundles

30. 5.5 Trace

31. 5.6 Determinant

32. 5.7 Higgs sheaves

33. 6 The $SU(r)$ Vafa-Witten invariant

34. 6.1 More general Higgs pairs

35. 7 Vanishing theorem and the two fixed loci

36. 7.1 The first fixed locus

37. 7.2 Vanishing theorem

38. 7.3 The second fixed locus

39. 8 Calculations on surfaces with positive canonical bundle

40. 8.1 Horizontal terms

41. 8.2 The virtual cycle

42. 8.3 The virtual normal bundle

43. 8.4 Tautological classes

44. 8.5 The answer in closed form

45. 8.6 Euler characteristics of nested Hilbert schemes

46. 8.7 Vertical terms

47. 8.8 One mixed term

48. 8.9 Comparison with Vafa-Witten prediction

1. Introduction

Fix a Riemannian 4-manifold $S$; from the next Section onwards it will be a complex projective surface (with a Kähler metric). Fix a compact Lie group $G$ and principal $G$-bundle $P\to S$. We let $\mathcal{A}_{P}$ denote the set of all $G$-connections on $P$, and $\Omega^{i}(\mathfrak{g}_{P})$ the $\mathbb{C}^{\infty}$ $i$-forms on $S$ with values in the adjoint bundle of $P$. Finally $\Omega^{+}\subset\Omega^{2}$ denotes the self-dual 2-forms with respect to the Riemannian metric.

1.1. The Vafa-Witten equations

The Vafa-Witten equations [VW] are

[TABLE]

Here $[B.B]$ is defined by Lie bracket on $\mathfrak{g}_{P}$ and the contraction $(T^{*})^{\otimes 2}\otimes(T^{*})^{\otimes 2}\to(T^{*})^{\otimes 2}$ given by the (inverse of the) metric on the second and third $T^{*}$ factors, followed by antisymmetrisation.

1.2. The equations on a Kähler surface

When $(S,\omega)$ is a Kähler surface, we can rewrite $B$ and $\Gamma$ in terms of a $\mathfrak{g}_{P}$-valued $(2,0)$-form $\phi\in\Omega^{2,0}(\mathfrak{g}_{P}\otimes\mathbb{C})$ and a $\mathfrak{g}_{P}$-valued multiple of $\omega$, resulting in

[TABLE]

Here $c$ is a topological constant. From now on we will restrict attention to the cases $G=U(r)$ or $G=SU(r)$, so that $P$ is the frame bundle of a hermitian vector bundle $E$. Then c=\pi i\int_{S}c_{1}(E)\wedge\omega\big{/}r\!\int_{S}\omega^{2}.

Thus, by the first equation, $\overline{\partial}_{A}$ defines an integrable holomorphic structure on $E$ with respect to which $\phi$ is holomorphic. Since the second equation is a moment map for the gauge group action, one can expect an infinite dimensional Kempf-Ness theorem that solutions (modulo $U(r)$ or $SU(r)$ gauge transformations) are equivalent to stable holomorphic pairs $(E,\phi)$ (modulo $GL(r,\mathbb{C})$ or $SL(r,\mathbb{C})$ gauge transformations) for an appropriate notion of stability.

1.3. Hitchin-Kobayashi correspondence

Such a Donaldson-Uhlenbeck-Yau-type result has been proved for Kähler surfaces $(S,\omega)$ in [AG], and for polarised surfaces $(S,\mathcal{O}_{S}(1))$ with integral Kähler form $h=c_{1}(\mathcal{O}_{S}(1))$ [Ta1]. We focus on the latter case. Solutions of the $U(r)$ Vafa-Witten equations are equivalent to slope polystable Higgs pairs

[TABLE]

while for $SU(r)$ we fix $\det E=\mathcal{O}_{S}$ and take $\phi$ to be trace-free. The stability condition is that

[TABLE]

for all proper $\phi$-invariant coherent subsheaves $F\subset E$. Replacing $<$ by $\leq$ defines semistability, while polystable pairs are direct sums of stable Higgs pairs of the same slope. The closely related but slightly more refined notion of Gieseker stability is described in Section 2.1.

Thus the moduli space of solutions of the Vafa-Witten equations has an obvious partial compactification given by taking Gieseker semistable (hence torsion-free) rank $r>0$ Higgs sheaves. In this paper “stability” always refers to Gieseker stability, and we always assume the Chern classes of $E$ are chosen so that semistability and stability coincide. In paper II [TT2] we tackle the semistable case.

1.4. Spectral construction

Finally, we can turn such pairs $(E,\phi)$ into compactly supported stable torsion sheaves $\mathcal{E}_{\phi}$ on the Calabi-Yau 3-fold $X$, where $X$ is the total space of the canonical line bundle $K_{S}$ of $S$.

Roughly speaking, over each point $x\in S$, we replace $(E_{x},\phi_{x})$ by the eigenspaces of \phi_{x}\in\operatorname{Hom}\big{(}E_{x},E_{x}\otimes(K_{S})_{x}\big{)} supported on their respective eigenvalues in $(K_{S})_{x}$. The result is an equivalence of categories

[TABLE]

see Section 2 for full details.

1.5. Localisation and a $U(r)$ Vafa-Witten invariant

After fixing Chern classes $(r,c_{1},c_{2})$ on $S$ for which semistability = stability, we let $\mathcal{N}$ denote the moduli space of stable Higgs sheaves $(E,\phi)$ on $S$, or, equivalently, compactly supported stable torsion sheaves $\mathcal{E}_{\phi}$ on $X$.

As a moduli space of sheaves on Calabi-Yau 3-fold, the results of [HT, Th] give a perfect obstruction theory on $\mathcal{N}$. Since $\mathcal{N}$ is noncompact, the resulting virtual cycle is uninteresting. However, it also carries a $\mathbb{C}^{*}$ action, given by scaling the Higgs field $\phi$, or, equivalently, the fibres of $X=K_{S}\to S$. The $\mathbb{C}^{*}$-fixed locus is compact, so we may apply virtual localisation [GP] to define a numerical invariant counting the sheaves $\mathcal{E}$. This is our preliminary $U(r)$ Vafa-Witten invariant

[TABLE]

described in Section 4. It is nothing but a local surface DT invariant of the Calabi-Yau 3-fold $X=K_{S}$.

1.6. \fortocVirtual technicalities and an $SU(r)$ VW invariant\excepttocVirtual technicalities and an $SU(r)$ Vafa-Witten invariant

However, as discussed in Remark 4.5, $\widetilde{\mathsf{VW}}$ (1.5) is zero unless both $H^{0,1}(S)=0=H^{0,2}(S)$. Most surfaces satisfying this condition satisfy a vanishing theorem, making the Vafa-Witten invariant just the (signed) topological Euler characteristic of the moduli space of stable sheaves on $S$. We would like to be able to handle more interesting cases.

The right fix is to consider the Vafa-Witten equations for the gauge group $SU(r)$ instead of $U(r)$. Thus we restrict to Higgs pairs $(E,\phi)$ where $E$ has fixed determinant111$SU(r)$ is a bit of a misnomer since for this definition we allow ourselves to fix $\det E=L$ for some nontrivial line bundle $L$ on $S$ (e.g. one whose degree is coprime to $\operatorname{rank}(E)$ so that semistability implies stability.) In paper II [TT2] we allow fixed trivial $\det E=\mathcal{O}_{S}$. and $\phi$ is trace-free. Equivalently, we work with torsion sheaves $\mathcal{E}$ on $X$ whose “centre of mass” on each fibre (i.e. the sum of its points of support in $K_{S}$, weighted by multiplicity) is zero, and whose pushdown to $S$ has fixed determinant.

Producing the right deformation theory for such things turned out to be unexpectedly complicated. At a single point $(E,\phi)\in\mathcal{N}$, the complication is roughly the following.

A natural resolution of the torsion sheaf $\mathcal{E}_{\phi}$ on $X\xrightarrow{\ \pi\ }S$ in terms of $\pi^{*}E$ and $\pi^{*}\phi$ (2.12) gives a long exact sequence

[TABLE]

relating the automorphisms, deformations and obstructions of $\mathcal{E}_{\phi}$ to those of $(E,\phi)$. The third arrow is easily seen to take deformations of $\mathcal{E}_{\phi}$ to the corresponding deformations of its pushdown $E=\pi_{*}\hskip 0.6pt\mathcal{E}_{\phi}$. What was surprisingly hard (for us) to show is that the second arrow is the one would expect — namely the derivative of the map from (an open set in) $\operatorname{Hom}(E,E\otimes K_{S})$ to $\mathcal{N}$ that takes $\phi$ to $(E,\phi)$.

Dealing with this issue (in full generality, over an arbitrary family to all orders at the level of perfect obstruction theories) is what takes up all of Section 5.

The result is that we can express our perfect obstruction theory for $\mathcal{N}$ in terms of Higgs data on $S$ instead of sheaves on $X$. That done, it allows us to easily fix $\det E\cong L$ and $\operatorname{tr}\phi=0$ to deduce a symmetric perfect obstruction theory for the moduli space222In Theorem 6.5 we extend this result from $K_{S}$-Higgs pairs to $\mathcal{L}$-Higgs pairs. Equivalently we work with torsion sheaves on the total space of $\mathcal{L}\to S$ in place of $K_{S}\to S$. The resulting 2-term perfect obstruction theory is not symmetric when $\mathcal{L}\neq K_{S}$, however.

[TABLE]

In fact, using the trace and identity maps on $S$ on the first and last terms of (1.6) produces a splitting

[TABLE]

This expresses the deformation-obstruction theory of $\mathcal{E}_{\phi}\in\mathcal{N}$ on the left hand side in terms of, respectively: the deformations and obstructions of $\operatorname{tr}\phi$, the deformations and obstructions of $\det E$, and the deformations and obstructions of $\mathcal{E}_{\phi}\in\mathcal{N}^{\perp}_{L}$ ; see Theorem 6.1.

Thus we get a better definition of the Vafa-Witten invariant by localising this new virtual cycle,

[TABLE]

If the Higgs field vanishes $\phi=0$ for all points $(E,\phi)\in(\mathcal{N}^{\perp}_{L})^{\mathbb{C}^{*}}$ then (1.8) is actually an integer (the virtual signed Euler characteristic of Section 7), but in general we work in the localised equivariant cohomology $H^{*}_{\mathbb{C}^{*}}(B\mathbb{C}^{*},\mathbb{Q})[t^{-1}]\cong\mathbb{Q}[t,t^{-1}]$ and get rational numbers. (The invariant is a constant, rather than a more general Laurent polynomial in the equivariant parameter $t$, because the virtual dimension of the problem is zero.)

$\mathsf{VW}_{r,c_{1},c_{2}}$ (1.8) is invariant under deformations of $L$, and deformations of $S$ which keep the class $c_{1}:=c_{1}(L)\in H^{2}(S)$ of Hodge type (1,1). Physics seems to predict that it ought to also be invariant under deformations of the polarisation $\mathcal{O}_{S}(1)$ (or more generally the stability condition defining $\mathcal{N}$) — i.e. the invariants’ wall crossing should be trivial. We intend to return to this point later using an extension of the wall crossing formula [KL2]. (Whereas the wall crossing formulae of Joyce-Song [JS] and Kontsevich-Soibelman [KS] use weighted Euler characteristics in a crucial way, Kiem-Li use virtual localisation.) When $L=\mathcal{O}_{S}$ the closely related invariant $\mathsf{vw}$ defined by Behrend localisation in paper II [TT2] does have trivial wall crossing.

The integral (1.8) is over the different components of the fixed locus of the $\mathbb{C}^{*}$ action scaling the Higgs field. They come in two flavours, which we name according to the conventions of [DPS, GK].

1. \tagform@1.9

The “instanton branch” $\phi=0$. Here we recover $\mathcal{M}_{\mathrm{asd}}$ or the moduli space $\mathcal{M}_{L}$ of stable sheaves of fixed determinant $L$ on $S$. By Proposition 7.4 the contribution of this locus to our invariant is the virtual signed Euler characteristic of $\mathcal{M}_{\mathrm{asd}}$ studied in [JT, GK], generalising (1.13). It is an integer. 2. \tagform@1.11

The “monopole branch” $\phi\neq 0$. We call the union of these components of the fixed locus $\mathcal{M}_{2}$. They correspond to $\mathbb{C}^{*}$-fixed sheaves $\mathcal{E}$ supported on scheme theoretic thickenings of the zero section $S\subset X$. When they have rank 1 on their support, they can be described in terms of nested Hilbert schemes; see [GSY1, GSY2] and Section 8. In general they correspond to flags of sheaves on $S$ as studied in the work of Negut (see [Ne], for instance). They contribute new rational numbers to the Vafa-Witten invariants.

1.7. Digression: derived algebraic geometry

The perfect obstruction theory for Higgs sheaves is of independent interest, but adds a great deal of complexity and length to the paper. It also requires us to work with Illusie’s full cotangent complex333Because the exact triangle relating the cotangent complexes of $\mathcal{N},\ \curly M$ and $\mathcal{N}/\curly M$ need not be exact after truncating. [Ill] instead of the truncated cotangent complex that normally suffices [HT].

Most readers should therefore ignore most of Sections 3, 4, 5 and skip straight to Section 6, taking on trust that the standard perfect obstruction theory [HT, Th] can be refined to the fixed trace and determinant case. But readers who are both derived and stacky will notice that a much quicker solution is to use the theory of derived algebraic stacks and the results of [STV, TVa].

That is, one shows that the stack $\curly M_{L}$ of rank $r$ torsion-free coherent sheaves $E$ on $S$ with fixed-determinant $\det E\cong L$ has a natural derived structure, inducing one on its $(-1)$-shifted cotangent bundle444At the level of points, this fibres over $\curly M_{L}$ with fibre over $E$ the dual of the obstruction space $\operatorname{Ext}^{2}(E,E)_{0}$ at that point. By the duality $\operatorname{Ext}^{2}(E,E)_{0}^{*}\cong\operatorname{Hom}(E,E\otimes K_{S})_{0}$ we indeed recover (1.3). $T^{*}[-1]\curly M_{L}$. This is the moduli stack of all Higgs pairs. The open substack of stable Higgs pairs is then the product of a derived scheme $\mathcal{N}_{L}^{\perp}$ and $T^{*}[-1]B\mathbb{C}^{*}$:

[TABLE]

Rigidifying (removing the $T^{*}[-1]B\mathbb{C}^{*}$ factor) gives a natural derived structure on $\mathcal{N}_{L}^{\perp}$ which is quasi-smooth. It therefore gives rise to a perfect obstruction theory on the underlying scheme. (This obstruction theory is also symmetric [Ca].) We are the wrong authors to use this technology honestly or competently, however, so we employ more classical (but lengthy!) techniques familiar to virtual cyclists like the second author.

1.8. Cotangent field theories

There is a symmetry between the domain and target of the VW equations (1.1). If we add a local Coulomb gauge fixing equation $d_{A_{0}}^{*}(A-A_{0})\in\Omega^{0}(\mathfrak{g}_{P})$ instead of dividing by gauge, and notice that $\mathcal{A}_{P}$ is an affine space modelled on $\Omega^{1}(\mathfrak{g}_{P})$, we see both sides contain the same pieces.

Roughly speaking, the equations come from the anti-self-dual (asd) equations $F_{A}^{+}=0$ by passing to their “$(-1)$-shifted cotangent bundle”, at least to first order in the new (cotangent) variables $B,\Gamma$.

The local model is the following. Start with a section $s$ of a vector bundle $E$ over an ambient space $A$ cutting out a moduli space of solutions $\mathcal{M}$:

[TABLE]

The setting could be real or holomorphic, finite or infinite dimensional. In the latter case it is called a balanced, co- or cotangent field theory in different references; see for example [CMR, DiM, Co, JT] and [VW, Section 2]. For the real VW equations555In fact their linearisation about $B=0=\Gamma$. we start with the asd equations, so the section is $F_{A}^{+}$ in the infinite dimensional bundle $\Omega^{+}(\mathfrak{g}_{P})$ over the infinite dimensional ambient space $\mathcal{A}_{P}$, plus the Coulomb gauge fixing equation (or we replace $\mathcal{A}_{P}$ by its quotient by gauge). In Section 1.9 we will work with a finite dimensional holomorphic version of this model, but in that case (we will see below) we will have to take $A$ to be a stack rather than a space.

Next pass to a bigger ambient space: the total space of $E^{*}\to A$ (in the VW setting this adds the variables $B$ and $\Gamma$). By pairing the section $s$ of $E$ with points of $E^{*}$ it defines a function $\tilde{s}$ on this ambient space $E^{*}$. Its gradient $d\tilde{s}$ is a section of the (co)tangent bundle of $E^{*}$.

[TABLE]

The zeros of $d\tilde{s}$ include the original zeros $s=0$ (inside the zero section $A$ of $E^{*}$ — i.e. it includes the asd moduli space where $F_{A}^{+}=0=B=\Gamma$), but in general there are more, giving the (dual) obstruction bundle on the asd moduli space (those $(B,\Gamma)$ perpendicular to the image of the linearised asd equation).

In “good” cases666There are different situations in which one can prove such a vanishing theorem, but they all involve $S$ having curvature which is positive in some sense. [VW, pp 23–25] the only solutions have $F_{A}^{+}=0=B=\Gamma$. In this case the original asd moduli space $\mathcal{M}_{\mathrm{asd}}$ is smooth with zero obstruction bundle. Thought of as the moduli space of solutions to the Vafa-Witten equations modulo gauge, however, it has obstruction bundle the (co)tangent bundle of $\mathcal{M}_{\mathrm{asd}}$, and so a natural associated integer invariant given by its Euler class

[TABLE]

When no vanishing result holds, it seems no one has yet managed to make a rigorous definition of the VW invariant on a general 4-manifold.

1.9. The complex case is more linear

Vafa and Witten’s equation is a nonlinear version of the above construction including also the quadratic term $[B.B]+[B,\Gamma]$. But in the projective surface case, the Hitchin-Kobayashi correspondence removes this nonlinearity at the expense of enlarging $\mathcal{M}_{\mathrm{asd}}$.

That is, if we let $\curly M$ denote the moduli stack of all bundles (or torsion-free sheaves) on $S$ — not just those which are stable, which form $\mathcal{M}_{\mathrm{asd}}$ — then the stable Higgs sheaves form a Zariski open in the $(-1)$-shifted cotangent bundle

[TABLE]

As a space (ignoring its derived structure or obstruction theory) $T^{*}[-1]\curly M=\operatorname{Spec}\,\operatorname{Sym}^{\bullet}\mathrm{Ob}$, where Ob is the obstruction sheaf of $\curly M$, so it can be thought of as the total space of the dual obstruction bundle over $\curly M$. Since the fibre of Ob over $E\in\curly M$ is $\operatorname{Ext}^{2}(E,E)$, Serre dual to $\operatorname{Hom}(E,E\otimes K_{S})$, we see $T^{*}[-1]\curly M\to\curly M$ is a fibration by Higgs fields $\phi\in\operatorname{Hom}(E,E\otimes K_{S})$.

Passing to the open locus which satisfies the stability condition (1.4) the stabiliser groups drop to $\mathbb{C}^{*}$. Rigidifying (removing them, and their $(-1)$-shifted cotangents in degree $-2$) gives the scheme $\mathcal{N}$ with a quasi-smooth derived structure. This induces the symmetric obstruction theory we used to define our invariant. Similarly our $SU(r)$ moduli space is an open set

[TABLE]

inducing the correct symmetric obstruction theory, again after rigidifying.

This enlargement of $\mathcal{M}_{\mathrm{asd}}$ to $\curly M_{L}$ is what results in there now being two types of fixed locus for the additional $\mathbb{C}^{*}$ action that scales the Higgs field $\phi$. The first (\tagform@1.9) is $\mathcal{M}_{\mathrm{asd}}$ but the second — the monopole branch $\mathcal{M}_{2}$ (\tagform@1.11) with $\phi\neq 0$ — need not lie in $T^{*}[-1]\mathcal{M}_{\mathrm{asd}}$.

1.10. Other localisations

There are three other natural ways to localise the virtual cycle of $\mathcal{N}$ to $\mathcal{N}^{\mathbb{C}^{*}}$, thus giving competing definitions of the VW invariant.

Let $\mathcal{N}$ be a scheme with a symmetric perfect obstruction theory. Behrend [Be] defines a constructible function

[TABLE]

such that if $\mathcal{N}$ is compact then the degree of its (zero dimensional) virtual cycle equals the Euler characteristic of $\mathcal{N}$ weighted by $\chi^{B}$,

[TABLE]

Since our moduli spaces of Higgs pairs $\mathcal{N}$ and $\mathcal{N}_{L}^{\perp}$ are noncompact, we instead take the right hand side as a definition. Of course $\chi^{B}$ is $\mathbb{C}^{*}$-invariant and the Euler characteristic of any non-fixed orbit is 0, so only the fixed points contribute. This gives the localisation

[TABLE]

When $h^{0,1}(S)>0$ this vanishes (since $e(\operatorname{Jac}(S))=0$) so we fix determinant and trace and localise on $\mathcal{N}_{L}^{\perp}$ instead. Thus we consider the Kai-localised $SU(r)$ Vafa-Witten invariant given by777At least when there are no strictly semistables. More generally we use the Joyce-Song formalism and consider invariants in $\mathbb{Q}$ counting semistables [TT2].

[TABLE]

When $h^{0,1}(S)=0$ we recover (1.14).

Similarly one can also use Kiem-Li’s cosection localisation [KL1]. Differentiating the $\mathbb{C}^{*}$ action on $\mathcal{N}$ and $\mathcal{N}_{L}^{\perp}$ defines vector fields thereon. Since these have symmetric obstruction theories, their obstruction sheaves are their cotangent sheaves. Pairing with the vector field gives cosections

[TABLE]

Their zero loci are the $\mathbb{C}^{*}$-fixed loci, so Kiem-Li’s construction gives localised cycles

[TABLE]

When $h^{0,1}(S)=0$ they coincide, otherwise the first vanishes. This suggests a third VW invariant

[TABLE]

Thirdly, we could take the signed topological Euler characteristic of $(\mathcal{N}^{\perp}_{L})^{\mathbb{C}^{*}}$. In fact these three rival definitions are closely related. By [Ji, JT] the first two are equal,

[TABLE]

but are different in general from the invariant $\mathsf{VW}$ defined by virtual localisation. Furthermore, when a “vanishing theorem” holds so that all $\mathbb{C}^{*}$-fixed stable Higgs pairs have $\phi=0$ (equivalently $\mathbb{C}^{*}$-fixed stable torsion sheaves on $X$ are all pushed forward from $S$) they also equal the third definition by “dimension reduction” [BBS, Da], [JT, Section 5]:

[TABLE]

While $\mathsf{vw}$ is not in general deformation invariant, it does have advantages over $\mathsf{VW}$: it is often more easily computable, for instance by cut-and-paste methods, and it admits a natural refinement and categorification (as expected in physics). While we expect that $\mathsf{vw}=\mathsf{VW}$ for surfaces with $K_{S}\leq 0$ — we prove this for $\deg K_{S}<0$ in [TT2] and it is proved for $S$ a K3 surface in [MT] — in general they differ. In particular when stability = semistability $\mathsf{vw}$ is an integer but $\mathsf{VW}$ is only rational in general.

So we need to chose between the two invariants somehow. We do this by seeing how they stand up to the modularity predictions of physics.

1.11. Calculations and modularity

Vafa-Witten predicted that for fixed rank $r$ and determinant $L$, S-duality should force the generating function

[TABLE]

to be, roughly speaking, a weight $w/2=-e(S)/2$ modular form, for some appropriate shift $s$. (The precise functional equations are described in [VW, Equation 5.39] in rank 2, and [LL, Equation 5.22] in prime rank.)

There have been many compelling calculations confirming these predictions on the instanton branch $\mathcal{M}_{\rm asd}$ (\tagform@1.9), usually in the presence of a vanishing theorem (for instance when $K_{S}\leq 0$) ensuring that it is both smooth and the monopole branch $\mathcal{M}_{2}$ (\tagform@1.11) is empty. Recent work of Göttsche and Kool [GK] extends these calculations to the much more difficult setting of general type surfaces $S$.

However, the monopole branch (\tagform@1.11) with nonzero Higgs field has been largely ignored until now. In Section 8 we compute the part of the generating series of Vafa-Witten invariants $\mathsf{VW}$ (1.8) coming from a series of components with nonzero Higgs field. With some tricks we manage to sum the series in closed form in Proposition 8.22. The result is far from modular — in fact it is an algebraic function of $q$,

[TABLE]

where $c=(-1)^{p_{g}(S)+g}\cdot 2^{-p_{g}(S)-1}$ and $g-1=c_{1}(S)^{2}$.

In Section 8.6 we repeat the calculation for the bare Euler characteristic version of $\mathsf{vw}$ (1.15) (this is reasonable since we expect the Behrend function to be 1 on $\mathcal{M}_{2}$). We find something which is modular form up to a factor of $(1-q)^{e(S)}$. And calculations with semistable sheaves on K3 surfaces in [TT2, Section 5] also give modular answers for $\mathsf{vw}$. So for some time this misled us into believing that $\mathsf{vw}$ is the correct invariant.

However, Martijn Kool pointed out that the coefficients of $q^{0}$ and $q^{1}$ in (1.16) match those in a modular form in [VW, Equation 5.38]. So in the rest of Section 8 we compute the contributions of the remaining components of the monopole branch $\mathcal{M}_{2}$ (\tagform@1.11) to $\mathsf{VW}$ for $c_{2}=2,3$. When added to (1.16) we get agreement also in $q^{2}$ and $q^{3}$. That is, we find in (8.39) that

[TABLE]

The right hand side is the (modular) second term in [VW, Equation 5.38].888Up to an overall sign $(-1)^{\operatorname{vd}}$ which arises from our orientation conventions. Vafa and Witten identify the tangent and cotangent bundles of moduli space using a Riemannian metric (which is natural from the real point of view), but this changes the natural complex orientation that we use.

This agreement may not sound like much, but given the complexity of the calculations across 15 pages, we feel it cannot be a coincidence. The upshot is that we now believe the virtual localisation invariant $\mathsf{VW}$ of this paper to be the “correct” one for physics.999At least in the stable case; for semistable Higgs pairs see [TT2]. Remarkably, it seems that the generating series of invariants coming from the two types of fixed component (\tagform@1.9, \tagform@1.11) can both be written separately in terms of modular-like functions, but the contributions of natural series of subcomponents of $\mathcal{M}_{2}$ (\tagform@1.11) cannot.

Even more remarkably, Vafa and Witten predicted the above formula without calculating on the nested Hilbert schemes making up $\mathcal{M}_{2}$ (\tagform@1.11). It seems our geometric techniques are still a lot less powerful then the physics of the early 1990s. The Kiem-Li cosection localisation we use in Section 8.2 may be a primitive analogue of their perturbation [VW, Equation 5.19] which they use to localise their calculation to “cosmic string” contributions. Our results also seem to indicate that the Gieseker compactification of moduli of sheaves is relevant to physics (which usually uses the Uhlenbeck compactification).

In Section 8.6 we discuss the modularity properties of the the alternative definition $\mathsf{vw}$ (1.15). These seem to be the wrong ones for physics (except when $K_{S}\leq 0$). In particular the modular forms in Section 8.6 depend on $S$ only through its Euler characteristic $c_{2}(S)$, whereas Vafa-Witten also predict a dependence on $c_{1}(S)^{2}$ when $S$ has general type.

1.12. Semistable case

We give a brief overview of the companion paper [TT2]. Since the work of Joyce-Song and Kontsevich-Soibelman uses the Behrend function and weighted Euler characteristics, we can use it — with some modifications when $h^{1}(\mathcal{O}_{S})>0$ — to generalise the invariant $\mathsf{vw}$ to the semistable case.

What came as more of a surprise to us was that modifying the Joycian formalism in a different way — using virtual localisation in place of weighted Euler characteristics, and replacing Joyce-Song’s universal formulae by its logarithm — seems to work similarly well to give an extension of $\mathsf{VW}$ to the semistable case.

That is, motivated by Mochizuki’s work [Mo] and Joyce-Song pairs [JS] we virtually enumerate certain stable pairs

[TABLE]

on $X=K_{S}$ (or equivalently triples $(E,\phi,s)$ on $S$.) Here $\mathcal{E}_{\phi}$ is a semistable torsion sheaf with centre of mass zero on the $K_{S}$ fibres (equivalently $\operatorname{tr}\phi=0$) and $\det\pi_{*\hskip 0.6pt}\mathcal{E}_{\phi}=\det E\cong\mathcal{O}_{S}$. Finally

[TABLE]

is any section which does not factor through a destabilising subsheaf of $\mathcal{E}_{\phi}$. Here $N\gg 0$ is fixed and sufficiently large that $H^{\geq 1}(\mathcal{E}_{\phi}(N))=0$ for all semistable sheaves $\mathcal{E}_{\phi}$ with the same Chern classes.

There is a symmetric obstruction theory for such pairs, given by combining the $R\operatorname{Hom}_{\perp}$ perfect obstruction theory of (1.7) with Joyce-Song’s pairs theory. Localising the virtual cycle to the fixed points of the usual $\mathbb{C}^{*}$ action defines invariants. We conjecture these can be written in terms of universal formulae in $N$; the coefficients of these formulae then define the Vafa-Witten invariants.

We show the conjecture holds in some cases, for instance when stability and semistability coincide: in this case the resulting invariants $\mathsf{VW}$ coincide with the definition (1.8) of this paper. The conjecture also holds for $\deg K_{S}<0$, and for K3 surfaces. In the latter case the invariants $\mathsf{VW}$ recover (and generalise) modular forms conjectured by Vafa and Witten.

1.13. Kapustin-Witten equations on projective surfaces

It would be natural to try to repeat the same trick for the Kapustin-Witten equations [KW] on a projective surface $S$. On any Kähler surface the equations reduce [Ta2] to Simpson’s equations, for which there is a Hitchin-Kobayashi correspondence [Si]. Via the spectral construction, solutions can be identified with torsion sheaves on the Calabi-Yau 4-fold

[TABLE]

Borisov-Joyce [BJ] describe a real analogue of perfect obstruction theory and virtual cycle for sheaves on Calabi-Yau 4-folds $X$, and Cao-Leung [CL] conjecture a way to extend virtual localisation to this setting. We would like to apply this to the natural $\mathbb{C}^{*}$ action on the fibres of $X\to S$ to define Kapustin-Witten invariants for $S$. But the $\mathbb{C}^{*}$ action does not preserve the Calabi-Yau form on $X$, so the theory does not immediately apply.

Acknowledgements. We are most grateful to Martijn Kool for the crucial observation that — despite not appearing to give modular forms — the first few terms of our calculation (8.24) match some formulae in [VW]. Special thanks to Amin Gholampour for pointing out various embarrassing mistakes before anyone else saw them, and a rapid and thorough referee for catching the rest. We also thank Kevin Buzzard, Julien Grivaux, Hiroshi Iritani, Dominic Joyce, Kotaro Kawatani, Davesh Maulik, Rahul Pandharipande, Mauro Porta, Artan Sheshmani, Balázs Szendrői, Yukinobu Toda and Edward Witten for useful conversations.

Y.T. was partially supported by JSPS Grant-in-Aid for Scientific Research numbers JP15H02054 and JP16K05125, and a Simons Collaboration Grant on “Special holonomy in Geometry, Analysis and Physics”. He thanks Seoul National University, NCTS at National Taiwan University, Kyoto University and BICMR at Peking University for support and hospitality during visits in 2015–17 where part of this work was done.

1.14. Notation

To keep the notation readable we suppress some obvious pullback maps, and, given a map, we often use the same notation for any basechange thereof. So for instance $\pi$ will denote the standard projection $X\to S$ of (2.1), but also the induced map $\mathcal{N}\times X\to\mathcal{N}\times S$ (which should properly be called $\operatorname{id}_{\mathcal{N}}\times\pi$). Therefore, when we let $p_{S}$ denote the projection $S\to\operatorname{Spec}\,\mathbb{C}$, it also denotes the projections $\mathcal{N}\times S\to\mathcal{N}$ and $\curly M\times S\to\curly M$.

Where possible we stick to the notation of

[TABLE]

for a Higgs pair on $S$ and the associated torsion sheaf on $X=K_{S}$ respectively. In families these get replaced by

[TABLE]

respectively. For instance these could denote the universal Higgs pair over $\mathcal{N}\times S$ and the universal torsion sheaf over $\mathcal{N}\times X$ respectively.

Stability always means Gieseker stability, and in this paper we always assume semistability implies stability. We reserve $\mathcal{M}$ for moduli spaces of stable sheaves on $S$, while $\curly M$ denotes the moduli stack of all coherent sheaves on $S$. Similarly $\mathcal{N}$ is used for moduli spaces of stable Higgs pairs on $S$ or, equivalently, compactly supported stable torsion sheaves on $X=K_{S}$.

Given a map $\pi$, we denote by $\curly H\!om_{\pi}$ the functor $\pi\circ\curly H\!om$ of Homs down the fibres of $\pi$, and by $R\curly H\!om_{\pi}\cong R\pi_{*}R\curly H\!om$ its derived functor.

For a sheaf or complex of sheaves $\mathcal{F}$, we use $\mathcal{F}^{\vee}=R\curly H\!om(\mathcal{F},\mathcal{O})$ to denote its derived dual. For a single sheaf, $\mathcal{F}^{*}$ denotes its underived dual $h^{0}(\mathcal{F}^{\vee})$.

We use $\mathbb{L}$ to denote Illusie’s cotangent complex, and $\mathbb{T}:=\mathbb{L}^{\vee}$ for its dual, the tangent complex.

2. Spectral construction

Fix a complex projective algebraic surface $S$ and let

[TABLE]

be the total space of a line bundle $\mathcal{L}\to S$. (We will mainly be interested in the case $\mathcal{L}=K_{S}$.) We recall the classical spectral construction relating $\mathcal{L}$-Higgs pairs $(E,\phi)$ on $S$ to compactly supported sheaves $\mathcal{E}_{\phi}$ on $X$.

Roughly speaking, over each point $x\in S$, we replace $(E_{x},\phi_{x})$ by the eigenspaces of $\phi_{x}\in\operatorname{Hom}(E_{x},E_{x}\otimes\mathcal{L}_{x})$ supported on their respective eigenvalues in $\mathcal{L}_{x}$. More precisely we get an equivalence of categories as follows.

Proposition 2.2**.**

There is an abelian category $\mathrm{Higgs}_{\mathcal{L}}(S)$ of $\mathcal{L}$-Higgs pairs $(E,\phi)$ on $S$, and an equivalence of categories

[TABLE]

with the category of compactly supported coherent sheaves on $X$.

Proof.

The maps in Higgs${}_{\mathcal{L}}(S)$ between $(E,\phi)$ and $(F,\psi)$ are maps $f\colon E\to F$ inducing a commutative diagram

[TABLE]

Taking kernels and cokernels of the columns defines kernel and cokernel Higgs pairs.

It is a classical result of Serre (see for instance [Ha, Exercise II.5.17]) that for affine maps $\pi\colon X\to S$, the functor $\pi_{*}$ is an equivalence between the abelian category of coherent $\mathcal{O}_{X}$-modules and the abelian category of $\pi_{*}\hskip 0.6pt\mathcal{O}_{X}$-modules on $S$. In our case

[TABLE]

where $\eta$ is the tautological section of $\pi^{*}\mathcal{L}$ which is linear on the fibres and cuts out the zero section $S\subset X$. The affine $X/S$ is Spec of this sheaf of $\mathcal{O}_{S}$-algebras, and sheaves $\mathcal{E}$ on $X$ are equivalent to sheaves of modules $\pi_{*}\hskip 0.6pt\mathcal{E}$ over $\pi_{*}\hskip 0.6pt\mathcal{O}_{X}$ (2.5).

But (2.5) is generated by $\mathcal{O}_{S}$ and $\mathcal{L}^{-1}\cdot\eta$, so a module over it is equivalent to an $\mathcal{O}_{S}$-module $E$ together with a commuting action of $\mathcal{L}^{-1}\cdot\eta$, i.e. an $\mathcal{O}_{S}$-linear map

[TABLE]

Thus we get an $\mathcal{L}$-Higgs pair

[TABLE]

Conversely, given a Higgs pair $(E,\phi)$ we get an action of $\mathcal{L}^{-i}\cdot\eta^{i}$ by

[TABLE]

Summing over $i$ gives an action of $\pi_{*}\hskip 0.6pt\mathcal{O}_{X}$ (2.5) on $E$. We denote by $\mathcal{E}_{\phi}$ the sheaf on $X$ that this $\pi_{*}\hskip 0.6pt\mathcal{O}_{X}$-module defines.

This equivalence is of course a functor: morphisms of sheaves $\mathcal{E}$ induce morphisms of their pushdowns $E=\pi_{*}\hskip 0.6pt\mathcal{E}$ which commute with the action of $\eta$, therefore giving commutative diagrams (2.4) and so morphisms in Higgs${}_{\mathcal{L}}(S)$.

Finally, for $\mathcal{E}$ coherent, the quasicoherent sheaf $\pi_{*}\hskip 0.6pt\mathcal{E}$ is coherent if and only if $\pi|_{\mathrm{supp}\,\mathcal{E}}$ is proper, if and only if $\mathcal{E}$ is compactly supported. ∎

2.1. Gieseker stability

The $\mathcal{L}$-Higgs pair $(E,\phi)$ is said to be Gieseker stable with respect to an ample line bundle $\mathcal{O}_{S}(1)$ if and only if

[TABLE]

for every proper $\phi$-invariant subsheaf $F\subset E$ on $S$. Here $\chi$ denotes holomorphic Euler characteristic. Replacing $<$ by $\leq$ defines semistability.

Gieseker stability implies slope semistability of $(E,\phi)$, and is implied by slope stability. Recall (1.4) this is defined by the inequality

[TABLE]

When the denominator and numerator of the right hand side (the rank and degree of $E$) are coprime, we further find that

slope semistable = Gieseker semistable = Gieseker stable = slope stable.

Lemma 2.9**.**

Under the equivalence of categories (2.3), Gieseker (semi)stability of the $\mathcal{L}$-Higgs pair $(E,\phi)$ with respect to $\mathcal{O}_{S}(1)$ is equivalent to Gieseker stability of the sheaf $\mathcal{E}_{\phi}$ with respect to $\pi^{*}\mathcal{O}_{S}(1)$.

Proof.

Under the equivalence, $\phi$-invariant subsheaves $F\subset E$ are equivalent to subsheaves $\mathcal{F}\subset\mathcal{E}_{\phi}$ on $X$. Moreover $\chi(E)=\chi(\pi_{*}\hskip 0.6pt\mathcal{E}_{\phi})=\chi(\mathcal{E}_{\phi})$, so Gieseker stability (2.8) of $(E,\phi)$ is equivalent to

[TABLE]

for all proper subsheaves $\mathcal{F}\subset\mathcal{E}_{\phi}$. Here we have suppressed the pullback map $\pi^{*}$ on $\mathcal{O}(1)$, and denoted by $r(\mathcal{E}_{\phi}):=\int_{X}c_{1}(\mathcal{E}_{\phi})\cdot h^{2}=\operatorname{rank}(E)\int_{S}h^{2}$ the leading coefficient of its Hilbert polynomial $\chi(\mathcal{E}_{\phi}(n))$.

But (2.10) is the usual Gieseker stability for the torsion sheaf $\mathcal{E}_{\phi}$ on $X$, so these two notions of Gieseker stability for $(E,\phi)$ and $\mathcal{E}_{\phi}$ coincide. ∎

2.2. Resolution

Since $\mathcal{E}_{\phi}$ is generated by its sections down $\pi$ we have a natural surjection $\pi^{*}E=\pi^{*}\pi_{*}\hskip 0.6pt\mathcal{E}\xrightarrow{\ \operatorname{ev}\ }\mathcal{E}_{\phi}$. Its kernel is given by the following.101010The result says that if we divide $\pi^{*}E$ by the minimal submodule to ensure that $\eta$ acts as $\pi^{*}\phi$ on the quotient, we get $\mathcal{E}_{\phi}$.

Proposition 2.11**.**

There is an exact sequence

[TABLE]

Proof.

Consider $X\times_{S}X$ — the total space of $\mathcal{L}^{\oplus 2}$ over $S$ — with its two natural projections $\pi_{1},\pi_{2}$ to $X$ and projection $p$ to $S$. On it is an obvious exact sequence

[TABLE]

where $\Delta\cong X\subset X\times_{S}X$ is the diagonal divisor.

Now $p^{*}\mathcal{L}$ has two tautological sections $\eta_{1},\eta_{2}$ whose difference $\eta_{1}-\eta_{2}$ cuts out the diagonal. Thus $\mathcal{O}(\Delta)\cong p^{*}\mathcal{L}\cong\pi_{2}^{*}\pi^{*}\mathcal{L}$ and we can rewrite (2.13) as

[TABLE]

Now apply $\pi_{2*}$, using the projection formula and the flat basechange formula $\pi_{2*}\pi_{1}^{*}=\pi^{*}\pi_{*}$. By (2.6) we get (2.12). ∎

2.3. Deformation theory

This resolution (2.12) of $\mathcal{E}$ is useful to relate the deformation theory of $\mathcal{E}$, governed by

[TABLE]

with that of the Higgs pair $(E,\phi)$, governed by the cohomology groups of the total complex of

[TABLE]

Proposition 2.14**.**

There is an exact triangle

[TABLE]

Proof.

Applying $R\curly H\!om(\ \cdot\ ,\mathcal{E})$ to (2.12) gives the exact triangle

[TABLE]

Applying $R\pi_{*}$ and adjunction gives

[TABLE]

Since $\pi_{*}\eta\colon\pi_{*}\hskip 0.6pt\mathcal{E}\to\pi_{*}\hskip 0.6pt\mathcal{E}\otimes\mathcal{L}$ is $\phi\colon E\to E\otimes\mathcal{L}$ by (2.6) this is

[TABLE]

Taking $R\Gamma_{\!S}$ gives (2.15). ∎

In particular, taking cohomology in (2.15) gives the long exact sequence (1.6) of the introduction.

2.4. In families

We can generalise to families of sheaves over $S$ and $X$, and, moreover, we can let $S$ and $X$ themselves vary smoothly (to prove deformation invariance of our invariants). So we now let $S\to B$ be a smooth projective morphism with 2-dimensional fibres, and we let $X\to B$ denote the total space of a line bundle $\mathcal{L}$ (soon to be the relative canonical bundle $K_{S/B}$) over $S$. On a first viewing, the reader should set $B=\operatorname{Spec}\,\mathbb{C}$ and forget all about it.

Let $\mathcal{N}\to B$ denote the moduli space of Gieseker stable Higgs pairs on the fibres of $S\to B$ with fixed rank111111In this paper we always take rank $r>0$. When $r=0$ the obstruction theory (1.7) has to be modified. and Chern classes $(r,c_{1},c_{2})$. Equivalently it is a moduli space of compactly supported stable torsion sheaves on the fibres of $X\to B$, with rank 0 and

[TABLE]

in $H^{*}_{c}(X_{t},\mathbb{Z})$. Here $\iota\colon S\hookrightarrow X$ is the zero section, $[S]$ its Poincaré dual, and $X_{t}$ is the fibre of $X\to B$ over any closed point $t\in B$.

Pick a (twisted)121212Although a global universal sheaf may not exist in general due to $\mathbb{C}^{*}$-automorphisms, one can work locally over $\mathcal{N}$ (where one always exists) or globally with a twisted universal sheaf. We can ignore this issue since we are only concerned with (derived functors of) $\curly H\!om(\mathcal{E},\mathcal{E})$, which exists globally and glues uniquely and independently of any choices. universal sheaf

[TABLE]

As usual we let $\pi$ denote both the projection $X\to S$ of (2.1) and (cf. Notation section) any basechange such as $\mathcal{N}\times_{B}X\to\mathcal{N}\times_{B}S$. Since $\curly E$ is flat over $\mathcal{N}$ and $\pi$ is affine,

[TABLE]

is also flat over $\mathcal{N}$. It is also coherent because $\curly E$ is finite over $\mathcal{N}\times_{B}S$. Thus it defines a classifying map

[TABLE]

to the moduli stack $\curly M$ of coherent sheaves on the fibres of $S\to B$ of the given rank and Chern classes. We will often not distinguish between the universal sheaf $\mathsf{E}$ over $\curly M\times S$ and its pullback $\mathsf{E}=\Pi^{*}\mathsf{E}$ over $\mathcal{N}\times S$.

By the same working as above, over the families

[TABLE]

instead of on single copies of $X$ and $S$, we replace (2.15) by the exact triangle

[TABLE]

Presently we will see how this relates the obstruction theories of $\mathcal{N}$, $\curly M$ and $\Pi\colon\mathcal{N}\to\curly M$ respectively. From now on, for simplicity we restrict attention to the case of interest $\mathcal{L}=K_{S/B}$. (Later in Section 6.1 we will sketch what small changes have to be made in the general case.) We always work relative to the base $B$, but often suppress mention of it.

Proposition 2.21**.**

Setting $\mathcal{L}=K_{S/B}$, the exact triangle (2.20) is its own Serre dual.

Proof.

The claim is that replacing each term in (2.20) with its relative Serre dual (down the $X$- and $S$-fibres respectively), and the arrows by their duals,

[TABLE]

we get the same exact triangle, just shifted.

For the right hand arrows, this is simple. We concentrate on the left hand arrows. The claim is that the following horizontal arrows

[TABLE]

intertwine the vertical pairings given by cup product and trace. Here $\partial$ is the coboundary morphism of the exact triangle (2.20). That is, identifying

[TABLE]

it is cup product with the canonical extension class

[TABLE]

of the resolution (2.12). So we want to show that the two parings

[TABLE]

are equal. Since $\operatorname{tr}(ab)=\operatorname{tr}(ba)$ the upper map may be rewritten

[TABLE]

when we identify the product of the first two terms in (2.24) as lying in

[TABLE]

(since the dualising complex of $\pi$ is $\pi^{*}K_{S/B}^{-1}[1]$) and the last in

[TABLE]

The identification (2.25) is isomorphic to the composition

[TABLE]

where $\epsilon\colon\pi_{*}\pi^{!}\to\operatorname{id}$ is the counit. Therefore the product of the first two terms of (2.24) is

[TABLE]

But $\epsilon\circ\pi_{*}(e)$ is the image of $e$ under the identification

[TABLE]

which is $\operatorname{id}_{E}$. Thus (2.24) is \operatorname{tr}_{S}\big{(}\pi_{*}(\ \cdot\ )\circ(\ \cdot\ )\big{)}, as required. ∎

Corollary 2.26**.**

The exact triangle (2.20) fits into the following commutative diagram of exact triangles with split columns

[TABLE]

where the suffix 0 denotes trace-free Homs. Letting $R\curly H\!om_{p_{X}}(\curly E,\curly E)_{\perp}$ denote the co-cone of the central column, it is Serre dual to its own shift by $[3]$.

Proof.

The left and right hand columns are split because $\operatorname{tr}\circ\operatorname{id}=\operatorname{rank}\neq 0$.

The composition

[TABLE]

is $\operatorname{rank}(E)\cdot\operatorname{id}$, so the first arrow is split. Apply $Rp_{S*}$ and use $Rp_{S*}R\pi_{*}=Rp_{X*}$ to give the maps

[TABLE]

lifting the corresponding maps on $R\curly H\!om_{p_{S}}(\mathsf{E},\mathsf{E})$.

The Serre duals of the maps (2.28) give the maps

[TABLE]

By Proposition 2.21 these also commute with the Serre duals of the identity and trace maps on $R\curly H\!om_{p_{S}}(\mathsf{E},\mathsf{E}\otimes K_{S/B})[-1]$. Since these are the trace and identity maps respectively, the result follows. ∎

This co-cone $R\curly H\!om_{p_{X}}(\curly E,\curly E)_{\perp}$ will eventually provide the symmetric perfect obstruction theory we need for the moduli space $\mathcal{N}_{L}^{\perp}$ of stable trace-free fixed-determinant Higgs pairs; see Theorem 6.1.

3. Perfect obstruction theory for $U(r)$ theory

We will describe the perfect obstruction theories of $\mathcal{N}$ and $\curly M$ and, for later use, their relationship under $\Pi$ (2.19). The exact triangle of cotangent complexes

[TABLE]

may fail to be exact if we use truncated cotangent complexes as in [HT]. Therefore we use full cotangent complexes, to which [HT] does not immediately apply. However, since we are deforming sheaves ($\mathcal{E}$ and $E$) rather than complexes of sheaves, we can use Illusie’s seminal work [Ill].

3.1. Atiyah classes

Illusie [Ill, Section III.2.3] defines the Atiyah class of a coherent sheaf $\mathcal{F}$ on a $B$-scheme $Z$ as follows. Using the notation of Gillam [Gi] we let

[TABLE]

be the trivial square-zero thickening of $Z$ by $\mathcal{F}$. Here $\mathcal{O}_{Z}\oplus\mathcal{F}$ is an $\mathcal{O}_{Z}$-algebra in the obvious way: $(f,s)\cdot(g,t):=(fg,ft+gs)$. It carries a $\mathbb{C}^{*}$ action, fixing $\mathcal{O}_{Z}$ and acting on $\mathcal{F}$ with weight 1. We get the following exact triangle of $\mathbb{C}^{*}$-equivariant cotangent complexes on $Z[\mathcal{F}]$,

[TABLE]

Applying $q_{*}$ and taking weight 1 parts is an exact functor. Applied to the last two terms of (3.1) it yields

[TABLE]

on $Z$. We call this map $\operatorname{At}_{\mathcal{F}}\in\operatorname{Ext}^{1}(\mathcal{F},\mathcal{F}\otimes\mathbb{L}_{Z/B})$ the Atiyah class of $\mathcal{F}$.

We apply this to the universal sheaf $\curly E$ on $\mathcal{N}\times_{B}X$ and project to the first summand of $\mathbb{L}_{\mathcal{N}\times_{B}X/B}=\mathbb{L}_{\mathcal{N}/B}\oplus\mathbb{L}_{X/B}$ (suppressing some pullback maps) to give the partial Atiyah class

[TABLE]

Similarly applied to $\mathsf{E}$ on $\mathcal{N}\times_{B}S$ and projecting $\mathbb{L}_{\mathcal{N}\times_{B}S}$ to $\mathbb{L}_{\mathcal{N}/B}$ defines a partial Atiyah class

[TABLE]

The relationship between these two classes turns out to be very simple, and will allow us to relate the deformation theories of $\mathcal{E}$ and $E$.

Proposition 3.5**.**

Applying $\pi_{*}$ to (3.3),

[TABLE]

gives (3.4). That is, $\pi_{*\!}\operatorname{At}_{\curly E\!,\,\mathcal{N}}=\operatorname{At}_{\mathsf{E},\,\mathcal{N}\,}$.

Proof.

We use the natural $\mathbb{C}^{*}$-equivariant131313Here we are using the $\mathbb{C}^{*}$ action which acts with weight 1 on $\curly E$ and $\mathsf{E}$ but fixes $X$ and $S$. Only later will we use the different $\mathbb{C}^{*}$ action on the fibres of $\pi\colon X\to S$. map

[TABLE]

defined by thinking of both as affine schemes over $\mathcal{N}\times_{B}S$ and using the pullback map on (sheaves of $\mathcal{O}_{\mathcal{N}\times_{B}S}$-) algebras:

[TABLE]

(The algebra structure on the left hand side is defined by the action (2.7) of each $K_{S/B}^{-i}$ on $\mathsf{E}$.) Combining the last two terms of (3.1) with pullback via $\rho$ (3.6) gives the commutative diagram

[TABLE]

Taking weight 1 parts gives

[TABLE]

Projecting to the $\mathbb{L}_{\mathcal{N}/B}$ factor in both terms on the right hand side gives the commutative diagram

[TABLE]

Apply the adjunction isomorphism

[TABLE]

to $a=\operatorname{At}_{\mathsf{E},\,\mathcal{N}}$ and $b=\operatorname{At}_{\curly E\!,\,\mathcal{N}}$. The commutativity of (3.7) says that the two maps on the left hand side of (3.8) agree. Therefore the two maps on the right hand side agree. That is, $a=\pi_{*\hskip 0.6pt}b$ as required. ∎

Consider the partial Atiyah class (3.3) as lying in the group

[TABLE]

using relative Serre duality for the projection $p_{X}\colon\mathcal{N}\times_{B}X\to\mathcal{N}$ of relative dimension 3. Of course the relative canonical bundle $K_{X/B}\cong\mathcal{O}$ is trivial, but not equivariantly with respect to the $\mathbb{C}^{*}$ action scaling the fibres of $X\to S$. This acts on $K_{X/B}$ with weight $-1$, so

[TABLE]

where $\mathfrak{t}$ is the standard 1-dimensional representation of $\mathbb{C}^{*}$. All told we get a map

[TABLE]

Using $\tau^{\leq 0}\hskip 0.6pt\mathbb{L}_{\mathcal{N}/B}=\mathbb{L}_{\mathcal{N}/B}$, where $\tau$ is the truncation functor, this gives

[TABLE]

Here we have removed $\tau^{\geq 3}R\curly H\!om_{p_{X}}(\curly E,\curly E)$ from $R\curly H\!om_{p_{X}}(\curly E,\curly E)$. By Serre duality and the fact that the sheaves $\curly E_{t}$ are simple (by stability), we have the isomorphisms

[TABLE]

for all closed points $t\in\mathcal{N}$. (Serre duality holds despite the noncompactness of $X$ because supp $\curly E_{t}$ is proper. Alternatively, push forward to the projective completion $\overline{X}=\mathbb{P}(\mathcal{O}_{S}\oplus K_{S/B})$, which leaves the Ext groups unchanged. Then ordinary projective Serre duality on $\overline{X}$ gives $\operatorname{Ext}^{3}(\curly E_{t},\curly E_{t})=\operatorname{Hom}(\curly E_{t},\curly E_{t}\otimes K_{\overline{X}})^{*}=\operatorname{Hom}(\curly E_{t},\curly E_{t})^{*}=\mathbb{C}$ since the restriction of $K_{\overline{X}}$ to X\supset\supp $\curly E_{t}$ is trivial.) Therefore $\tau^{\geq 3}R\curly H\!om_{p_{X}}(\curly E,\curly E)\cong R^{3}p_{*}\mathcal{O}$ by basechange and the Nakayama lemma, so the above truncation

[TABLE]

is perfect of amplitude $[0,2]$.

Theorem 3.13**.**

The map (3.10) is a relative obstruction theory for $\mathcal{N}/B$ in the sense of [BF, Definition 4.4 and Section 7].

Proof.

Here we can ignore the $\mathbb{C}^{*}$ action. Fix a morphism of $B$-schemes,

[TABLE]

and an extension of $B$-schemes $T_{0}\subset T$ with ideal $I$ such that $I^{2}=0$.

Compose the pullback of (3.10),

[TABLE]

with the natural map $f^{*}\mathbb{L}_{\mathcal{N}/B}\to\mathbb{L}_{T_{0}/B}$ followed by the composition141414This composition $\mathbb{L}_{T_{0}/B}\to I[1]$ is the Kodaira-Spencer class of $T_{0}\subset T$.

[TABLE]

This defines an element

[TABLE]

By [BF, Theorem 4.5] our Theorem will follow if we can show that $o$ vanishes if and only if there exists an extension from $T_{0}$ to $T$ of the map $f$ of $B$-schemes, and that when $o=0$ the set of extensions is a torsor under $\operatorname{Ext}^{-2}_{T_{0}}(f^{*}\tau^{\leq 2}R\curly H\!om_{p_{X}}(\curly E,\curly E),I)$.

Let

[TABLE]

and let $\bar{p}_{X}$ be the projection

[TABLE]

Since $\bar{p}_{X}$ is flat and $\tau^{\geq 3}R\curly H\!om_{p_{X}}(\curly E,\curly E)=R^{3}p_{X*}\mathcal{O}[-3]$ is locally free, basechange gives

[TABLE]

By the obvious functoriality of Atiyah classes, the pullback (3.14) of the partial Atiyah class is the partial Atiyah class $\operatorname{At}_{\bar{f}^{*}\curly E\!,\,T_{0}}$. Therefore, unwinding the working above and the relative Serre duality of (3.1), we can write (3.16)

[TABLE]

as the composition of

[TABLE]

with the Kodaira-Spencer class (3.15). But in [Ill, Proposition III.3.1.8], Illusie shows that precisely this composition is his class $\omega(B,\curly E)$, which in [Ill, Proposition III.3.1.5] he shows vanishes if and only if $\bar{f}^{*}\curly E$ deforms from $T_{0}\times_{B}X$ to $T\times_{B}X$. Furthermore, when it does vanish, he shows such deformations form a torsor under $\operatorname{Ext}^{1}_{T_{0}\times_{\!B}X}(\bar{f}^{*}\curly E,\bar{f}^{*}\curly E\otimes I)$, which by (3.17) and relative Serre duality for $\bar{p}_{X}$ is $\operatorname{Ext}^{-2}_{T_{0}}(f^{*}\tau^{\leq 2}R\curly H\!om_{p_{X}}(\curly E,\curly E),I)$.

Since deformations of $\bar{f}^{*}\curly E$ from $X\times_{B}T_{0}$ to $X\times_{B}T$ are in 1-1 correspondence with extensions from $T_{0}$ to $T$ of the $B$-map $f$, we are done. ∎

Corollary 3.18**.**

There is a perfect relative obstruction theory of amplitude $[-1,0]$ for $\mathcal{N}$,

[TABLE]

Proof.

Again, for this proof we can ignore the $\mathbb{C}^{*}$ action. We have already seen in (3.12) the complex $\tau^{\leq 2}R\curly H\!om_{p_{X}}(\curly E,\curly E)$ is perfect of amplitude $[0,2]$. Since stable sheaves are automatically simple we have the isomorphism

[TABLE]

for all closed points $t\in\mathcal{N}$. Therefore, by the standard Nakayama lemma arguments (see for instance [HT, Lemma 4.2]),

[TABLE]

is also perfect, of amplitude [1,2].

By (the dual of) [STV, Proposition 3.2]151515By [STV, Remark A.1] their definition of Atiyah class coincides with Illusie’s. we have a commutative diagram

[TABLE]

where $\det\colon\mathcal{N}\to\operatorname{Pic}(X/B)$ takes $\curly E$ to its determinant $\mathcal{O}_{X}(rS)$. Since this is the constant map,161616We can even do without this fact; since $\operatorname{Pic}(X/B)\to B$ is smooth, $\mathbb{L}_{\operatorname{Pic}(X/B)/B}$ is concentrated only in degree 0. Therefore the composition (3.22) is zero in degree $-2$, as required. we conclude the composition

[TABLE]

which factors through \tau^{\leq 0}\big{(}R\curly H\!om_{p_{X}}(\curly E,\curly E)[2]\big{)}, is zero. By (3.21) the perfect obstruction theory (3.10) therefore factors through a map

[TABLE]

Since we have only modified (3.10) in degree $-2$, it still induces an isomorphism on cohomology in degree 0 and a surjection in degree $-1$. Thus it is a perfect obstruction theory. ∎

4. The $U(r)$ Vafa-Witten invariant

From now on we will forget about the base $B$, taking it to be a point for simplicity. It will reappear in Section 6, where it will prove deformation invariance of our Vafa-Witten invariants.

4.1. Localisation

The $\mathbb{C}^{*}$ action of weight 1 on the fibres of $X=K_{S}\to S$, corresponding to the grading (2.5) on the sheaf of algebras $\pi_{*}\mathcal{O}$, induces a $\mathbb{C}^{*}$-action on $\mathcal{N}$, and the obstruction theory (3.19) is naturally $\mathbb{C}^{*}$-equivariant. By [GP] the $\mathbb{C}^{*}$-fixed locus $\mathcal{N}^{\mathbb{C}^{*}}$ inherits a perfect obstruction theory

[TABLE]

by taking the fixed (weight 0) part $(\ \cdot\ )^{f}$ of (3.19). This defines a virtual cycle

[TABLE]

The derived dual of the moving (nonzero weight) part of (3.19) is called the virtual normal bundle,

[TABLE]

Here the second expression follows from $\mathbb{C}^{*}$ equivariant Serre duality on $X$. If $\mathcal{N}$ were compact, its perfect obstruction theory (3.19) of virtual dimension zero would define a zero dimensional virtual cycle [BF]. Its length would be a deformation invariant integer which we could calculate by the localisation formula [GP] as

[TABLE]

Here we represent $N^{\operatorname{vir}}$ as a 2-term complex $\{V_{0}\to V_{1}\}$ of locally free $\mathbb{C}^{*}$-equivariant sheaves with nonzero weights and define

[TABLE]

where $t=c_{1}(\mathfrak{t})$ is the generator of $H^{*}(B\mathbb{C}^{*})=\mathbb{Z}[t]$ and $c_{\mathrm{top}}^{\mathbb{C}^{*}}$ denotes the $\mathbb{C}^{*}$-equivariant top Chern class lying in the localised $\mathbb{C}^{*}$-equivariant cohomology

[TABLE]

Since $\mathcal{N}$ is noncompact but $\mathcal{N}^{\mathbb{C}^{*}}$ is compact, we instead take the residue integral (4.3) as a definition. In this noncompact setting it is only a rational number in general.

Preliminary definition 4.4**.**

Let $S$ be a smooth projective complex surface. For $(r,c_{1},c_{2})$ for which all Gieseker semistable Higgs sheaves are Gieseker stable we define

[TABLE]

Note this is a constant Laurent polynomial in $\mathbb{Q}[t,t^{-1}]$ because $\mathcal{N}$ has virtual dimension zero. Equivalently, over each component of $\mathcal{N}^{\mathbb{C}^{*}}$, the virtual dimension of the obstruction theory (4.1) equals $-\operatorname{rank}(N^{\operatorname{vir}})$. (Both can vary from component to component, but they jump together.) $\widetilde{\mathsf{VW}}(S)$ is just a local DT invariant of $X$.

Remark 4.5**.**

This is only a sensible definition for surfaces with $h^{0,1}(S)=0=h^{0,2}(S)$. The obstruction sheaf of the obstruction theory (4.1) has a trivial $H^{2}(\mathcal{O}_{S})$ summand, so the virtual cycle is zero whenever $h^{0,2}(S)>\nolinebreak 0$. And when $h^{0,1}(S)>0$ the invariance of the obstruction theory under tensoring by flat line bundles means the integrand is pulled back from a lower dimensional space $\mathcal{N}/\operatorname{Jac}(S)$, so the integral vanishes.

For more general surfaces we would like to fix the determinant of $E$ and make $\phi$ trace-free, replacing the groups $\operatorname{Ext}^{i}_{S}(E,E),\ \operatorname{Ext}^{i}_{S}(E,E\otimes K_{S})$ by their trace-free counterparts.171717This is not the same as replacing $\operatorname{Ext}^{i}_{X}(\mathcal{E},\mathcal{E})$ by its trace-free version $\operatorname{Ext}^{i}_{X}(\mathcal{E},\mathcal{E})_{0}$. Instead one should use the kernel of the map $\operatorname{Ext}^{i}_{\overline{X}}(\mathcal{E},\mathcal{E})\to H^{1,i+1}(\overline{X})$ given by cup product with the Atiyah class on $\overline{X}$ of the pushforward of $\mathcal{E}$ followed by the trace map; see for instance [KT1, Equation 27]. It is simpler to interpret this on $S$. In gauge theory language, we want to replace $U(r)$ Higgs bundles with $SU(r)$. So instead of working on $X$ with the first term of (2.15, 2.20) we need to work on $S$ with the other two terms of (2.15, 2.20). Thus we need to relate the deformations of $\mathcal{E}$ to those of $E$ and $\phi$. Equivalently, we need to express the deformation theory of $\mathcal{N}$ in terms of that of $\curly M$ and the fibres of $\Pi\colon\mathcal{N}\to\curly M$ (2.19). This is what we do next.

5. Deformation theory of the Higgs field

By Proposition 3.5, the diagram

[TABLE]

commutes. Dualising and taking cones gives, via the exact triangle (2.20),

[TABLE]

[TABLE]

The right hand vertical arrow — produced by filling in the cones — is the projection of $\operatorname{At}_{\curly E\!,\,\mathcal{N}}$ from $\mathbb{L}_{\mathcal{N}}$ to $\mathbb{L}_{\mathcal{N}/\curly M}$, which factors through the top right hand term in the diagram since it is zero on the top left.

Now any fibre of $\Pi\colon\mathcal{N}\to\curly M$ — the space of (stable) Higgs fields $\phi$ for a fixed sheaf $E\in\curly M$ — can be considered to be a space of quotients $\pi^{*}E\to\mathcal{E}\to 0$ of $\pi^{*}E$. That is, $\mathcal{N}/\curly M$ is part181818It is the Zariski open in Quot consisting of all quotients $\pi^{*}E\to\mathcal{E}$ which induce an isomorphism $E\to\pi_{*}\hskip 0.6pt\mathcal{E}$ on sections. of a relative Quot scheme of $\pi^{*}\mathsf{E}$. Since the kernel of the quotient $\pi^{*}E\to\mathcal{E}$ is $\pi^{*}E\otimes K_{S}^{-1}$ (2.12), its deformations and obstructions are governed by

[TABLE]

and

[TABLE]

respectively. By Serre duality these are cohomologies of $(R\operatorname{Hom}(E,E)[2])^{\vee}$, so this is compatible with the last vertical arrow of the above diagram. So we need to check that the arrow is what we expect, i.e. that the diagram induces the usual obstruction theory for the relative Quot scheme $\mathcal{N}/\curly M$.

5.1. Deformations of quotients

Illusie describes a reduced Atiyah class for quotients (or, more generally, maps of modules) and relates it to his Atiyah class of (3.1, 3.2). In our situation we again consider $\mathcal{N}\times X[\curly E]$ with its projection to \mathcal{N}\times X\big{/}\curly M\times X. We now also consider $\mathcal{N}\times X[\pi^{*}\mathsf{E}]$ with the embedding

[TABLE]

induced from the surjection $\pi^{*}\mathsf{E}\to\curly E\to 0$. It has ideal

[TABLE]

the kernel (2.12) of $\pi^{*}\mathsf{E}\to\curly E$.

These maps of spaces induce a commutative diagram of exact triangles

[TABLE]

on $X\times\mathcal{N}[\curly E]$. As in (3.1, 3.2), taking the degree 1 part of the pushdown to $\mathcal{N}\times X$ of the right hand square gives

[TABLE]

where the bottom left hand term is the ideal (5.4). The lower horizontal arrow defines the reduced Atiyah class191919Also known as the second fundamental form of $\pi^{*}\mathsf{E}\otimes K_{S}^{-1}\hookrightarrow\pi^{*}\mathsf{E}$. of the quotient $\pi^{*}\mathsf{E}\to\curly E$,

[TABLE]

where the isomorphism follows from $\pi_{*\hskip 0.6pt}\curly E\cong\mathsf{E}$.

Proposition 5.8**.**

The right hand arrow of (5.1) is $\operatorname{At}^{\operatorname{red}}_{\Phi}$.

Proof.

This is just the statement that (5.6) commutes, since, by construction, the upper horizontal arrow is the Atiyah class $\operatorname{At}_{\curly E\!,\,\mathcal{N}/\curly M}$ of $\curly E$. ∎

That is, the obstruction theory on $\mathcal{N}/\curly M$ induced from considering $\mathcal{N}$ to be a moduli space of sheaves $\mathcal{E}$ on $X$ (i.e. the right hand arrow of (5.1)) is the same as the standard obstruction theory for quotients $\pi^{*}E\to\mathcal{E}\to 0$ provided by the reduced Atiyah class.

5.2. Deformations of Higgs fields

The alternative description of any fibre of $\mathcal{N}\to\curly M$ as a (linear!) space of Higgs fields $\phi\in\operatorname{Hom}(E,E\otimes K_{S})$ means it has tangent space

[TABLE]

and similarly an obstruction space

[TABLE]

Putting these together in a family we will show — in (5.22) for instance — this description induces a perfect relative obstruction theory

[TABLE]

Of course (5.9, 5.10) are the same as the groups (5.2, 5.3) we got by considering the deformation-obstruction theory of quotients $\pi^{*}E\to\mathcal{E}_{\phi}\to 0$. So to prove these are really the same obstruction theories we would like to show the arrow in (5.11) is $\operatorname{At}^{\operatorname{red}}_{\Phi}$ (5.1). Proposition 5.8 would then imply that the perfect obstruction theory given by the right hand arrow of (5.1) is the same as the one induced by the linear structure on the fibres of $\mathcal{N}/\curly M$. This latter description will allow us to pass more easily to the obstruction theory of trace-free Higgs fields (which is harder to describe from the Quot point of view).

5.3. Matrices

We start by showing the equivalence of the two obstruction theories in the case of Higgs fields over a point, i.e. matrices. (Everything else we need will follow from this case by working in families.) So we fix vector spaces $E$ and $\mathcal{L}$ of ranks $r$ and $1$ respectively, and let $H$ denote the moduli space of Higgs fields $\phi$ on $E$. It is the vector space

[TABLE]

By the spectral construction, each $\phi\in H$ is equivalent to a length-$r$ torsion sheaf $\mathcal{E}_{\phi}$ on $\mathcal{L}$ with an exact sequence

[TABLE]

Here $\pi$ is the projection from $\mathcal{L}$ to a point, and $\eta$ is the tautological section of $\pi^{*}\mathcal{L}$ on $\mathcal{L}$.

At any point $\phi\in H$ consider the maps

[TABLE]

where the the first isomorphism is the identification of a linear space with its tangent space. We want to show the composition (5.12) is the identity to deduce that the two descriptions of $H$ — as a linear space, or as a space of quotients — give rise to the same description of the tangent space.

Lemma 5.13**.**

The composition (5.12) is the identity.

Proof.

Let $R$ denote the dual numbers $\mathbb{C}[t]/(t^{2})$. Fix any tangent vector $\dot{\varphi}\in H\cong T_{\phi}H$, or equivalently a map $\operatorname{Spec}\,R\to H$. This corresponds to the family $\phi+t\dot{\varphi}$ of Higgs fields over $\operatorname{Spec}\,R$, and so the family of quotients

[TABLE]

defined by the cokernel of the second arrow.

Illusie [Ill, Section IV.3.2] shows any such family (flat over $R$) defines a deformation class in

[TABLE]

described as follows. We compare the constant quotient (the right hand horizontal arrow $\operatorname{ev}\otimes 1_{R}$ in the diagram below) with the varying quotient (5.14) (the left hand horizontal arrow):

[TABLE]

Since $t^{2}=0$ the upper diagonal arrow is zero, so the horizontal composition induces the lower diagonal arrow. This has composition zero with the projection to $\mathcal{E}_{\phi}$ so lifts uniquely to a map to $\mathcal{E}_{\phi}\cdot t$.

Splitting $\mathcal{E}_{\phi}\otimes R=\mathcal{E}_{\phi}\cdot t\oplus\mathcal{E}_{\phi}$ and $\pi^{*}E\otimes R=\pi^{*}E\cdot t\oplus\pi^{*}E$, we see from the diagram that this map is $\dot{\varphi}$. Since Illusie also shows the class is $\operatorname{At}^{\operatorname{red}}_{\phi}\cdot\,\dot{\varphi}$, this proves the Lemma. ∎

From this universal case, we can deduce the same result for any family of Higgs fields $\phi$ on the vector space $E$, parametrised by any (possibly singular) base $\mathcal{H}$. Let $f\colon\mathcal{H}\to H$ be its classifying map and let $\Phi$ denote the universal Higgs field.

Corollary 5.16**.**

The map

[TABLE]

is the same as the derivative

[TABLE]

Proof.

We first observe this is true for the universal family $H$ itself (with $f,\ Df$ the identity). Since $H$ is smooth its tangent complex $\mathbb{T}_{H}$ is a vector bundle, and to show an endomorphism of a vector bundle on a smooth space is the identity it is sufficient to check it on restriction to any point. Therefore Lemma 5.13 proves the claim.

That is, $\operatorname{At}^{\operatorname{red}}_{\Phi}=\operatorname{id}$. The obvious functoriality $f^{*}\operatorname{At}^{\operatorname{red}}_{\Phi}=\operatorname{At}^{\operatorname{red}}_{f^{*}\Phi}$ therefore gives $\operatorname{At}^{\operatorname{red}}_{f^{*}\Phi}=f^{*}\operatorname{id}=Df$, as required. ∎

5.4. Higgs bundles

We can now deduce the same result in a family of vector spaces, i.e. when $E$ is a vector bundle.

Suppose that $\curly M$ parametrises only vector bundles on $S$ (or shrink it so that it does). Let $X$ be the total space of $K_{S}$ as usual. From the universal bundle $\mathsf{E}$ on $\curly M\times S$ we form the vector bundle

[TABLE]

Over $\mathsf{H}\times_{\curly M\times S}\curly M\times X\xrightarrow{\ \pi\ }\mathsf{H}$ we get a universal Higgs field $\Phi$ and, by the spectral construction, a universal quotient

[TABLE]

Using the smooth linear structure on the fibres of $\mathsf{H}\to\curly M\times S$ gives the first isomorphism in the sequence of maps

[TABLE]

Lemma 5.18**.**

The composition (5.17) is the identity.

Proof.

Lemma 5.13 shows that restricted to any closed point of $\mathsf{H}$ the composition is the identity. If $\curly M$ is reduced this proves the claim.

In general it is sufficient to check the claim locally, since maps of sheaves (rather than complexes of sheaves) are equal if and only if they are equal locally. So, shrinking $\curly M$ and $S$ if necessary, we can assume that both $\mathsf{E}$ and $K_{S}$ are trivial. Then applying Corollary 5.16 to $f$ the projection $\mathsf{H}\cong H\times\curly M\times S\to H$ proves that (5.17) is the identity. ∎

Thinking of $\mathcal{N}/\curly M$ as a moduli space of sections of $\mathsf{H}\to\curly M\times S$, the graph of the universal section $\Phi$ gives an embedding

[TABLE]

where $\Pi\colon\mathcal{N}\times S\to\curly M\times S$ is the projection. Its normal bundle is the fibrewise tangent bundle of $\Pi^{*}\mathsf{H}\to\mathcal{N}\times S$, which by the linear structure is just $\Pi^{*}\mathsf{H}$:

[TABLE]

Therefore, by [BF, Proposition 6.2] applied to $(\curly M\times S)$-maps (i.e. sections) from $\curly M\times S$ to $\mathsf{H}$ we get the linear relative obstruction theory for $\mathcal{N}$ promised in (5.11) by starting with the map

[TABLE]

and applying adjunction:

[TABLE]

Proposition 5.23**.**

The relative obstruction theory (5.22) is the same as the one given by the right hand arrow of (5.1).

Proof.

By Proposition 5.8 the right hand arrow of (5.1) is given by the reduced Atiyah class $\operatorname{At}^{\operatorname{red}}_{\Phi}$ of the universal quotient $\pi^{*}\mathsf{E}\to\curly E_{\Phi}$ on $\mathcal{N}\times X$. It is induced from the composition of the arrow (5.21) with the pullback by $\Phi^{*}\Pi^{*}$ of (5.17). By Lemma 5.18 the latter is the identity, so we are done. ∎

5.5. Trace

Taking the trace of the section $\Phi$ (5.19) of $\mathsf{H}$ we get a section $\operatorname{tr}\Phi$ of $K_{S}$ instead:

[TABLE]

Replacing the above analysis (5.20, 5.21, 5.22) for the section (5.19) (the graph of $\Phi$) by the (simpler) analysis for the section (5.24) (the graph of $\operatorname{tr}\Phi$) gives a relative obstruction theory

[TABLE]

for the moduli space $\curly M\times\Gamma(K_{S})\to\curly M$ of pairs $(E,\sigma),\ \sigma\in\Gamma(K_{S})$, together with a commutative diagram

[TABLE]

compatible with the map

[TABLE]

Moreover, identifying $\mathbb{T}_{\curly M\times\Gamma(K_{S})/\curly M}$ with $\Gamma(K_{S})\otimes\mathcal{O}$ using the obvious linear structure, the construction of the right hand arrow of (5.25) (i.e. the analogue of (5.20) for the section $\operatorname{tr}\Phi$) shows that it is the canonical embedding

[TABLE]

Taking co-cones in (5.25) gives the commutative diagram of exact triangles

[TABLE]

[TABLE]

where $\operatorname{At}^{\operatorname{red}}_{0}$ is the trace-free component of $\operatorname{At}^{\operatorname{red}}_{\Phi}$ in the splitting of the top row. The (dual of the) left hand map defines an obstruction theory by the following standard Lemma.

Lemma 5.28**.**

Given maps of spaces $A\to B\to C$ and a map of distinguished triangles on $A$,

[TABLE]

if the central and right hand vertical arrows are relative obstruction theories, then so is the left hand vertical arrow.

Proof.

Taking sheaf cohomology we get the maps of long exact sequences

[TABLE]

where $\mathbb{L}^{-1}:=h^{-1}(\mathbb{L})$. The obstruction theories give isomorphisms on $h^{0}$ and surjections on $h^{-1}$ as shown by the arrows marked and respectively. By a simple diagram chase, the first vertical arrow is an isomorphism and the fourth is a surjection. ∎

Fitting (5.27) in with (5.1) we get the diagram of exact triangles

[TABLE]

[TABLE]

(where $R\curly H\!om_{p_{X}}(\curly E,\curly E)^{0}$ is defined by taking cones in the central row) receiving Atiyah class maps from the diagram

[TABLE]

[TABLE]

with everything commutative. Along the top rows the maps are obstruction theories, as we just showed following (5.27). In the middle row the maps on the central and right hand terms are obstruction theories by Theorem 3.13 and Proposition 5.23 respectively. Therefore the map on the first terms of the middle rows

[TABLE]

is also an obstruction theory, by Lemma 5.28 again.

5.6. Determinant

Having dealt with the trace of Higgs fields, we now also deal with the determinant of Higgs bundles on $S$. This is more standard. We start with the left hand column of (5.29), making it the central column of the following commutative diagram of exact triangles

[TABLE]

[TABLE]

Here the central row is defined in Corollary 2.26. We then get the left hand column by taking co-cones. We claim this diagram is mapped to commutatively by the diagram of exact triangles202020Here, as before, $\det\colon\mathcal{N}\to\operatorname{Pic}(S)$ is the map which on points takes $(E,\phi)\mapsto\det E$.

[TABLE]

[TABLE]

The maps between central columns come from the Atiyah class maps between (5.30) and (5.29). Between the right hand columns we similarly take the Atiyah class of the line bundle $\det\curly E$. These give a commutative square by (the dual of) [STV, Proposition 3.2]. Taking co-cones gives the maps on the left hand column.

The splitting of the central row of (5.32) defines a splitting of the Atiyah class $\operatorname{At}_{\curly E\!,\,\mathcal{N}}$ into components \big{(}\!\operatorname{At}_{\curly E\!,\,\mathcal{N}}^{\perp}\,,\,\operatorname{At}_{\det\curly E\!,\,\mathcal{N}}\big{)}.

Proposition 5.34**.**

Over the open locus of Higgs bundles in $\mathcal{N}$, the map

[TABLE]

defined above is a 2-term symmetric perfect relative obstruction theory for \mathcal{N}\big{/}\Gamma(K_{S})\times\operatorname{Pic}(S).

Proof.

Consider the maps between the middle rows of (5.32) and (5.33). Between the central terms the map is the obstruction theory (5.31). Between the right hand terms it is trivially an obstruction theory. Therefore the map between the left hand terms is also a (relative) obstruction theory by Lemma 5.28.

That $R\curly H\!om_{p_{X}}(\curly E,\curly E)_{\perp}[2]$ is perfect of amplitude $[-1,0]$ follows from basechange just as in Corollary 3.18. Its symmetry was noted in Corollary 2.26. Recording the $\mathbb{C}^{*}$ action explicitly as in Corollary 3.18 explains the $\mathfrak{t}^{-1}$. ∎

5.7. Higgs sheaves

When the rank of the vector space jumps in the family — i.e. when $E$ is a sheaf rather than a bundle — things are more complicated and we need to replace sheaves by locally-free resolutions.

We first shrink $\curly M$ to the image of $\Pi\colon\mathcal{N}\to\curly M$. This ensures boundedness since $\mathcal{N}$ is quasi-projective. As $\curly M$ parametrises only torsion-free sheaves, these all have homological dimension $\leq 1$ on $S$. Therefore the universal sheaf $\mathsf{E}$ also has homological dimension $\leq 1$.

Fix an ample line bundle $\mathcal{O}(1)$ on $S$ (and use the same notation $\mathcal{O}(1)$ for its pullbacks to $\curly M\times S$ and $\mathcal{N}\times S$ as usual). Then for sufficiently large $N\gg 0$ we get a surjection

[TABLE]

Since $\mathsf{E}$ is flat over $\curly M$, for $N\gg 0$ we deduce that this $\mathsf{E}_{1}$ is locally free. Letting $\mathsf{E}_{2}$ be the kernel of (5.35)), it is also locally free since $\mathsf{E}$ has homological dimension $\leq 1$. So from now on we fix this two-term locally free resolution

[TABLE]

Its advantage is its functoriality, which implies that any (twisted, Higgs field) endomorphism of $E$ induces compatible endomorphisms of the $E_{i}$. The universal case is over $\mathcal{N}/\curly M$, where the universal Higgs field $\Phi$ on (the pullback to $\mathcal{N}\times S$ of) $\mathsf{E}$ induces canonical maps

[TABLE]

i.e. $\Phi_{i}\in\operatorname{Hom}(\mathsf{E}_{i},\mathsf{E}_{i}\otimes K_{S})$ such that

[TABLE]

Moreover any maps $\Phi_{i}$ satisfying (5.37) induce a map $\Phi$ which is unique: other choices differ by an element of

[TABLE]

since the choice of $\mathsf{E}_{1}$ ensures that $p_{S*}(\mathsf{E}_{2}(N))=0$. We can phrase this as saying that $\mathcal{N}$ is cut out of

[TABLE]

by the equation (5.37). Here $\mathcal{N}_{i}\to\curly M$ denotes the moduli space of Higgs fields $\phi_{i}$, with fibre over $E\in\curly M$ given by $\operatorname{Hom}(E_{i},E_{i}\otimes K_{S})$ (where the $E_{i}$ are the restriction to $S\times\{E\}$ of the $\mathsf{E}_{i}$ (5.36)).

Using (5.36), $R\curly H\!om(\mathsf{E},\mathsf{E})$ is computed by the total complex of

[TABLE]

Since the top left corner injects into $\mathsf{E}_{1}^{*}\otimes\mathsf{E}_{1}$ this is quasi-isomorphic to

[TABLE]

[TABLE]

with the given trace and identity maps. The identity $\operatorname{tr}\circ\operatorname{id}=\operatorname{rank}\neq 0$ gives the usual splitting

[TABLE]

Over $\mathcal{N}$ the universal Higgs field $\Phi$ defines a section $\operatorname{tr}\Phi$:

[TABLE]

where as usual we have abbreviated $\pi^{*}\mathsf{E}$ to $\mathsf{E}$ and also suppressed the pullback on $K_{S}$. So we have all the same ingredients as we had in the Higgs bundles case, except we have not yet shown that the composition

[TABLE]

is the same as212121In other words, we will not extend all of Proposition 5.23 from bundles to sheaves to give the full equivalence of the two natural perfect relative obstruction theories for $\mathcal{N}/\curly M$, as it is enough for our purposes to deal only with the trace.

[TABLE]

To prove the equality of (5.40) and (5.41) we repeat our earlier analysis for Higgs fields on vector bundles to both $(\mathsf{E}_{1},\phi_{1})$ and $(\mathsf{E}_{2},\phi_{2})$. Section 5.1 gives reduced Atiyah classes (5.1)

[TABLE]

which we consider as maps

[TABLE]

They determine the reduced Atiyah class $\operatorname{At}^{\operatorname{red}}_{\Phi}$ of $\mathsf{E}$ as follows.

Proposition 5.43**.**

Using the resolution (5.36) to write $R\curly H\!om$ as a total complex (5.39), the reduced Atiyah class of $\mathsf{E}$,

[TABLE]

becomes

[TABLE]

where $\operatorname{At}_{\phi_{i}}^{\operatorname{red}}$ denotes the reduced Atiyah class (5.42) restricted to $\mathcal{N}$ by (5.38).

Note that (5.45) really defines a map in the derived category to the total complex on the right hand side because $d\circ\operatorname{At}^{\operatorname{red}}_{\phi_{2}}=\operatorname{At}^{\operatorname{red}}_{\phi_{1}}\circ\,d$ by (5.37) and the functoriality of Atiyah classes under the pullback to $\mathcal{N}\subset\mathcal{N}_{1}\times_{\curly M}\mathcal{N}_{2}$.

Proof.

We follow the definition of reduced Atiyah class in (5.5). The maps of sheaves on $\mathcal{N}\times X$,

[TABLE]

induce maps of spaces

[TABLE]

where the vertical maps are embeddings with ideals $\pi^{*}\mathsf{E}_{2}\otimes K_{S}^{-1},\,\pi^{*}\mathsf{E}_{1}\otimes K_{S}^{-1}$ and $\pi^{*}\mathsf{E}\otimes K_{S}^{-1}$ respectively. These induce maps of cotangent complexes

[TABLE]

Taking weight 1 parts of their pushdowns to $\mathcal{N}\times X$ gives

[TABLE]

which is equivalent, by adjunction and $\pi_{*}\hskip 0.6pt\mathcal{E}_{i}=E_{i},\ \pi_{*}\hskip 0.6pt\curly E=\mathsf{E}$, to the following diagram of horizontal exact triangles on $\mathcal{N}\times S$:

[TABLE]

The commutativity of this diagram gives the claimed result. ∎

In particular

[TABLE]

by applying (5.25, 5.26) to both Higgs bundles $(E_{i},\phi_{i})$. Hence (5.40) equals (5.41) and Sections 5.5 and 5.6 go through for Higgs sheaves as well as bundles. Thus Proposition 5.34 holds for Higgs sheaves too:

Theorem 5.46**.**

There is a 2-term symmetric perfect relative obstruction theory

[TABLE]

for \mathcal{N}\big{/}\Gamma(K_{S})\times\operatorname{Pic}(S). $\square$

6. The $SU(r)$ Vafa-Witten invariant

Restricting the relative obstruction theory of Theorem 5.46 to the fibre $\mathcal{N}_{L}^{\perp}$ over

[TABLE]

we get an absolute obstruction theory.

Theorem 6.1**.**

The moduli space $\mathcal{N}_{L}^{\perp}$ of stable Higgs sheaves $(E,\phi)$ with $\det E\cong L$ and trace-free $\phi\in\operatorname{Hom}(E,E\otimes K_{S})_{0}$ admits a 2-term symmetric perfect obstruction theory

[TABLE]

$\hfill\square$**

Localising as in Section 4, we can now give a general definition of our Vafa-Witten invariant for any surface $S$ (which agrees with Preliminary Definition 4.4 when $h^{0,1}(S)=0=h^{0,2}(S)$).

Definition 6.2**.**

Let $S$ be a smooth projective complex surface, and fix $(r,c_{1},c_{2})$ with $r>0$ for which all Gieseker semistable Higgs sheaves are Gieseker stable. We define

[TABLE]

where $L$ is any line bundle on $S$ with $c_{1}(L)=c_{1}$.

Remark 6.4**.**

This is deformation invariant under deformations of $S$ for which $c_{1}$ remains of type (1,1). More precisely, suppose $\mathcal{S}\to B$ is a smooth family of projective surfaces over a connected base $B$ with a global class $c_{1}\in H^{1,1}(\mathcal{S})$ whose restriction to any fibre $\mathcal{S}_{b},\ b\in B,$ we also denote by $c_{1}\in H^{1,1}(\mathcal{S}_{b})$.222222Equivalently the classifying map from $B$ to the moduli stack of surfaces $S$ has image in the Noether-Lefschetz locus of $(S,c_{1})$. Let $\mathcal{L}$ be any line bundle over $\mathcal{S}$ with $c_{1}(\mathcal{L})=c_{1}$. Then just as in Section 3 we may do everything relative to $B$ so that $\mathcal{N}_{\mathcal{L}}^{\perp}\to B$ has a perfect relative obstruction theory, inducing one on its $\mathbb{C}^{*}$-fixed locus by [GP]. Thus

[TABLE]

is independent of $b\in B$ by [BF, Proposition 7.2] and conservation of number [Fu, Theorem 10.2].

The invariant need not be deformation invariant, however, under deformations of $S$ for which the Hodge type of $c_{1}(L)$ does not remain of Hodge type $(1,1)$. In this situation there are no sheaves with $\det E=L$ on the deformed surface, so the invariant becomes zero.

6.1. More general Higgs pairs

In Theorem 6.1 we can replace $K_{S}$ by any line bundle $\mathcal{L}\to S$ with $\deg\mathcal{L}\geq\deg K_{S}$ at the expense of dropping the word “symmetric”. That is, we consider the moduli space $\mathcal{N}_{r,L,c_{2}}^{\perp}$ of stable $\mathcal{L}$-Higgs pairs

[TABLE]

on $S$ of rank $r$ and second Chern class $c_{2}$. Equivalently, it is the moduli space of stable torsion sheaves $\mathcal{E}_{\phi}$ on the total space

[TABLE]

with centre of mass zero on the fibres of $X\to S$ and $\det\pi_{*}\hskip 0.6pt\mathcal{E}_{\phi}\cong L$.

Theorem 6.5**.**

Fix $(r,L,c_{2})$ such that semistability implies stability. Suppose $\mathcal{L}$ satisfies

• •

$\deg\mathcal{L}>\deg K_{S}$, or

• •

$\deg\mathcal{L}=\deg K_{S}$* but $\mathcal{L}^{\otimes r}\not\cong K_{S}^{\otimes r}$, or*

• •

$\mathcal{L}=K_{S}$.

Then $\mathcal{N}_{r,L,c_{2}}^{\perp}$ admits a 2-term perfect obstruction theory

[TABLE]

We briefly sketch the modifications that have to be made to the proof of Theorem 6.1 to derive this. Proposition 2.21 is the first place we set $\mathcal{L}=K_{S}$. For general $\mathcal{L}$ it is modified to say that the exact triangle (2.20)

[TABLE]

is Serre dual (up to a shift) to the exact triangle

[TABLE]

made in the same way: by applying $Rp_{X*}$ to the exact triangle (2.16) and its twist by $K_{X/B}\cong\pi^{*}(K_{S/B}\otimes\mathcal{L}^{-1})$ respectively.

Corollary 2.26 then gets modified by replacing $K_{S/B}$ by $\mathcal{L}$. In the proof we note the triangle (2.27) can be twisted by the line bundle $K_{X}\cong\mathcal{L}^{-1}\otimes K_{S}$ to become

[TABLE]

allowing us to replace (2.29) by

[TABLE]

This allows us to remove $Rp_{S*\hskip 0.6pt}\mathcal{L}[-1]\oplus Rp_{S*\hskip 0.6pt}\mathcal{O}_{S}$ from $R\curly H\!om_{p_{X}}(\curly E,\curly E)$ to form $R\curly H\!om_{p_{X}}(\curly E,\curly E)_{\perp}$ much as before. Serre duality inserts an extra $K_{X}$ in both (3.10),

[TABLE]

and (3.19). To make Corollary 3.18 run we have to ensure the trace-free part of the left hand side of (6.6) is 2-term. To do this we modify (3.11) by

[TABLE]

when $\mathcal{L}\neq K_{S}$. When $\deg\mathcal{L}>\deg K_{S}$ then $\deg K_{X}<0$, so (6.7) follows from stability. When $\deg\mathcal{L}=\deg K_{S}$, any nonzero element of $\operatorname{Hom}_{X}(\curly E_{t},\curly E_{t}\otimes K_{X})$ must be an isomorphism by stability. By pushdown we get an isomorphism $\mathsf{E}_{t}\to\mathsf{E}_{t}\otimes\mathcal{L}^{-1}K_{S}$ whose determinant is an isomorphism $\mathcal{L}^{\otimes r}\to K_{S}^{\otimes r}$, contradicting our assumption and so proving (6.7).

The rest of the proof of Theorem 6.1 can then be copied almost verbatim, with a few twists by line bundles, to deduce Theorem 6.5.

7. Vanishing theorem and the two fixed loci

The integral (6.3) is over two types of fixed components, as mentioned in (\tagform@1.9, \tagform@1.11).

7.1. The first fixed locus

The instanton branch has $\phi=0$ and so gives the moduli space $\mathcal{M}_{L}$ of Gieseker stable232323Recall we are fixing $(r,c_{1},c_{2})$ for which stability = semistability. sheaves on $S$ of fixed determinant $L$. Consider the dual of the top row of the diagram of Corollary 2.26. Its coboundary map is $[\ \cdot\ ,\phi]$ by (2.20) which vanishes when $\phi=0$. Therefore the exact triangle splits the obstruction theory of Theorem 6.1

[TABLE]

into fixed and moving parts. By [GP] the former defines a perfect obstruction theory for $\mathcal{M}_{L}$,

[TABLE]

Here $E^{\bullet}$ is quasi-isomorphic to a 2-term complex of $\mathbb{C}^{*}$-fixed locally free sheaves $E^{-1}\to E^{0}$. Similarly, dualising the moving parts gives the virtual normal bundle

[TABLE]

Therefore the contribution of the fixed locus $\mathcal{M}_{L}$ to the invariant (4.3) is

[TABLE]

where $r$ and $s$ are the ranks of $E^{-1}$ and $E^{0}$ respectively. The integrand is homogeneous of degree $s-r$ equal to the virtual dimension

[TABLE]

of the perfect obstruction theory (7.1) for $\mathcal{M}_{L}$. Therefore only the $t^{0}$ coefficient has the correct degree $\operatorname{vd}$ over $\mathcal{M}_{L}$ to have nonzero integral against $[\mathcal{M}_{L}]^{\operatorname{vir}}$, and we may set $t=1$ in the above to give

[TABLE]

Here $c_{\bullet}(\,\cdot\,)$ denotes the total Chern class. This is

[TABLE]

the top (in a virtual, or derived sense) Chern class of the virtual cotangent bundle $E^{\bullet}$ of $\mathcal{M}_{L}$. This is the Ciocan-Fontanine-Kapranov/Fantechi-Göttsche signed Euler characteristic of $\mathcal{M}_{L}$ studied in [JT].

7.2. Vanishing theorem

Under certain positive curvature hypotheses, the above integer is the only contribution to $\mathsf{VW}_{r,c_{1},c_{2}}(S)$.

Proposition 7.4**.**

If $\deg K_{S}\leq 0$ then any stable $\mathbb{C}^{*}$-fixed Higgs pair $(E,\phi)$ has Higgs field $\phi=0$. If $r,c_{1},c_{2}$ are chosen so that semistability implies stability, then $\mathsf{VW}_{r,c_{1},c_{2}}(S)$ is the signed virtual Euler characteristic (7.3) of the moduli space $\mathcal{M}_{L}(S)$ of stable sheaves on $S$ with fixed determinant $L$.

Proof.

Since both $\ker\phi$ and $\operatorname{im}\phi$ are $\phi$-invariant subsheaves (of $E$ and $E\otimes K_{S}$ respectively) Gieseker stability gives the following inequalities on reduced Hilbert polynomials,

[TABLE]

unless $\operatorname{im}\phi$ is 0 or all of $E\otimes K_{S}$. But $\deg K_{S}\leq 0$ implies that $p_{E\otimes K_{S}}(n)\leq p_{E}(n)$ for $n\gg 0$. So either $\phi=0$ or $\phi$ is an isomorphism. Since $\phi$ is $\mathbb{C}^{*}$-fixed it has determinant 0, so it cannot be an isomorphism. ∎

Remark. If either $\deg K_{S}<0$ or $K_{S}\cong\mathcal{O}_{S}$ then for any $E\in\mathcal{M}_{L}$,

[TABLE]

In the first case this is because $E$ is stable and $\deg E\otimes K_{S}<\deg E$ so the only map $E\to E\otimes K_{S}$ is zero. In the second case it is because stable sheaves $E$ are simple. Therefore the obstruction space vanishes at any $E\in\mathcal{M}_{L}$, so $\mathcal{M}_{L}$ is smooth of dimension $\operatorname{vd}$ (7.2). In particular, the signed virtual Euler characteristic (7.3) is just the signed topological Euler characteristic

[TABLE]

The same proof with non-strict inequalities in (7.5) gives the following.

Proposition 7.6**.**

If $\deg K_{S}<0$ then any semistable $\mathbb{C}^{*}$-fixed Higgs pair $(E,\phi)$ has Higgs field $\phi=0.\hfill\square$

7.3. The second fixed locus

Connected components of the monopole branch $\mathcal{M}_{2}$ (\tagform@1.11), with $\phi\neq 0$, are more interesting. Since the $\mathbb{C}^{*}$-fixed stable sheaves $\mathcal{E}_{\phi}$ are simple, we can make them $\mathbb{C}^{*}$-equivariant by [Ko, Proposition 4.4].242424In [TT2, Proposition 5.1] we prove the same holds for general (not necessarily simple) $\mathbb{C}^{*}$-fixed Higgs pairs, which we apply in the semistable case. Equivalently, $E$ carries a $\mathbb{C}^{*}$-action $\psi\colon\mathbb{C}^{*}\to\operatorname{Aut}(E)$ such that

[TABLE]

Therefore $E=\oplus_{i}E_{i}$ splits into weight spaces $E_{i}$ on which $t$ acts as $t^{i}$. That is, with respect to this splitting we can write $\psi_{t}=\mathrm{diag}(t^{i})$. By (7.7), this action acts on the Higgs field with weight $1$. Conversely, a Higgs pair $(E,\phi)$ with a $\mathbb{C}^{*}$ action on $E$ whose induced action on $\phi$ has weight $1$ clearly defines a fixed point of our original $\mathbb{C}^{*}$ action.

Since $\phi$ decreases weight, it maps the lowest weight torsion subsheaf to zero. This subsheaf is therefore $\phi$-invariant, and so zero by stability. In particular each of the $E_{i}$ are torsion-free, and so in particular have rank $>0$. Thus $\phi$ acts blockwise through maps

[TABLE]

and we get flags of torsion-free sheaves on $S$ of the sort studied by Negut [Ne]. When the $E_{i}$ have rank 1 they are twists of ideal sheaves by line bundles (since they are torsion-free by stability), and the maps (7.8) define nesting of these ideals. Therefore one can express these components of the $\mathbb{C}^{*}$-fixed locus $\mathcal{M}_{2}$ in terms of nested Hilbert schemes (of curves and points) on $S$. In particular one gets the virtual cycles on nested Hilbert schemes discovered recently in [GSY1, GSY2], at least when $h^{0,1}(S)=0=h^{0,2}(S)$.

We explain an extended example with $\operatorname{rank}(E)=2$ in detail next, leaving the general case to [GSY1]. Since the $E_{i}$ are torsion free and rank $>0$, they have rank 1, they are ideal sheaves tensored with line bundles, and there are only two of them:

[TABLE]

Without loss of generality $i>j$. Since the Higgs field has weight 1, it takes weight $k$ to weight $k-1$. It is also nonzero, so we must have $j=i-1$ and the only nonzero component of $\phi$ maps $E_{i}$ to $E_{i-1}$.

Let $\mathfrak{t}$ denotes the one dimensional $\mathbb{C}^{*}$ representation of weight 1. By tensoring $E$ by $\mathfrak{t}^{-i}$ (i.e. multiplying the $\mathbb{C}^{*}$ action on $E$ by $\lambda^{-i}\cdot\operatorname{id}_{E}$) we may assume without loss of generality that $i=0$ and $j=-1$. Considering $\phi$ as a weight 0 element of $\operatorname{Hom}(E,E\otimes K_{S})\otimes\mathfrak{t}$, we have

[TABLE]

We now calculate invariants from these on certain general type surfaces.

8. Calculations on surfaces with positive canonical bundle

Let $(S,\mathcal{O}_{S}(1))$ be a smooth, connected polarised surface with

• •

$h^{1}(\mathcal{O}_{S})=0$, and

• •

a smooth nonempty connected canonical divisor $C\in|K_{S}|$, such that

• •

$L=\mathcal{O}_{S}$ is the only line bundle satisfying $0\leq\deg L\leq\frac{1}{2}\deg K_{S}$,

where degree is defined by $\deg L=c_{1}(L)\cdot c_{1}(\mathcal{O}_{S}(1))$. Examples include

• •

the generic quintic surface in ${\mathbb{P}}^{\hskip 0.6pt3}$,

• •

the generic double cover of ${\mathbb{P}}^{\hskip 0.6pt2}$ branched over an octic,

• •

general type surfaces with $\operatorname{Pic}(S)=\mathbb{Z}\cdot K_{S}$, and

• •

the blow up of K3 in a point.

More generally, we expect many general type surfaces with $h^{1}(\mathcal{O}_{S})=0$ and $p_{g}>0$ to have enough deformations to move the Hodge structure on $H^{2}(S)$ so the only integral (1,1) classes are rational multiples of $c_{1}(S)$. Such surfaces therefore satisfy the above conditions if $c_{1}(S)\in H^{2}(S,Z)$ is primitive.

By adjunction the genus $g$ of $C$ is

[TABLE]

while the bundles $K_{S}$ and $K_{S}^{2}$ have sections

[TABLE]

and

[TABLE]

By the obvious long exact sequences and Serre duality it is easy to show they have no higher cohomology.

We consider $\mathbb{C}^{*}$-fixed rank 2 Higgs pairs $(E,\phi)$ of fixed determinant $K_{S}$ in the monopole branch $\mathcal{M}_{2}$ (\tagform@1.11). They were described in (7.9) above. In particular, $E_{-1}\subset E$ is a $\phi$-invariant subsheaf, so by semistability

[TABLE]

But the existence of the nonzero map $\Phi\colon E_{0}\to E_{-1}\otimes K_{S}$ implies

[TABLE]

Together these give $0\leq\deg E_{-1}\leq\frac{1}{2}\deg K_{S}$, which by our assumptions on $S$ implies that $\det E_{-1}=\mathcal{O}_{S}$ and $\det E_{0}=K_{S}$. Therefore

[TABLE]

for some ideal sheaves $I_{i}$. Since $\Phi$ defines a nonzero map $I_{0}\to I_{1}$ we must have $I_{0}\subseteq I_{1}$. That is, denoting by $Z_{j}$ the 0-dimensional subscheme defined by the ideal $I_{j}$,

[TABLE]

This can be done in families to prove the following scheme-theoretic description of the $\mathbb{C}^{*}$-fixed loci.

Lemma 8.3**.**

Fix $r=2,\ c_{1}=-c_{1}(S),\ c_{2}\in\mathbb{Z}$. If $c_{1}(S)^{2}\geq 0$, then for $c_{2}<0$ the $\mathbb{C}^{*}$-fixed locus is empty. For $c_{2}\geq 0$ it is the disjoint union of $\mathcal{M}_{r,c_{1},c_{2}}$ and the nested Hilbert schemes

[TABLE]

of subschemes $Z_{1}\subseteq Z_{0}\subset S$ of lengths

[TABLE]

$\hfill\square$**

In general $S^{[i,j]}$ is singular, but connected. In particular we have the cases

• •

$c_{2}(E)<0$. Then $\mathcal{M}_{2}$ is empty and — if $c_{1}(S)^{2}\geq 0$ — $\mathcal{M}_{L}$ is also empty by the Bogomolov inequality for stable sheaves. So $\mathsf{VW}_{r,\hskip 0.6ptK_{S},\hskip 0.6ptc_{2}<0}=0$.

• •

$c_{2}(E)=0$. Then $\mathcal{M}_{2}$ is the single point with $E=K_{S}\oplus\mathcal{O}\!\cdot\!\mathfrak{t}^{-1}$ and $\phi\colon E\to E\otimes K_{S}\!\cdot\!\mathfrak{t}$ is \left(\begin{array}[]{cc}\!\!0&0\!\!\\ \!\!1&0\!\!\end{array}\right). This Higgs pair already appears in [VW, Equation 2.70].

• •

$c_{2}(E)=1$. Then $\mathcal{M}_{2}\cong S$ with $x\in S$ corresponding to $E=I_{x}\otimes K_{S}\oplus\mathcal{O}\!\cdot\!\mathfrak{t}^{-1}$ and Higgs field \phi=\left(\begin{array}[]{cc}\!\!0&0\!\!\\ \!\!\iota_{x}&0\!\!\end{array}\right), where $\iota_{x}$ is the (twist by $K_{S}$ of) the inclusion $I_{x}\hookrightarrow\mathcal{O}$.

• •

$c_{2}(E)=2$. Now $\mathcal{M}_{2}$ has two components,

[TABLE]

The first $S^{[2]}:=\operatorname{Hilb}^{2}S$ is similar to the previous example, with length-2 subschemes $Z\subset S$ replacing $\{x\}\subset S$. The second is a copy of $S$, with $x\in S$ corresponding to $E=I_{x}\otimes K_{S}\oplus I_{x}\!\cdot\!\mathfrak{t}^{-1}$ and Higgs field \phi=\left(\begin{array}[]{cc}\!\!0&0\!\!\\ \!\!1&0\!\!\end{array}\right).

8.1. Horizontal terms

We will calculate first for $i=0$ in (8.4), i.e. $Z_{1}$ empty ($I_{1}=\mathcal{O}$) and

[TABLE]

Denoting $|Z_{0}|=c_{2}(E)$ by $n$, this is a connected component $S^{[n]}=S^{[0,n]}$ of the $\mathbb{C}^{*}$-fixed locus, as noted above. Even this case turns out to be both hard and interesting. It describes all $\mathbb{C}^{*}$-fixed points when $c_{2}\leq 1$ and $c_{1}(S)^{2}\geq 0$, by the Bogomolov inequality.

The deformation theory of $(E,\phi)$ with $\det E=K_{S}$ fixed and $\operatorname{tr}\phi=0$ is governed by $R\operatorname{Hom}_{X}(\mathcal{E}_{\phi},\mathcal{E}_{\phi})_{\perp}$, which is the co-cone on

[TABLE]

Here we have been more explicit about the $\mathbb{C}^{*}$ action, inserting the character $\mathfrak{t}$ so that the Higgs field $\phi$ becomes of weight 0.

The splitting of $E$ induces a splitting of $\operatorname{Hom}(E,E)$ which we denote, in the obvious notation, by

[TABLE]

Similarly, $\operatorname{Hom}(E,E\otimes K_{S}\!\cdot\!\mathfrak{t})$ splits as

[TABLE]

Since \phi=\left(\begin{array}[]{cc}\!\!0&0\!\!\\ \!\!\iota&0\!\!\end{array}\right), the map $[\ \cdot\ ,\phi]$ between them acts by

[TABLE]

Setting $b=-a$ gives the map on trace-free groups. We find the map $\operatorname{Hom}(E,E)_{0}\to\operatorname{Hom}(E,E\otimes K_{S}\otimes\mathfrak{t})_{0}$ is injective and has cokernel

[TABLE]

Next we compute $\operatorname{Ext}^{1}(E,E)=\operatorname{Ext}^{1}(E,E)_{0}$ as

[TABLE]

where in the bottom left entry we have used Serre duality on $S$.

Similarly $\operatorname{Ext}^{1}(E,E\otimes K_{S}\!\cdot\!\mathfrak{t})=\operatorname{Ext}^{1}(E,E\otimes K_{S})_{0}\cdot\mathfrak{t}$ is

[TABLE]

The map $[\ \cdot\ ,\phi]$ between them acts by

[TABLE]

Lemma 8.7**.**

This map vanishes.

Proof.

In the exact sequence

[TABLE]

the first arrow is an isomorphism: it is the identity map from $T_{Z_{0}}S^{[n]}$ to itself. Therefore the second arrow $\iota$ is zero, and $\iota v=0$.

Similarly we can see $s\iota$ from

[TABLE]

But $\iota^{*}$ is the Serre dual of the second arrow in (8.8), which we saw was zero.

Finally $\iota s$ lies in $H^{1}(K_{S})$, which vanishes since $h^{1}(\mathcal{O}_{S})=0$. ∎

So we are now ready to analyse the deformation-obstruction theory of $(E,\phi)$. Let $T^{i}$ be the cohomology groups of the cone (8.5). We have the exact sequence

[TABLE]

We have shown that the third arrow is injective, so $T^{0}=0$. This says that $(E,\phi)$ has no trace-free infinitesimal automorphisms, as we knew already from stability. By Serre duality, the third-from-last map is surjective and $T^{3}=0$. From (8.6) and Lemma 8.7 the sequence splits into

[TABLE]

and its Serre dual for $T^{2}$:

[TABLE]

[TABLE]

8.2. The virtual cycle

In particular we see the fixed (weight 0) part of $T^{1}$ is just $T_{Z_{0}}S^{[n]}$, as expected.

The weight 1 part of $T^{1}$ comes from putting together H^{0}(K_{S})\big{/}\iota H^{0}(I_{0}K_{S}) and $H^{1}(I_{0}K_{S})$. Together we claim these give $\Gamma(K_{S}|_{Z_{0}})$. Although we don’t need this — we only require the K-theory class of $T^{1}$ to compute the invariant (6.3) — for completeness we briefly describe how to understand it.

Digression. We do this by working with the $\mathbb{C}^{*}$-invariant sheaf $\mathcal{E}_{\phi}$ on $X$ which is equivalent, via the spectral construction, to the Higgs pair $(E,\phi)$. It is

[TABLE]

the pushforward from $2S$ to $X$ of the ideal sheaf of $Z_{0}\subset 2S$, all twisted by $K_{S}$. (Here $2S\subset X$ is the thickening of the zero section $S\subset X$ defined by the ideal $I_{S\subset X}^{2}$.) Let $s_{0}$ denote the tautological section of $\pi^{*}K_{S}$ on $X=K_{S}$, vanishing on the zero section $S$, and fix any section $s$ of $K_{S}$. The support of $\mathcal{E}_{\phi}$ is $2S=\{s_{0}^{2}=0\}$; a weight-1 deformation of this (parametrised by $t$) is given by

[TABLE]

splitting $2S$ into the two sections $s_{0}=\pm ts$. Taking the structure sheaf of this deformation and twisting by both $K_{S}$ and the ideal sheaf

[TABLE]

gives a corresponding deformation of $\mathcal{E}_{\phi}$. However, away from $Z_{0}$ it is the trivial deformation to first order. That is, working over $\operatorname{Spec}\,\mathbb{C}[t]/(t^{2})$ we see that (8.10) is just $s_{0}^{2}=0$ — i.e. the same $2S$ as at $t=0$. The deformation does move $Z_{0}$ to first order, however, if and only if $s\not\in I_{Z_{0}\,}$, as can be seen from (8.11). This describes the H^{0}(K_{S})\big{/}H^{0}(I_{0}K_{S}) part of the weight-1 part of the first order deformations $T^{1}$.

Since the deformation is zero away from $Z_{0}$ we do not need $s$ to be a global section of $K_{S}$. Any local section defined near $Z_{0}$ also defines the above deformation — glued to zero away from $Z_{0}$. This describes all of the weight-1 part of the first order deformations $T^{1}$ as $\Gamma(K_{S}|_{Z_{0}})$ — both the deformations H^{0}(K_{S})\big{/}H^{0}(I_{0}K_{S}) coming from global sections $s$, and the quotient $H^{1}(I_{0}K_{S})$ by these.

Now $\Gamma(K_{S}|_{Z_{0}})$ is the fibre over $Z_{0}\in S^{[n]}$ of the rank $n$ bundle $K_{S}^{[n]}$ on $S^{[n]}$, and this bundle is globally the weight 1 part of $T^{1}$. Dually, the (fixed part of the) obstruction bundle on our smooth $\mathbb{C}^{*}$-fixed moduli space $\mathcal{M}_{2}$ is \big{(}K_{S}^{[n]}\big{)}^{*}.

Therefore the virtual cycle that $\mathcal{M}_{2}$ inherits from [GP] is the Euler class of this obstruction bundle:

[TABLE]

Fix the section of $K_{S}$ cutting out the smooth canonical curve $C$. It induces a section of $K_{S}^{[n]}$ with zero locus $C^{[n]}=\operatorname{Sym}^{n}C\subset S^{[n]}$. Since this has the correct dimension, it is Poincaré dual to e\big{(}K_{S}^{[n]}\big{)}, and we find that

[TABLE]

8.3. The virtual normal bundle

Reading off the virtual normal bundle $N^{\operatorname{vir}}$ from the moving parts of the above computations (8.9) gives

[TABLE]

at $Z_{0}\in\mathcal{M}_{2}$. Notice the final term has weight 1 — if Serre duality did not shift weights in this way, this term would have weight 0 and would be the fixed obstruction bundle. Therefore the virtual class $[\mathcal{M}_{2}]^{\operatorname{vir}}$ would be a signed Euler characteristic as before. As it is, the $\mathcal{M}_{2}$ contribution is more interesting.

We only care about its K-theory class, for which it simplifies things to express $R\Gamma(I_{0}K_{S}^{2})$ as $H^{0}(K_{S}^{2})-H^{0}(K_{S}^{2}|_{Z_{0}})$. As $Z_{0}$ moves through $S^{[n]}$, the first term is a fixed $\mathbb{C}^{P_{2}}$, where the plurigenus $P_{2}(S)=h^{0}(K_{S}^{2})=p_{g}(S)+g$ is the constant (8.1). The second term is the fibre of the vector bundle $(K_{S}^{2})^{[n]}$. That is, in $\mathbb{C}^{*}$-equivariant K-theory, $N^{\operatorname{vir}}$ is

[TABLE]

Therefore

[TABLE]

where $c_{s}(E):=1+sc_{1}(E)+\cdots+s^{r}c_{r}(E)$ for a bundle $E$ of rank $r$; when $s=1$ this is $c_{\bullet}$ (the total Chern class).

When we take the degree $n$ part of this and integrate over the virtual cycle (-1)^{n}\big{[}C^{[n]}\big{]} (8.12), only the $t^{0}$ part contributes because (8.13) has total degree $n$ (that is $-\operatorname{rank}(N^{\operatorname{vir}})=n$ equals the virtual dimension of $\mathcal{M}_{2}$). So we can set $t=1$ to get the same answer. We can also use the fact that $C^{[n]}$ is cut out by a transverse section of $K_{S}^{[n]}$, so its normal bundle is the restriction of $K_{S}^{[n]}$. Thus $T_{S^{[n]}}|_{C^{[n]}}=T_{C^{[n]}}\oplus K_{S}^{[n]}|_{C^{[n]}}$ in K-theory, and

[TABLE]

8.4. Tautological classes

To evaluate this integral we express it in terms of two tautological cohomology classes on $C^{[n]}$. Fix a basepoint $c_{0}\in C$. The first class is

[TABLE]

where $C^{[n-1]}\hookrightarrow C^{[n]}$ is the map taking $Z\mapsto Z+c_{0}$.

For the second we use the Abel-Jacobi map

[TABLE]

Tensoring with powers of $\mathcal{O}(c_{0})$ makes the $\operatorname{Pic}^{n}C$ isomorphic for all $n$, so we may pull back the theta divisor from $\operatorname{Pic}^{g-1}C$. Its cohomology class

[TABLE]

maps $\alpha,\beta\in H^{1}(C,\mathbb{Z})$ to $\int_{C}\alpha\wedge\beta$. We also use $\theta$ to denote its pullback $\operatorname{AJ}^{*}\theta$, giving our second tautological class

[TABLE]

The integrals of these classes are given by

[TABLE]

The first equality follows from repeated intersection with the divisor $C^{[n-1]}\subset C^{[n]}$; the last two are Poincaré’s formula [ACGH, Section I.5].

In our application we also need to know the two identities

[TABLE]

and

[TABLE]

from, for example, [ACGH, Section VIII.2].

Plugging these into (8.14) makes the integrand $(-2)^{-P_{2}}2^{n}$ times by

[TABLE]

Therefore our invariant is

[TABLE]

To simplify this integral we notice from (8.17) that

[TABLE]

so whenever $\theta^{i}/i!$ is integrated against only powers of $\omega$ we may replace it by ${g\choose i}\omega^{i}$. In particular,

[TABLE]

for $\alpha$ a power series in $\omega$. The $\sim$ becomes an equality if we integrate both sides over $C^{[n]}$ against a power series in $\omega$. In (8.18) this gives

[TABLE]

Therefore, in terms of generating series,

[TABLE]

8.5. The answer in closed form

The series (8.19) is the diagonal [St2, Section 6.3] of the double generating series

[TABLE]

That is, if we write (8.20) as $\sum_{i,n}a_{in}x^{i}t^{n}$, then our generating series (8.19) is $\sum_{n}a_{nn}q^{n}$.

We can evaluate (8.20) by first summing over $n$ to give

[TABLE]

Since the integrand is independent of $i$, and $\int_{C^{[i]}}\omega^{j}=\delta_{ij}$, the operator $\sum_{i=0}^{\infty}x^{i}\int_{C^{[i]}}$ simply acts by setting $\omega$ to be $x$. So we get

[TABLE]

To find the diagonal (8.19) of this series we substitute $t=q/x$ and consider the integral [St2, Section 6.3]

[TABLE]

around a small loop containing only those poles which tend to the origin as $q\to 0$. Thus (8.19) is the residue of

[TABLE]

at the root

[TABLE]

of the quadratic $(1+q)x^{2}-(1+q)x-2q$ in $x$. This is

[TABLE]

where $x_{1}=\frac{1}{2}\left(1+\sqrt{1+\frac{8q}{1+q}}\,\right)$ is the other root. Substituting in $x_{1}-x_{0}=\sqrt{1+\frac{8q}{1+q}}$ and (1-2x_{0})^{g}=\big{(}1+\frac{8q}{1+q}\big{)}^{g/2}, and multiplying out the denominator this becomes

[TABLE]

Since $x_{0}$ was chosen as a root of $x_{0}^{2}-x_{0}-\frac{2q}{1+q}=0$, and is given by the formula (8.21), we find the terms in the denominator are

[TABLE]

Rearranging and substituting into (8.19) finally gives the following answer.

Proposition 8.22**.**

[TABLE]

where $c=(-1)^{p_{g}(S)+g}\cdot 2^{-p_{g}(S)-1}$. Expanding in powers of $q$ gives

[TABLE]

Even after multiplying by a shift $q^{-s}$, this is clearly not modular [BE].

8.6. Euler characteristics of nested Hilbert schemes

If instead of the virtual theory we use Euler characteristics252525It is reasonable to hope the Behrend function is 1 at all $\mathbb{C}^{*}$ fixed points; cf. [TT2, Proposition 5.9] and [MT, Corollary 5.8]. then by Göttsche’s famous calculation we get the generating series

[TABLE]

which is modular after multiplying by the shift $q^{-e(S)/24}$. In fact we can do better and sum over all nested Hilbert schemes — corresponding to all partitions, not just the horizontal terms — and again get something close to a modular form. By [St1, Equation 1.128] the generating series of nested partitions is

[TABLE]

Therefore by similar arguments to Göttsche’s we find the generating series of the bare Euler characteristic versions of the $\mathsf{vw}$ invariants is

[TABLE]

After multiplying by the shift $q^{-s}=q^{-\frac{e(S)}{12}}$ this becomes

[TABLE]

So up to the rational function $(1-q)^{e(S)}$ this is the weight $-e(S)$ modular form (with character) $\eta(q)^{-2e(S)}$.

We do not understand the significance of the rational factor, the modular form, nor the weight (which is twice the standard Vafa-Witten prediction). Note that a similar formula has recently been obtained for nested Hilbert schemes with 3 steps [Bo], relevant to $SU(3)$ Vafa-Witten theory.

For some time we found this calculation suggestive that $\mathsf{vw}$ is the “correct” Vafa-Witten invariant, but as explained in Section 1.11 we no longer think this. For one thing, the formula (8.25) depends only on $c_{2}(S)$, whereas the predictions in [VW, Equation 5.38] also involve $c_{1}^{2}(S)$.

The solution seems to be that we have to add in the non-horizontal terms — the contributions of $S^{[i,j]}$ with nonzero $i$ — to (8.23). Then we appear to get the modular forms predicted by Vafa and Witten, as we show in the next few Sections.

8.7. Vertical terms

Having dealt with the length $(Z_{1})=0$ component of $\mathcal{M}_{2}$, we now turn to the other extreme: the length $(Z_{1})=$ length $(Z_{0})$ component. In the description of Lemma 8.3, the Higgs field $\Phi\colon I_{0}\to I_{1}$ must be an isomorphism, so we get the Hilbert scheme

[TABLE]

of Higgs pairs

[TABLE]

where $Z\subset S$ is a 0-dimensional subscheme of length $n=c_{2}(E)/2$. The torsion sheaf $\mathcal{E}_{\phi}$ on $X$ that this corresponds to is the twist by $\pi^{*}K_{S}$ of

[TABLE]

To work out the deformation theory it is convenient to work with the description (8.26). By the exact sequence

[TABLE]

we find

[TABLE]

The section of $\mathcal{O}(2S)$ cutting out $2S\subset X$ annihilates $\mathcal{F}_{Z}$, so the second arrow is zero. By adjunction and $\pi_{*}\hskip 0.6pt\mathcal{F}=I_{Z}\,\oplus\,I_{Z}\otimes\!K_{S}^{-1}\cdot\mathfrak{t}^{-1}$ we therefore find

[TABLE]

The $SU(2)$ perfect obstruction theory $R\operatorname{Hom}_{X}(\mathcal{F}_{Z},\mathcal{F}_{Z})_{\perp}[1]$ (the derived dual of Theorem 6.1) comes from taking trace-free parts of the first and last terms, so $\operatorname{Hom}_{\perp}=0=\operatorname{Ext}^{3}_{\perp}$,

[TABLE]

and $\operatorname{Ext}^{2}_{\perp}$ is its dual tensored with $\mathfrak{t}$ (by Serre duality and $\omega_{X}\cong\mathcal{O}_{X}\cdot\mathfrak{t}^{-1}$). The first term of (8.27) is just $T_{Z}S^{[n]}$, the fixed part of the deformations. The last term vanishes, so by duality so does the fixed part of the obstructions. Therefore

[TABLE]

and we can read off the virtual normal bundle:

[TABLE]

Integrating the reciprocal of its equivariant Euler class over $S^{[n]}$ is a project for future work, but in the case $n=1$ it is easy enough. We get

[TABLE]

Here we have repeatedly used the isomorphism $\operatorname{Ext}^{1}(I_{p},I_{p})\cong T_{p}S$ and its Serre dual, for any point $p\in S$. We have also computed $\operatorname{Ext}^{1}(I_{p},I_{p}\otimes K_{S}^{2})=T_{p}S\otimes K_{S}^{2}$ from the local-to-global spectral sequence $\oplus_{i+j=k\,}H^{i}(\curly Ext^{j})\ \ext@arrow 0359\Rightarrowfill@{}{\hskip 8.53581pt}\ \operatorname{Ext}^{k}$ (using that $H^{1}(\curly H\!om)=H^{2}(\curly H\!om)=0$ in this case). It follows from Serre duality that $\operatorname{Ext}^{1}(I_{p},I_{p}\otimes K_{S}^{-1})=T_{p}S\otimes K_{S}^{-1}$ (proving directly the vanishing of the differential $H^{0}(\curly Ext^{1})\to H^{2}(\curly H\!om)$ would be more troublesome in this case).

In the notation of (8.13) this is

[TABLE]

Since only the $t^{0}$ term contributes, we may set $t=1$ to give

[TABLE]

where $c_{i}:=c_{i}(S)$. Substituting in the identities

[TABLE]

gives

[TABLE]

Remark. Finally we note the above calculations can be generalised further to the rank $r$ case, where the “vertical” component of the $\mathbb{C}^{*}$-fixed moduli space

[TABLE]

parametrises $\mathbb{C}^{*}$-fixed torsion sheaves

[TABLE]

The corresponding Higgs pairs have rank $r$, determinant $K_{S}^{-r(r-1)/2}$ and $c_{2}=r+r(r-1)(r-2)(3r-1)/24$. A similar analysis to the above shows that when $g\geq 2$ the invariant is

[TABLE]

8.8. One mixed term

Finally we compute the contribution from the other component with $c_{2}=3$, namely the nested Hilbert scheme

[TABLE]

The torsion sheaf $\mathcal{E}$ corresponding to $Z_{1}\subset Z_{0}\subset S$ (with ideal sheaves $I_{1}\supset I_{0}$ respectively) is an extension

[TABLE]

Here $\iota\colon S\hookrightarrow X$ is the zero section, and multiplication by the tautological section of $\pi^{*}K_{S}$ vanishing on $S\subset X$ acts on $\mathcal{E}$ by the inclusion $\iota_{*}I_{0}\subset\iota_{*}I_{1}$. From (8.29) we see that in equivariant K-theory, the class of $R\operatorname{Hom}(\mathcal{E},\mathcal{E})[1]$ is the same as that of

[TABLE]

Using $R\operatorname{Hom}_{X}(\iota_{*}A,\iota_{*}B)=R\operatorname{Hom}_{S}(A,B)\,\oplus\,R\operatorname{Hom}_{S}(A,B\otimes K_{S})\mathfrak{t}[-1]$ this becomes

[TABLE]

where $\langle\ \cdot\ ,\ \cdot\ \rangle$ denotes the K-theory class of $R\operatorname{Hom}_{S}(\ \cdot\ ,\ \cdot\ )$.

To get the $SU(2)$ deformation-obstruction theory of Theorem 6.1 we remove a copy of $H^{*}(\mathcal{O}_{S})$ from $\langle I_{0},I_{0}\rangle+\langle I_{1},I_{1}\rangle$ (via trace), and a copy of $H^{*}(K_{S})\mathfrak{t}$ from $\langle I_{0},I_{0}\otimes K_{S}\rangle\mathfrak{t}+\langle I_{1},I_{1}\otimes K_{S}\rangle\mathfrak{t}$. In particular the fixed part of the obstruction theory has the same K-theory class as

[TABLE]

(where the suffix 0 denotes trace-free), and the virtual normal bundle has the same K-theory class as

[TABLE]

More generally this works over all of $S^{[2,1]}$ instead of at a single point. Let $\mathcal{E}$ denote the universal sheaf on $X\times S^{[2,1]}$, let $I_{i}$ denote the universal ideal sheaves on $S\times S^{[2,1]}$, and let $\pi_{X},\,\pi_{S}$ be the projections from these spaces to $S^{[2,1]}$. Then the (Serre dual) obstruction theory of $S^{[2,1]}$ of Theorem 6.1,

[TABLE]

has a class in $K^{\mathbb{C}^{*}\!}(S^{[2,1]})$ which is given by the same formula when we use $\langle\ \cdot\ ,\ \cdot\ \rangle$ to denote the K-theory class of $R\curly H\!om_{\pi_{S}}(\ \cdot\ ,\ \cdot\ )$. Thus (8.30) is $(E^{\bullet})^{\vee,\mathrm{\hskip 0.6ptfix}}$ and (8.31) is $(E^{\bullet})^{\vee,\mathrm{\hskip 0.6ptmov}}$.

We use the notation

[TABLE]

for $S^{[2,1]}\cong\operatorname{Bl}_{\Delta_{S}}(S\times S)$. Here $S^{[2]}$ parametrises $Z_{0}$ and $S_{1}=S$ parametrises $Z_{1}$. The support of $I_{Z_{1}}/I_{Z_{0}}$ is a single point parametrised by $S_{2}$. We explain and justify these claims as follows.

Consider $S_{1}$ (respectively $S_{2}$) as parametrising points $Z_{1}\subset S$ (respectively $Z_{2}\subset S$). There are corresponding universal points $\mathcal{Z}_{i}\subset S_{i}\times S$. Pulling them back by $\pi_{1}\times\operatorname{id}_{S},\,\pi_{2}\times\operatorname{id}_{S}$ we get (by a small abuse of notation) universal points

[TABLE]

Set

[TABLE]

Since $\mathcal{Z}_{1}\cap\mathcal{Z}_{2}$ is a divisor in both $\mathcal{Z}_{i}$ (it is a copy of the exceptional divisor $E\subset\operatorname{Bl}_{\Delta_{S}}(S\times S)$) it follows that $\mathcal{Z}_{0}$ is a flat family of subschemes of $S$ over $\operatorname{Bl}_{\Delta_{S}}(S\times S)$. Together with the subfamily $\mathcal{Z}_{1}\subset\mathcal{Z}_{0}$ we get a flat family of flags of subschemes of $S$ of lengths 1 and 2 respectively, with a classifying map $\operatorname{Bl}_{\Delta_{S}}(S\times S)\to S^{[2,1]}$. To construct the inverse map, note $\mathcal{Z}_{0}/S^{[2,1]}$ defines a classifying map $S^{[2,1]}\to S^{[2]}$, while the subscheme $\mathcal{Z}_{1}\subset\mathcal{Z}_{0}/S^{[2,1]}$ defines classifying map from $S^{[2,1]}$ to the total space of the universal subscheme over $S^{[2]}$, which is its double cover $\operatorname{Bl}_{\Delta_{S}}(S\times S)$.

From $\mathcal{Z}_{1}\cap\mathcal{Z}_{2}\cong E$ we get the exact sequence

[TABLE]

and so

[TABLE]

Using this (8.30) becomes

[TABLE]

which we can write as

[TABLE]

using Serre duality to compute the last term $\curly Ext^{2}_{\pi_{S}}(\mathcal{O}_{\mathcal{Z}_{2}},\mathcal{O}_{\mathcal{Z}_{2}})=K_{S_{2}}^{-1}$. Finally, writing $I_{1}=\mathcal{O}-\mathcal{O}_{\mathcal{Z}_{1}}$ in K-theory, we obtain

[TABLE]

where we have suppressed some obvious (flat) pullback maps.

To compute the fourth term we can work on $S_{1}\times S_{2}$ (and later pull back to $\operatorname{Bl}_{\Delta_{S}}(S\times S)$ by using Bondal-Orlov’s basechange [BO, Lemma 1.3]). The universal subschemes $\mathcal{Z}_{1},\,\mathcal{Z}_{2}\subset(S_{1}\times S_{2})\times S$ intersect transversally in the small diagonal $\delta_{S}\subset S^{\times 3}$. Therefore $R\curly H\!om(\mathcal{O}_{\mathcal{Z}_{1}},\mathcal{O}_{\mathcal{Z}_{2}})\cong\mathcal{O}_{\delta_{S}}\otimes\Lambda^{2}N[-2]$, where $N$ is the normal bundle to $\mathcal{Z}_{1}$. Pushing down to $S_{1}\times S_{2}$ we get $\mathcal{O}_{\Delta_{S}}\otimes\Lambda^{2}N[-2]$. By a standard Koszul resolution argument the pullback of $\mathcal{O}_{\Delta_{S}}$ to $\operatorname{Bl}_{\Delta_{S}}(S\times S)$ has $h^{0}=\mathcal{O}_{E}$ and $h^{-1}=\mathcal{O}_{E}(E)\otimes\Lambda^{2}N^{*}$, so using $\Lambda^{2}N\cong K_{S}^{-1}$ we get

[TABLE]

Plugging this into (8.33) gives

[TABLE]

There is an obvious exact sequence on $S^{[2,1]}\cong\operatorname{Bl}_{\Delta_{S}}(S_{1}\times S_{2})$,

[TABLE]

Since $N_{\Delta_{S}}\cong T_{S}$ has rank 2, its quotient by $\mathcal{O}_{E}(E)$ is $\Lambda^{2}N_{\Delta_{S}}(-E)\cong K_{S}^{-1}|_{E}(-E)$. Therefore $T_{S_{1}\times S_{2}}=T_{S^{[2,1]}}+K_{S}^{-1}|_{E}(-E)$ in K-theory, so (8.35) simplifies to give

[TABLE]

at least at the level of K-theory. And $S^{[2,1]}$ is smooth, so its virtual cycle is just the top Chern class of its obstruction bundle:

[TABLE]

In particular

[TABLE]

Using (8.32) we rewrite (8.31) as

[TABLE]

Cancelling, using Serre duality, and substituting $[I_{1}]=[\mathcal{O}]-[\mathcal{O}_{\mathcal{Z}_{1}}]$ gives

[TABLE]

Substituting in (8.34) yields

[TABLE]

Finally this can be rewritten

[TABLE]

for any $i,j,k\in\{1,2\}$. Now we use $e(F\otimes\mathfrak{t}^{w})=(wt)^{r}c_{1/wt}(F)$ for any complex $F$ of rank $r$, and note that $N^{\operatorname{vir}}$ has rank $-3$, so only terms in $t^{0}$ contribute to the integral (8.36). We may therefore set $t=1$ to give

[TABLE]

Multiplying out gives a polynomial in $[E]$. The constant $[E]^{0}$ term can be easily integrated on $S_{1}\times S_{2}$. The $[E]^{1}$ term is an integral on $E\cong\mathbb{P}(T_{S})\to\Delta_{S}$ of Chern classes pulled back from $\Delta_{S}$, so it is zero. The $[E]^{\geq 2}$ terms can be evaluated using the Grothendieck formula [E]\big{|}_{E}^{2}-c_{1}(S)\cdot[E]\big{|}_{E}+c_{2}(S)=0 on the projective bundle $E\cong\mathbb{P}(T_{S})\to\Delta_{S}$. Pushing down to $\Delta_{S}$ using the projection formula — and the fact that [E]\big{|}_{E}\in H^{2}(E) pushes down to $-1\in H^{0}(\Delta_{S})$ — gives

[TABLE]

After much cancellation the final result is the following.

Proposition 8.37**.**

[TABLE]

8.9. Comparison with Vafa-Witten prediction

We can rewrite the previous formula in the notation of [VW] as

[TABLE]

where $\nu=\chi(\mathcal{O}_{S})=\frac{1}{12}(c_{1}(S)^{2}+c_{2}(S))=p_{g}(S)+1$ and $g=c_{1}(S)^{2}+1$ is the canonical genus. Adding (8.38) to the horizontal (8.24) and vertical (8.28) terms gives the full generating series of monopole branch contributions up to degree 3:

[TABLE]

[TABLE]

On the other hand, the second term on the first line of [VW, Equation 5.38] is, in their notation,

[TABLE]

where by [VW, Equation 5.16],

[TABLE]

Therefore (8.40) is $q^{(1-g)/6-\nu}$ times by

[TABLE]

Ignoring terms of $O(q^{4})$ this is

[TABLE]

which can be expanded as

[TABLE]

Multiplying out, this is

[TABLE]

which agrees perfectly262626Up to the sign $(-1)^{-g-\nu+1}=(-1)^{\operatorname{vd}}$ (7.2) of footnote 8. with (8.39) up to $q^{3}$. We find this completely extraordinary: while Vafa and Witten do briefly consider a nonzero Higgs field in [VW, Equation 2.70], it is on a bundle rather than a sheaf. The components we have calculated with in this section consist entirely of non-locally-free sheaves not considered at all in [VW]. The physics reasoning (“cosmic strings”) used to derive [VW, Equation 2.70] is still much more powerful than our tools more than 20 years on.

Remark 8.41**.**

The first term of [VW, Equation 5.38] seems to be the $\det E=\mathcal{O}_{S}$ contribution to the generating series, to which we plan to return in the future. The other terms in their equation are what Göttsche and Kool [GK] conjecture to be the contribution of the instanton branch $\mathcal{M}_{L}$ (\tagform@1.9) — i.e. the virtual signed Euler characteristic (7.3) of the moduli space of stable sheaves on $S$ with determinant $L$. Using Mochizuki’s work [Mo] they prove a universality result similar to those in [EGL, GNY]: this signed virtual Euler characteristic of $\mathcal{M}_{L}$ is a universal expression in 7 topological constants on any surface $S$ with $h^{1}(\mathcal{O}_{S})=0$ and $p_{g}(S)>0$. Since it is universal, the expression can be calculated on toric surfaces, which they do by torus localisation and computer calculation.272727In fact they do even better, computing the (virtual) $\chi_{y}$-genus refinement of the virtual Euler characteristic. The result indeed reproduces the other terms of [VW, Equation 5.38] for powers $q^{c_{2}(E)}$ of $q$ up to $c_{2}(E)=30$.

Remark 8.42**.**

In [GT] it is shown how to rewrite integrals over the virtual cycle of a nested Hilbert schemes as integrals over a product of ordinary Hilbert schemes, with the insertion of a Carlsson-Okounkov operator. The latter can be re-expressed in terms of Grojnowski-Nakajima operators, so this may give an easier way to compute the integrals of this Section in higher degrees.