Nested Hilbert schemes on surfaces: Virtual fundamental class

TL;DR

This paper constructs virtual fundamental classes on nested Hilbert schemes of points and curves on surfaces, linking them to Seiberg-Witten, stable, and Donaldson-Thomas theories, and provides explicit formulas for related integrals.

Contribution

It introduces the construction of virtual fundamental classes on nested Hilbert schemes, connecting them to various enumerative invariants and deriving explicit integral formulas.

Findings

Virtual fundamental classes recover Seiberg-Witten and stable theory classes.

Integrals over nested Hilbert schemes relate to products of Hilbert schemes.

Explicit formulas are obtained via Carlsson-Okounkov vertex operator methods.

Abstract

We construct virtual fundamental classes on nested Hilbert schemes of points and curves in complex nonsingular projective surfaces. These classes recover the virtual classes of Seiberg-Witten theory as well as the (reduced) stable theory, and play a crucial role in the reduced Donaldson-Thomas theory of local-surface-threefolds that we study in [GSY17b] (arXiv:1807.05697). We show that certain integrals against the virtual fundamental classes of punctual nested Hilbert schemes are expressed as integrals over the products of the Hilbert scheme of points. We are able to find explicit formulas for some of these integrals by relating them to Carlsson-Okounkov's vertex operator formulas.

Equations

$$(Z_1,Z_2,\dots ,Z_r),(C_1,\dots ,C_{r-1})$$

$$I_{Z_i}(-C_i)\subseteq I_{Z_{i+1}},$$

$$I_{Z_1}(-C_1-\cdots -C_{r-1})\subseteq I_{Z_2}(-C_2-\cdots -C_{r-1})\subseteq \cdots \subseteq I_{Z_r}\subseteq {\mathcal{O}}_S.$$

$$[S_{\mathbf{\beta }}^{[\mathbf{n}]}{]}^{\mathrm{vir}}\in A_d(S_{\mathbf{\beta }}^{[\mathbf{n}]}),d=n_1+n_r+\frac{1}{2}\sum _{i=1}^{r-1}{\beta }_i\cdot ({\beta }_i-c_1({\omega }_S)).$$

$$[S_{\beta }^{[n_1,n_2]}{]}_{\mathrm{red}}^{\mathrm{vir}}\in A_{n_1+n_2+\frac{1}{2}\beta \cdot (\beta -c_1({\omega }_S))+p_g(S)}(S_{\beta }^{[n_1,n_2]})$$

$$S^{[n_1\ge n_2]}:=S_{\beta =0}^{[n_1,n_2]}$$

$$\iota :S^{[n_1\ge n_2]}\hookrightarrow S^{[n_1]}\times S^{[n_2]}$$

$$\mu =({\mu }_1\ge {\mu }_2\ge {\mu }_3\ge \dots )\text{such that}d=\sum _i{\mu }_i.$$

$$\left\{({\mu }_P^{\prime }\subseteq {\mu }_P{)}_{P\in S^{\mathbf{T}}}\mid {\mu }_P^{\prime }⊢d_P^{\prime },{\mu }_P⊢d_P,n_2=\sum _P{d}_P^{\prime },n_1=\sum _P{d}_P\right\}.$$

$${\mathrm{tr}}_{{\mathcal{T}}_Q^{\mathrm{vir}}}(t_1,t_2)=\sum _{P\in S^{\mathbf{T}}}{\mathsf{V}}_P,$$

$$\mathsf{E}^{n_{1},n_{2}}_{M}:=\sum_{i=0}^{2}(-1)^{i}\big{(}H^{i}(S,M)\otimes\mathcal{O}-\mathcal{E}xt^{i}_{\pi}(\mathcal{I}_{1},\mathcal{I}_{2}\otimes p^{*}M)\big{)}.$$

$${\iota }_{\ast }[S^{[n_1\ge n_2]}{]}^{\mathrm{vir}}=[S^{[n_1]}\times S^{[n_2]}]\cap c_{n_1+n_2}({\mathsf{E}}^{n_1,n_2}).$$

$$(5)\subset {\mathbb{P}}^3,(3,3)\subset {\mathbb{P}}^4,(4,2)\subset {\mathbb{P}}^4,(3,2,2)\subset {\mathbb{P}}^5,(2,2,2,2)\subset {\mathbb{P}}^6.$$

$${\int }_{S^{[n_1]}\times S^{[n_2]}}-\cup c_{n_1+n_2}({\mathsf{E}}_M^{n_1,n_2})$$

$$\sum _{n_1\ge n_2\ge 0}(-1{)}^{n_1+n_2}$$

$$\prod _{n>0}{\left(1-q_2^{n-1}{q}_1^n\right)}^{⟨K_S,K_S-M⟩}{\left(1-q_1^n{q}_2^n\right)}^{⟨K_S-M,M⟩-e(S)},$$

$$A_0\oplus A_1\to B_0\oplus B_1,A_0\oplus A_1\stackrel{\mathrm{id}\oplus s}{\to }{A}_0\oplus C_1$$

$$k^0\left({\mathbb{L}}_{(B_0\oplus B_1)/(A_0\oplus A_1)}^{\bullet ,\text{gr}}\right)\cong {\mathbb{L}}_{B_0/A_0}^{\bullet },k^1\left({\mathbb{L}}_{(A_0\oplus C_1)/(A_0\oplus A_1)}^{\bullet ,\text{gr}}\right)\cong \mathrm{coker}(s).$$

$$A_0\oplus A_1\to B_0\oplus B_1\to C_0\oplus C_1$$

$${\mathbb{L}}_{(B_0\oplus B_1)/(A_0\oplus A_1)}^{\bullet ,\text{gr}}{\otimes }_{(B_0\oplus B_1)}(C_0\oplus C_1)\to {\mathbb{L}}_{(C_0\oplus C_1)/(A_0\oplus A_1)}^{\bullet ,\text{gr}}\to {\mathbb{L}}_{(C_0\oplus C_1)/(B_0\oplus B_1)}^{\bullet ,\text{gr}}$$

$$[Z]=\beta ,c_2(I_Z)=m.$$

$$\mathrm{div}:S_{\beta }^{[m]}\to S_{\beta },\det :S_{\beta }^{[m]}\to \mathrm{Pic}(S),\mathrm{pts}:S_{\beta }^{[m]}\to S^{[m]},$$

$$T_{S^{[m]}}\cong\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[m]},\mathcal{O}_{\mathcal{Z}^{[m]}}\big{)}\cong\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[m]},\mathcal{I}^{[m]}\big{)}_{0}[1]\cong\mathcal{E}xt^{1}_{\pi}\big{(}\mathcal{I}^{[m]},\mathcal{I}^{[m]}\big{)}_{0},$$

$$\iota :S_{\mathbf{\beta }}^{[\mathbf{n}]}\hookrightarrow S_{{\beta }_1}^{[n_1]}\times \cdots \times S_{{\beta }_{r-1}}^{[n_{r-1}]}\times S^{[n_r]}$$

$${\mathcal{I}}_1(-{\mathcal{D}}_1-{\mathcal{D}}_2-\cdots -{\mathcal{D}}_{r-1})\subseteq {\mathcal{I}}_2(-{\mathcal{D}}_2-\cdots -{\mathcal{D}}_{r-1})\subseteq \cdots \subseteq {\mathcal{I}}_{r-1}(-{\mathcal{D}}_{r-1})\subseteq {\mathcal{I}}_r,$$

$$(Z_1,\dots ,Z_r)\in S^{[n_1]}\times \cdots \times S^{[n_r]},(C_1,\dots ,C_{r-1})\in S_{{\beta }_1}\times \cdots \times S_{{\beta }_{r-1}}$$

$${\Phi }_i:{\mathcal{I}}^{[n_i]}\to {\mathcal{I}}_{{\beta }_i}^{[n_{i+1}]}1\le i<r$$

$$\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i+1}]},\mathcal{I}^{[n_{i+1}]}\big{)}\xrightarrow{\Xi_{i}}\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},\mathcal{I}^{[n_{i+1}]}_{\beta_{i}}\big{)},$$

$$\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},\mathcal{I}^{[n_{i}]}\big{)}\xrightarrow{\Xi^{\prime}_{i}}\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},\mathcal{I}^{[n_{i+1}]}_{\beta_{i}}\big{)}.$$

$$\displaystyle\bigoplus_{i=1}^{r}\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},\mathcal{I}^{[n_{i}]}\big{)}\xrightarrow{\tiny\left[\begin{array}[]{cccccc}-\Xi^{\prime}_{1}&\Xi_{1}&0&\dots&0&0\\ 0&-\Xi^{\prime}_{2}&\Xi_{2}&0&\dots&0\\ \cdot&\cdot&\dots&\cdot&\cdot&\cdot\\ \cdot&\cdot&\dots&\cdot&\cdot&\cdot\\ \cdot&\cdot&\dots&\cdot&\cdot&\cdot\\ 0&0&\dots&0&-\Xi^{\prime}_{r-1}&\Xi_{r-1}\\ \end{array}\right]}\bigoplus_{i=1}^{r-1}\mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},\mathcal{I}^{[n_{i+1}]}_{\beta_{i}}\big{)}.$$

$${\left[\underset{i=1}{\overset{r}{⨁}}\mathbf{R}\mathcal{H}o{m}_{\pi }\left({\mathcal{I}}^{[n_i]},{\mathcal{I}}^{[n_i]}\right)\right]}_0:$$

Full text

1. 1 Introduction

2. 1.1 Nested Hilbert schemes on surfaces

3. 1.2 Punctual nested Hilbert schemes

4. 1.3 Notation and conventions

5. 2 Nested Hilbert schemes on surfaces

6. 2.1 $2$-step nested Hilbert schemes

7. 2.2 Gillam’s construction

8. 2.3 Reduced perfect obstruction theory

9. 2.4 Invariants

10. 2.5 Proof of Theorem 1 (Proposition 2.2)

11. 3 Special cases

12. 4 Punctual nested Hilbert schemes

13. 4.1 Toric surfaces

14. 4.2 Proof of Theorem 3

15. 4.3 Relative nested Hilbert schemes

16. 4.4 Double point relation

17. 5 Vertex operator formulas and proof of Theorem 4

1 Introduction

Hilbert scheme of points on a nonsingular surface $S$ have been vastly studied. They are nonsingular varieties with rich geometric structures some of which have applications in physics (see [N99] for a survey). We are mainly interested in the enumerative geometry of Hilbert schemes of points [G90, L99, CO12, GS16]. This has applications in curve counting problems on $S$ [LT14, R17]. The first two authors of this paper have studied the relation of some of these enumerative problems to the Donaldson-Thomas theory of 2-dimensional sheaves in threefolds and to S-duality conjectures [GS13]. In contrast, Hilbert scheme of curves on $S$ can be badly behaved and singular. They were studied in detail by Dürr-Kabanov-Okonek [DKO07] in the context of Poincaré invariants (algebraic Seiberg-Witten invariants [CK13]). More recently, the stable pair invariants of surfaces have been employed in the context of curve counting problems [PT10, MPT10, KT14, KST11]. The moduli space of stable pairs on $S$ is identified with the Hilbert scheme of points on curves on $S$, and so is a nested Hilbert scheme on $S$. This paper studies general nested Hilbert schemes of points and curves on $S$. The geometry of nested Hilbert scheme of points was studied in [C98]. We construct a reduced and a non-reduced perfect obstruction theories for the nested Hilbert scheme of curves and points on $S$. The reduced one is an extension of the reduced perfect obstruction theory of stable pairs worked out in [KT14], and its construction closely follows their technique. Our main application of the non-reduced perfect obstruction theory is in the study of (reduced!) Donaldson-Thomas theory of local surfaces that is carried out in [GSY17b]. The non-reduced perfect obstruction theory appears in the study of Vafa-Witten theory of $S$ [TT17], and also in the work of A. Negut [N17].

1.1 Nested Hilbert schemes on surfaces

Suppose that $r\in\mathbb{Z}_{>0}$, ${\boldsymbol{n}}:=n_{1},n_{2},\dots,n_{r}\in\mathbb{Z}_{\geq 0}$, and ${\boldsymbol{\beta}}:=\beta_{1},\dots,\beta_{r-1}\in H^{2}(S,\mathbb{Z})$. We denote the corresponding nested Hilbert scheme $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$, whose underlying set of closed points consists of tuples of subschemes of $S$

[TABLE]

where $Z_{i}$ is a 0-dimensional of length $n_{i}$, and $C_{i}$ a divisor with $[C_{i}]=\beta_{i}$, and for any $i<r$,

[TABLE]

where $I_{Z_{i}}\subseteq\mathcal{O}_{S}$ is the ideal sheaf of $Z_{i}$. 111Taking double dual shows that an inclusion of ideals $I_{Z_{i}}(-D_{i})\subseteq I_{Z_{i+1}}(-D_{i+1})\subset\mathcal{O}_{S}$ is equivalent to (1) with $C_{i}=D_{i}-D_{i+1}$. So we are not losing information by starting with $r-1$ effective divisors $C_{1},\dots,C_{r-1}$ instead of $r$ of them. In other words, a closed point of $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$ corresponds to a chain of subschemes of $S$ given by the ideals

[TABLE]

To define invariants of $S$ arising from $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$ (see Section 2.4), we construct a virtual fundamental class $[S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}]^{\operatorname{vir}}$. More precisely, we construct a natural (non-reduced) perfect obstruction theory over $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$. This is done by studying the deformation/obstruction theory of the maps of coherent sheaves given by the natural inclusions (1) following Illusie [Ill] and Joyce-Song [JS12] 222Another approach would be to study the deformation/obstruction theory of quotients $I_{Z_{i+1}}/I_{Z_{i}}(-C_{i})$ following the work of Gillam [G11], which in turn also uses a great deal of [Ill]. We give a brief sketch of this approach in Subsection 2.2.. As we will see, this in particular provides a uniform way of studying all known perfect obstruction theories on the Hilbert schemes of points and curves, as well as the stable pair moduli spaces on $S$. The main result of the paper is (Propositions 2.2, 2.5 and Corollary 2.6):

Theorem 1**.**

Let $S$ be a nonsingular projective surface over $\mathbb{C}$ and $\omega_{S}$ be its canonical bundle.The nested Hilbert scheme $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$ with $r\geq 2$ carries a natural perfect obstruction theory with the virtual fundamental class

[TABLE]

For $r\geq 2$ and $\beta_{i}=0$, $S^{[{\boldsymbol{n}}]}:=S^{[{\boldsymbol{n}}]}_{(0,\dots,0)}$ is the nested Hilbert scheme of points on $S$ parameterizing flags of 0-dimensional subschemes $Z_{r}\subseteq\dots\subseteq Z_{2}\subseteq Z_{1}\subset S$. $S^{[{\boldsymbol{n}}]}$ is in general singular of actual dimension $2n_{1}$.

We are specifically interested in the case $r=2$ in this paper: $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}=S^{[n_{1},n_{2}]}_{\beta}$ for some $\beta\in H^{2}(S,\mathbb{Z})$. The cases $S_{\beta}:=S^{[0,0]}_{\beta}$ and $S^{[0,n_{2}]}_{\beta}$ (when $\beta\neq 0$) are respectively isomorphic to the Hilbert scheme of divisors and the moduli space of stable pairs in classes $\beta$. The comparison of the virtual class of Theorem 1 to other known virtual classes is carried out in Proposition 3.1.

In certain cases, we construct a reduced virtual fundamental class

[TABLE]

by reducing the perfect obstruction theory in Theorem 1 (Propositions 2.7, 2.10). The reduced virtual fundamental class $[S^{[0,n_{2}]}_{\beta}]^{\operatorname{vir}}_{\operatorname{red}}$ match with the reduced virtual fundamental class of stable pair theory constructed in [KT14] (Proposition 3.1).

1.2 Punctual nested Hilbert schemes

Sections 4 and 5 are devoted to study of the nested Hilbert schemes of points

[TABLE]

in more detail. Let

[TABLE]

be the natural inclusion. If $S$ is toric with the torus $\mathbf{T}$ and the fixed set $S^{\mathbf{T}}$, in Section 4.1 we provide a purely combinatorial formula for computing $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ by torus localization along the lines of [MNOP06]. Let $d$ be a positive integer, by a partition $\mu$ of $d$, denoted by $\mu\vdash d$, we mean a finite sequence of positive integers

[TABLE]

The number of $\mu_{i}$’s is called the length of the partition $\mu$, and is denoted by $\ell(\mu)$. If $\mu^{\prime}\vdash d^{\prime}$ with $d^{\prime}\leq d$, we say $\mu^{\prime}\subseteq\mu$ if $\ell(\mu^{\prime})\leq\ell(\mu)$ and $\mu^{\prime}_{i}\leq\mu_{i}$ for all $1\leq i\leq\ell(\mu^{\prime})$.

Theorem 2**.**

For a toric nonsingular surface $S$ the $\mathbf{T}$-fixed set of $S^{[n_{1},n_{2}]}$ is isolated and in bijective correspondence by the tuples of nested partitions:

[TABLE]

Moreover, the $\mathbf{T}$-character of the virtual tangent bundle $\mathcal{T}^{\operatorname{vir}}$ of $S^{[n_{1}\geq n_{2}]}$ at the fixed point $Q=(\mu^{\prime}_{P}\subseteq\mu_{P})_{P\in S^{\mathbf{T}}}$ is given by

[TABLE]

where $t_{1},t_{2}$ are the torus characters and $\mathsf{V}_{P}$ is a Laurent polynomial in $t_{1},t_{2}$ that is completely determined by the partitions $\mu^{\prime}_{P}$ and $\mu_{P}$ and is given by the right hand side of formula (36).

The bijection in Theorem 2 is deduced from the standard bijection between the set of the fixed points of punctual Hilbert schemes on the affine plane and the set of monomial ideals of finite colegnths in the polynomial rings in two variables.

Let $\mathcal{I}_{1},\mathcal{I}_{2}$ be the universal ideal sheaves on $S\times S^{[n_{1}]}\times S^{[n_{2}]}$, and let $\pi$ be the projection to the last two factors $S^{[n_{1}]}\times S^{[n_{2}]}$ and $p$ be the projection to $S$. Following [CO12], for any line bundle $M$ on $S$, we define a $K$-theory class on $S^{[n_{1}]}\times S^{[n_{2}]}$ by

[TABLE]

If $M$ is trivial we drop it from the notation (see Definition 4.2). When $S$ is toric, by torus localization, we can express $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ in terms of the fundamental class of the product of Hilbert schemes $S^{[n_{1}]}\times S^{[n_{2}]}$ (Proposition 4.4):

Theorem 3**.**

If $S$ is a nonsingular projective toric surface, then,

[TABLE]

Theorem 3 holds in particular for $S=\mathbb{P}^{2},\;\mathbb{P}^{1}\times\mathbb{P}^{1}$, which are the generators of the cobordism ring of nonsingular projective surfaces. Similar formulas was worked by A. Negut and others (see [N12, N17] and the references within). We use a refinement of this fact together with a degeneration formula developed for $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ (Proposition 4.5) to prove that for any nonsingular projective surface $S$, certain integrals against $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ can be expressed as integrals against $S^{[n_{1}]}\times S^{[n_{2}]}$ (Corollary 4.13, Proposition 4.14). A generalization of such formulas have been recently worked out in [GT17, GT19] using degeneracy loci techniques.

This last result (Proposition 4.14) is used in [GSY17b] to express some of the reduced localized DT invariants of $S$ as sums of integrals over the product of Hilbert schemes of points on $S$, when $S$ is one of five types generic complete intersections

[TABLE]

Such integrals also have applications in evaluating Vafa-Witten invariants recently defined in [TT17].

The operators

[TABLE]

were studied by Carlsson-Okounkov in [CO12]. They expressed these operators in terms of explicit vertex operators. Using this, we prove the following explicit formula (Proposition 5.2):

Theorem 4**.**

Let $S$ be a nonsingular projective surface, $\omega_{S}$ be its canonical bundle, and $K_{S}=c_{1}(\omega_{S})$. Then,

[TABLE]

where $\langle-,-\rangle$ is the Poincaré paring on $S$.

This is the only situation that we have been able to find a closed formula for the complete generating series of invariants. It would be interesting to seek similar formulas for the more involved integrals over nested Hilbert schemed that appear in Vafa-Witten theory or reduced local DT theory of $S$.

Aknowledgement

We are grateful to Richard Thomas for providing us with many valuable comments. We would like to thank Eric Carlsson, Andrei Negut, Davesh Maulik, Hiraku Nakajima, Takurō Mochizuki, Alexey Bondal and Mikhail Kapranov for useful discussions.

A. G. was partially supported by NSF grant DMS-1406788. A. S. was partially supported by World Premier International Research Center Initiative (WPI initiative), MEXT, Japan, as well as NSF DMS-1607871, NSF DMS-1306313 and Laboratory of Mirror Symmetry NRU HSE, RF Government grant, ag. No 14.641.31.0001. S.-T. Y. was partially supported by NSF DMS-0804454, NSF PHY-1306313, and Simons 38558. A. S. would like to further sincerely thank the center for Quantum Geometry of Moduli Spaces at Aarhus University, the Center for Mathematical Sciences and Applications at Harvard University and the Laboratory of Mirror Symmetry in Higher School of Economics, Russian federation, for the great help and support.

1.3 Notation and conventions

We will use the symbol $\mathbb{L}^{\bullet}$ for the (full) cotangent complex, and $\mathbb{L}^{\bullet,\text{gr}}$ for the cotangent complex for the sheaves of graded algebras. Our sheaves of graded algebras are always of the form $A_{0}\oplus A_{1}$ where $A_{i}$ is in degree $i$. If $M$ is a complex of graded $(A_{0}\oplus A_{1})$-modules then $k^{i}(M)$ takes the degree $i$ part in the grading. If

[TABLE]

are graded homomorphisms of sheaves of graded algebras in which $s$ is injective then, we will use the following isomorphisms proven in [Ill, IV.2.2.4), IV.2.2.5, IV.3.2.10]

[TABLE]

Associated, to graded homomorphisms of graded sheaves of algebras

[TABLE]

is a natural exact triangle

[TABLE]

that is referred to as the transitivity triangle [Ill, IV.2.3]. 2. 2.

We will denote the universal ideal sheaves of $S^{[m]}_{\beta}$, $S^{[m]}$, and $S_{\beta}$ respectively by $\mathcal{I}^{[m]}_{-\beta}$, $\mathcal{I}^{[m]}$, and $\mathcal{I}_{-\beta}$, and the corresponding universal subschemes respectively by $\mathcal{Z}^{[m]}_{\beta}$, $\mathcal{Z}^{[m]}$, and $\mathcal{Z}_{\beta}$. We will also write $\mathcal{I}^{[m]}_{\beta}$ for $\mathcal{I}^{[m]}\otimes\mathcal{O}(\mathcal{Z}_{\beta})$. Using the universal property of the Hilbert scheme, it can be seen that $\mathcal{I}^{[m]}_{-\beta}\cong\mathcal{I}^{[m]}\otimes\mathcal{O}(-\mathcal{Z}_{\beta})$. 3. 3.

Let $\pi:S\times S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}\to S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$ be the projection, we denote the derived functor $\mathbf{R}\pi_{*}\mathbf{R}\mathcal{H}om$ by $\mathbf{R}\mathcal{H}om_{\pi}$ and its $i$-th cohomology sheaf by $\mathcal{E}xt^{i}_{\pi}$. $\pi$ is a smooth morphism of relative dimension 2 and hence by Grothendieck-Verdier duality $\pi^{!}(-):=\pi^{*}(-)\otimes\omega_{\pi}[2]$ is a right adjoint of $\mathbf{R}\pi_{*}$. 4. 4.

Throughout the paper, we slightly abuse notation and suppress many of the symbols $f^{*}$ and $f^{-1}$ for the pullback of sheaves on $Y$ via a given morphism $f:X\to Y$ of schemes. This makes most of the formulas notationally lighter and hence more readable. 5. 5.

For any line bundle $L$ on $S$ we define $L^{D}:=L^{-1}\otimes\omega_{S}$. Similarly, for any class $\beta\in H^{2}(S,\mathbb{Z})$ we define $\beta^{D}:=K_{S}-\beta$.

2 Nested Hilbert schemes on surfaces

Let $S$ be a nonsingular projective surface over $\mathbb{C}$. We denote the canonical line bundle on $S$ by $\omega_{S}$ and $K_{S}:=c_{1}(\omega_{S})$. For any nonnegative integer $m$ and effective curve class $\beta\in H^{2}(S,\mathbb{Z})$, we denote by $S^{[m]}_{\beta}$ the Hilbert scheme of 1-dimensional subschemes $Z\subset S$ such that

[TABLE]

If $\beta=0$ we drop it from the notation and denote by $S^{[m]}$ the Hilbert scheme of $m$ points on $S$. Similarly, in the case $m=0$ but $\beta\neq 0$ we drop $m$ from the notation and use $S_{\beta}$ to denote the Hilbert scheme of curves in class $\beta$. There are natural morphisms

[TABLE]

where $\operatorname{div}$ sends a 1-dimensional subscheme $Z\subset S$ to its underlying divisor on $S$, $\operatorname{det}(Z):=\mathcal{O}(\operatorname{div}(Z)),$ and $\operatorname{pts}(Z)$ is the 0-dimensional subscheme of $S$ defined by the ideal $I_{Z}(\operatorname{div}(Z))$. In fact $\operatorname{div}(-)$ is well-behaved with respect to basechange (see [F, KM77]) and sends a flat family of 1-dimensional subschemas of $S$ to a flat family of divisors on $S$, and so using the universal property of the Hilbert schemes, these maps are morphisms of schemes. There is a natural isomorphism of schemes $S^{[m]}_{\beta}\cong S^{[m]}\times S_{\beta}$ defined by mapping $Z\mapsto(\operatorname{pts}(Z),\operatorname{div}(Z))$. Under this, we have the following relation among the universal ideal sheaves: $\mathcal{I}^{[m]}_{-\beta}\cong\mathcal{I}^{[m]}\boxtimes\mathcal{O}(-\mathcal{Z}_{\beta})$.

It is well known that $S^{[m]}$ is a nonsingular variety of dimension $2m$. The tangent bundle of $S^{[m]}$ is identified with

[TABLE]

where the index 0 indicates the trace-free part.

The main object of study in this paper is the following

Definition 2.1**.**

Suppose that ${\boldsymbol{n}}:=n_{1},n_{2},\dots,n_{r}$ is a sequence of $r\geq 1$ nonnegative integers, and ${\boldsymbol{\beta}}:=\beta_{1},\dots,\beta_{r-1}$ is a sequence of classes in $H^{2}(S,\mathbb{Z})$. The nested Hilbert scheme is the closed subscheme

[TABLE]

naturally defined by the $r$-tuples $(Z^{\prime}_{1},\dots,Z^{\prime}_{r})$ of subschemes of $S$ such that $\operatorname{pts}(Z^{\prime}_{i})\subset Z^{\prime}_{i-1}$ is a subscheme for all $1<i\leq r$. We drop ${\boldsymbol{\beta}}$ or ${\boldsymbol{n}}$ from the notation respectively when $\beta_{i}=0$ for all $i$ or $n_{i}=0,\beta_{j}\neq 0$ for all $i,j$. The scheme $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$ represents the functor that takes a scheme $U$ to the set of flat families of ideals $\mathcal{I}_{1},\dots,\mathcal{I}_{r}\subseteq\mathcal{O}_{S\times U}$ and flat families of Cartier divisors $\mathcal{D}_{1}\dots,\mathcal{D}_{r-1}\subset S\times U$ such that

[TABLE]

and on restriction to any closed fiber $\mathcal{I}_{i}|_{S\times\{u\}}$ has colength $n_{i}$ and $[\mathcal{D}_{i}|_{S\times\{u\}}]=\beta_{i}$.

As a set we can think of $S^{[{\boldsymbol{n}}]}_{{\boldsymbol{\beta}}}$ as given by the tuples of subschemes

[TABLE]

together with the nonzero maps $\phi_{i}:I_{Z_{i}}\to I_{Z_{i+1}}(C_{i})$, up to multiplication by scalars, for all $1\leq i<r.$ Note that each $\phi_{i}$ is necessarily injective, and taking double dual, it gives (up to a scalar) the tautological section of $\mathcal{O}(C_{i})$. In this correspondence, $Z^{\prime}_{r}=Z_{r}$, and for $1\leq i<r$ the ideal sheaf of the subscheme $Z^{\prime}_{i}$ is $I_{Z_{i}}(-C_{i})$.

By the construction of nested Hilbert schemes, the maps $\phi_{i}$ above are induced from the universal maps

[TABLE]

defined over $S\times S^{[{\boldsymbol{n}}]}_{{\boldsymbol{\beta}}}$.

Applying the functors \mathbf{R}\mathcal{H}om_{\pi}\big{(}-,\mathcal{I}^{[n_{i+1}]}_{\beta_{i}}\big{)} and \mathbf{R}\mathcal{H}om_{\pi}\big{(}\mathcal{I}^{[n_{i}]},-\big{)} to the universal map $\Phi_{i}$, we get the following morphisms in derived category

[TABLE]

[TABLE]

Consider the map

[TABLE]

We will show that this map factors through the trace free part

[TABLE]

The following proposition that implies Theorem 1 is proven in Section 2.5:

Proposition 2.2**.**

$S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$* is equipped with a perfect absolute obstruction theory $F^{\bullet}\to\mathbb{L}^{\bullet}_{S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}}$ whose virtual tangent bundle is given by*

[TABLE]

where the map is the one defined above.333The negative signs on the diagonal of the matrix in (10) were missing in the first draft of the paper. We were notified of the corrected form of the matrix by Richard Thomas.

2.1 $2$-step nested Hilbert schemes

In this section we study $S^{[n_{1},n_{2}]}_{\beta}:=S^{[{\boldsymbol{n}}]}_{{\boldsymbol{\beta}}}$ in the case $r=2$. Recall from Definition 2.1 that for a pair of nonnegative integers $n_{1},n_{2}$ and an effective curve class $\beta\in H^{2}(S,\mathbb{Z})$, we defined the projective scheme $S^{[n_{1},n_{2}]}_{\beta}$, whose set of closed points is given by

[TABLE]

There are universal objects defined over $S\times S^{[n_{1},n_{2}]}_{\beta}$ as before:

[TABLE]

Applying the functors $\mathbf{R}\mathcal{H}om_{\pi}(-,\mathcal{I}^{[n_{2}]}_{\beta})$ and $\mathbf{R}\mathcal{H}om_{\pi}(\mathcal{I}^{[n_{1}]},-)$ to the universal map $\Phi$, we get the following morphisms in derived category

[TABLE]

Let $T$ be any scheme over $\mathbb{C}$-scheme $U$, and let

[TABLE]

be a square zero extension over $U$ with the ideal $J$. As $J^{2}=0$, $J$ can be considered as an $\mathcal{O}_{T}$-module. Suppose we have a Cartesian diagram

[TABLE]

where $\operatorname{pr}_{1}$ is the composition of $\iota$ with the projection $S^{[n_{1}]}_{\beta}\times S^{[n_{2}]}\to S^{[n_{1}]}_{\beta}$. The bottom row of (12) corresponds to a flat $U$-family $Z_{1,U}\subset S\times U$ of subschemes of length $n_{1}$, and the top row corresponds to the data

[TABLE]

consisting of $Z_{1,T}=Z_{1,U}\times_{S\times U,(\operatorname{id},a)}S\times T,$ a $T$-flat family $Z_{2,T}\subset S\times T$ of subschemes of length $n_{2}$, a $T$-flat family $C_{T}\subset S\times T$ of effective Cartier divisors in class $\beta$, and (up to a scalar) a homomorphism

[TABLE]

Let $\pi_{T}$ be the projections from $S\times T\to T$. By [Ill, Prop. IV.3.2.12] and [JS12, Thm 12.8], there exists an element

[TABLE]

whose vanishing is necessary and sufficient to extend the $T$-family (13) to a $\overline{T}$-family

[TABLE]

such that $Z_{1,\overline{T}}=Z_{1,U}\times_{S\times U,(\operatorname{id},\overline{a})}S\times\overline{T}$. In fact by [Ill, Prop. IV.3.2.12], $\operatorname{ob}$ is the obstruction to deforming the morphism $\phi_{T}$ while the deformation $I_{Z_{1,\overline{T}}}$ of $I_{Z_{1,T}}$ is given. Suppose that $\phi_{\overline{T}}:I_{Z_{1,\overline{T}}}\to\mathcal{F}$ is such a deformation, where $\mathcal{F}$ is a flat family of rank 1 torsion free sheaves with $\mathcal{F}|_{S\times T}=I_{Z_{2,T}}(C_{T})$. Then by [K90, Lemma 6.13] the double dual $\mathcal{F}^{**}$ is an invertible sheaf. Now $\phi_{\overline{T}}^{**}:\mathcal{O}_{S\times\overline{T}}\to\mathcal{F}^{**}$ is injective when restricted to closed fibers of $S\times\overline{T}\to\overline{T}$, and hence by [HL10, Lemma 2.1.4], $\operatorname{coker}(\phi_{\overline{T}}^{**})$ is also flat over $\overline{T}$. Thus, there exists a $\overline{T}$-flat subscheme $C_{\overline{T}}\subset S\times\overline{T}$ (i.e. the Cartier divisor cut out by $\phi_{\overline{T}}^{**}$) that restricts to $C_{T}$ and $\mathcal{F}^{**}\cong\mathcal{O}(C_{\overline{T}})$. We conclude from $\mathcal{F}\subseteq\mathcal{F}^{**}$ that $\mathcal{F}\cong I_{Z_{2,\overline{T}}}(C_{\overline{T}})$ for some $\overline{T}$-flat subscheme $Z_{2,\overline{T}}\subset S\times\overline{T}$ restricting to $Z_{2,T}$.

If $\operatorname{ob}=0$ then by [Ill, Prop. IV.3.2.12] again, the set of isomorphism classes of deformations forms a torsor under

[TABLE]

Lemma 2.3**.**

$\operatorname{ob}=\and_{\operatorname{red}}(\phi_{T})\cup(\pi^{*}_{T}e(\overline{T})\otimes\operatorname{id})$, where

[TABLE]

is a certain reduction of the Atiyah class $\and_{\mathcal{O}_{S\times T}/\mathcal{O}_{S\times U}}(\phi_{T})$ of $\phi_{T}$ [Ill, IV.2.3], and

[TABLE]

in which

[TABLE]

is the Kodaira-Spence class associated to the square zero extension $T\hookrightarrow\overline{T}$.

Proof.

The proof is the same as the proof of [JS12, Thm 12.9]: in diagram (12.17) of [ibid], the first vertical arrow needs to be replaced by

[TABLE]

This is the degree 1 part of the transitivity triangle (see item 1 in Subsection 1.3) associated to natural homomorphism of sheaves of graded algebras on $S\times T$

[TABLE]

By flatness of $I_{Z_{1,U}}$ over $\mathcal{O}_{U}$, we see that $\mathcal{O}_{S\times U}\oplus I_{Z_{1,U}}$ is flat over $\mathcal{O}_{U}$ and hence

[TABLE]

so [Ill, II.2.2] $\mathbb{L}^{\bullet,\text{gr}}_{(\mathcal{O}_{S\times T}\oplus I_{Z_{1,T}})/\mathcal{O}_{S\times U}}$ is isomorphic to

[TABLE]

and hence as in the proof of [JS12, Thm 12.9], composing $\and_{\mathcal{O}_{S\times T}/\mathcal{O}_{S\times U}}(\phi_{T})$ with the projection

[TABLE]

we arrive at the definition of $\and_{\operatorname{red}}(\phi_{T})$ in (15).

∎

Now the idea is that from the deformation/obstruction theory of the universal map $\Phi:\mathcal{I}^{[n_{1}]}\to\mathcal{I}^{[n_{2}]}_{\beta}$ reviewed above, we construct a relative perfect obstruction theory $F^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{S^{[n_{1},n_{2}]}_{\beta}/S^{[n_{1}]}}$ (Proposition 2.4). Using the fact that $S^{[n_{1}]}$ is nonsingular, we will deduce the absolute perfect obstruction theory $F^{\bullet}\to\mathbb{L}^{\bullet}_{S^{[n_{1},n_{2}]}_{\beta}}$ from $F^{\bullet}_{\operatorname{rel}}$ (Proposition 2.5). This will prove Proposition 2.2 for $r=2$.

Proposition 2.4**.**

The complex

[TABLE]

defines a relative perfect obstruction theory for the morphism $\operatorname{p}:S^{[n_{1},n_{2}]}_{\beta}\to S^{[n_{1}]}$. In other words, $F^{\bullet}_{\operatorname{rel}}$ is perfect with amplitude contained in $[-1,0]$, and there exists a morphism in the derived category $\alpha:F^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{\operatorname{p}}$, such that $h^{0}(\alpha)$ and $h^{-1}(\alpha)$ are respectively isomorphism and epimorphism. The rank of $F^{\bullet}_{\operatorname{rel}}$ is equal to

[TABLE]

Proof.

Step 1: (perfectness) We show that the complex $F^{\bullet\vee}_{\operatorname{rel}}$ is perfect with amplitude $[0,1]$. By basechange and the same argument as in the proof of [HT10, Lemma 4.2], it suffices to show that $h^{i}(\mathbf{L}t^{*}F^{\bullet\vee}_{\operatorname{rel}})=0$ for $i\neq 0,1$, where $t:P\hookrightarrow S^{[n_{1},n_{2}]}_{\beta}$ is the inclusion of an arbitrary closed point $P=(Z_{1},C,Z_{2},\phi)\in S^{[n_{1},n_{2}]}_{\beta}$. Therefore, by the definition of $F^{\bullet\vee}_{\operatorname{rel}}$ we get the exact sequence

[TABLE]

All the $\operatorname{Ext}^{i}_{S}$ for $i\neq 0,1,2$ vanish, so we deduce easily that $h^{i}(\mathbf{L}t^{*}F^{\bullet\vee}_{\operatorname{rel}})=0$ for $i\neq-1,0,1,2$. From the sequence above we see that

[TABLE]

But by definition this map is induced by applying $\operatorname{Hom}_{S}(-,I_{Z_{2}}(C))$ to the map $\phi:I_{Z_{1}}\to I_{Z_{2}}(C)$, so it takes $\operatorname{id}$ to $\phi$, and hence it is injective. Therefore, $h^{-1}(\mathbf{L}t^{*}F^{\bullet\vee}_{\operatorname{rel}})=0$.

To prove $h^{2}(\mathbf{L}t^{*}F^{\bullet\vee}_{\operatorname{rel}})=0$, we show that the map

[TABLE]

in the exact sequence above is surjective, or equivalently by Serre duality, the dual map

[TABLE]

is injective. But this follows after applying the left exact functor $\operatorname{Hom}_{S}(I_{Z_{2}},-)$ to the injection $I_{Z_{1}}\otimes\omega_{S}(-C)\to I_{Z_{2}}\otimes\omega_{S}$ that is induced by tensoring the map $\phi$ above by $\omega_{S}(-C)$.

Step 2: (map to the cotangent complex) We construct a morphism in derived category $\alpha:F^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{\operatorname{p}}$. Consider the reduced Atiyah class (15) in the case $T=S^{[n_{1},n_{2}]}_{\beta}$ and $U=S^{[n_{1}]}$. It defines an element in

[TABLE]

So under the identification above, the reduced Atiyah class defines a morphism in derived category

[TABLE]

But by Grothendieck-Verdier duality again,

[TABLE]

and hence we are done.

Step 3: (obstruction theory) We show $h^{0}(\alpha)$ and $h^{-1}(\alpha)$ are respectively isomorphism and epimorphism. We use the criterion in [BF97, Theorem 4.5]. Suppose we are in the situation of the diagram (12). Define

[TABLE]

Composing $e(\overline{T})$ (given in (16)) and the natural morphism of cotangent complexes $\mathbf{L}g^{*}\mathbb{L}^{\bullet}_{\operatorname{p}}\to\mathbb{L}^{\bullet}_{a}$ gives the element $\varpi(g)\in\operatorname{Ext}^{1}_{T}(\mathbf{L}g^{*}\mathbb{L}^{\bullet}_{\operatorname{p}},J)$ whose image under $\alpha$ is denoted by

[TABLE]

For $i=0,1$, we will use the following identifications:

[TABLE]

Here we have used the fact that $\mathbf{L}g^{*}\dashv\mathbf{R}g_{*}$ i.e. $\mathbf{L}g^{*}$ is the left adjoint of $\mathbf{R}g_{*}$, and Grothendieck-Verdier duality. Now using $\mathbf{R}f_{*}\pi_{T}^{*}=\pi^{*}\mathbf{R}g_{*}$ in the last $\operatorname{Ext}$ above, we get

[TABLE]

Similar to the Step 2 it can be seen that the composition

[TABLE]

arises from $\and_{\operatorname{red}}(\phi_{T})$ over $S\times T$ (see (15)). Therefore,

[TABLE]

is identified with the element $\alpha^{*}\varpi(g)$ via the identifications above for $i=1$. By the definition of the obstruction class $\operatorname{ob}(\phi_{T},J)$, this means that $\alpha^{*}\varpi(g)$ vanishes if and only if there exists an extension $\overline{g}$ of $g$ corresponding to (14). Using the identifications above, this time for $i=0$, we can see that if $\alpha^{*}\varpi(g)=0$, then the set of extensions forms a torsor under $\operatorname{Hom}_{T}(\mathbf{L}g^{*}F^{\bullet},J)$. Now by [BF97, Theorem 4.5] $\alpha$ is an obstruction theory.

Step 4: (rank of $F^{\bullet}_{\operatorname{rel}}$) The claim about the rank follows from

[TABLE]

where $(Z_{1},C,Z_{2},\phi)$ is a closed point of $S^{[n_{1},n_{2}]}_{\beta}$.

∎

Proposition 2.5**.**

$S^{[n_{1},n_{2}]}_{\beta}$* is equipped with the absolute perfect obstruction theory $F^{\bullet}\to\mathbb{L}^{\bullet}_{S^{[n_{1},n_{2}]}_{\beta}}$. Its virtual tangent bundle is given by*

[TABLE]

Proof.

Since $S^{[n_{1}]}$ is nonsingular, by the standard techniques (see [MPT10, Section 3.5])

[TABLE]

gives a perfect absolute obstruction theory over $S^{[n_{1},n_{2}]}_{\beta}$, where $\theta$ is the composition of $\alpha:F^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{\operatorname{p}}$ and the Kodaira-Spencer map $c:\mathbb{L}^{\bullet}_{\operatorname{p}}\to\operatorname{p}^{*}\Omega_{S^{[n_{1}]}}[1]$. We claim that $\theta$ is given by

[TABLE]

where the first and second maps are respectively induced by $\Phi:\mathcal{I}^{[n_{1}]}\to\mathcal{I}^{[n_{2}]}_{\beta}$ and the natural map $c^{\prime}:\operatorname{coker}(\Phi)\to\mathcal{I}^{[n_{1}]}[1]$. To see the claim consider the commutative diagram of graded sheaves of algebras on $S\times S^{[n_{1},n_{2}]}_{\beta}$ with all unlabelled arrows are the obvious natural maps444following our convenction we have suppressed the symbols for pullbacks via natural morphisms.:

[TABLE]

Taking the transitivity triangles (see item 1 in Subsection 1.3) of the top two rows and applying the same construction leading to the definition of $\and_{\operatorname{red}}(\Phi)$ in the proof of Lemma 2.3, we obtain the commutative diagram of sheaves on $S\times S^{[n_{1},n_{2}]}_{\beta}$

[TABLE]

in which both vertical arrows are the natural maps, the bottom horizontal arrow is the pullback of the Atiyah class

[TABLE]

composed with $\operatorname{q}^{*}\mathbb{L}^{\bullet}_{S^{[n_{2}]}_{\beta}}\to\mathbb{L}^{\bullet}_{S^{[n_{1},n_{2}]}_{\beta}}$ induced by the natural morphism $\operatorname{q}\colon S^{[n_{1},n_{2}]}_{\beta}\to S^{[n_{2}]}_{\beta}$. Taking cones over the vertical arrows in turn induces the commutative diagram

[TABLE]

On the other hand, the transitivity triangles of the bottom two rows of (18) imply that the top row in (19) is in fact the composition

[TABLE]

where $\and(\mathcal{I}^{[n_{1}]})$ is the (pullback of the) usual Atiyah class. The claim now follows from this and the construction of the map $\alpha$ using the class $\and_{\operatorname{red}}(\Phi)$ in Step 2 of proof of Proposition 2.4. To ease the notation let

[TABLE]

and denote by $E^{\bullet}$ the right hand side of the expression in the proposition. By Proposition 2.4, $F^{\bullet\vee}_{\operatorname{rel}}=\operatorname{Cone}\left(B^{\bullet}\to C^{\bullet}\right)$, so by (2) and the claim above, (17) can be rewritten as

[TABLE]

Consider the commutative diagram

[TABLE]

in which the bottom row is the natural exact triangle. Taking the cone of the diagram one gets the exact triangle

[TABLE]

Next consider the commutative diagram

[TABLE]

in which both rows are exact triangles, and in the rightmost vertical arrow we use the splitting $A^{\bullet}\cong A^{\bullet}_{0}\oplus\mathbf{R}\pi_{*}\mathcal{O}$ given by the trace map (the left square is obviously commutative, and the right square is commutative by the claim we proved above). Now by construction the vertical arrows in the above diagram factor through the exact triangle (20), and hence we arrive at the following commutative diagram in which all the rows and columns are exact triangle.

[TABLE]

In fact the columns and the top and middle rows are exact triangles with commutative top squares, and the bottom row is induced from the rest of the diagram by taking the cone. Therefore, the bottom row must also be an exact triangle which means that $E^{\bullet}\cong F^{\bullet\vee}$ as desired. ∎

This finishes the proof of Proposition 2.2 in the case $r=2$. Propositions 2.4 and 2.5 imply

Corollary 2.6**.**

The perfect obstruction theory of Proposition 2.5 defines a virtual fundamental class on $S^{[n_{1},n_{2}]}_{\beta}$ denoted by

[TABLE]

∎

2.2 Gillam’s construction

The relative obstruction theory $F^{\bullet}_{\operatorname{rel}}$ is obtained from the deformation/obstruction theory of the universal map $\Phi:\mathcal{I}^{[n_{1}]}\to\mathcal{I}^{[n_{2}]}_{\beta}$. As mentioned in the introduction, following [G11], one can can instead use deformation/obstruction theory of the quotient $\operatorname{coker}(\Phi)$ to construct a relative perfect obstruction theory

[TABLE]

This amounts to identifying $S^{[n_{1},n_{2}]}_{\beta}$ with a component of the relative quot scheme

[TABLE]

of quotients of $\mathcal{I}^{[n_{2}]}$ and applying [G11, Theorem 4.6]. By a similar argument as Proposition 2.5 (using the smoothness of $S^{[n_{2}]}$ this time), one can then deduce an absolute perfect obstruction theory $F^{\prime\bullet}\to\mathbb{L}^{\bullet}_{S^{[n_{1},n_{2}]}_{\beta}}$. By comparing the K-theory classes of $F^{\bullet}$ and $F^{\prime\bullet}$ the reader can verify that the resulting virtual class from this approach is the same as that of Corollary 2.6 (see [S04]).

2.3 Reduced perfect obstruction theory

In this section we assume that for any effective line bundle $L$ on $S$ with $c_{1}(L)=\beta$, we have

[TABLE]

Recall that the relative virtual tangent bundle of Proposition 2.4 is given by

[TABLE]

We get a natural map

[TABLE]

that induces

[TABLE]

We claim that $h^{1}(\mu)$ is surjective. To see this, by basechange, it suffices to prove that $h^{1}(\mu)$ is fiberwise surjective. Let $t:P\hookrightarrow S^{[n_{1},n_{2}]}_{\beta}$ be the inclusion of an arbitrary closed point $P=(Z_{1},C,Z_{2},\phi)\in S^{[n_{1},n_{2}]}_{\beta}$. Then, by basechange we have the natural exact sequence555Note that $\operatorname{Ext}^{3}_{S}(\operatorname{coker}(\phi),I_{Z_{2}}(C))=\operatorname{Ext}^{3}_{S}(I_{Z_{2}}(C),I_{Z_{2}}(C))=0$.

[TABLE]

The surjectivity of the map $u$ was established in Step 1 of the proof of Proposition 2.4. We have

[TABLE]

[TABLE]

By assumption (21), $H^{0}(\mathcal{O}_{S}(C)^{D})=0$, and hence $h^{1}(\mu)_{P}$ is surjective and the claim follows. We now have the diagram

[TABLE]

where the first row is exact by the proof Proposition 2.5. But since

[TABLE]

the surjection $h^{1}(\mu)_{P}$ factors through $h^{1}(\mathbf{L}t^{*}F^{\bullet\vee})$. Therefore, by basechange again there exists a surjection $h^{1}(F^{\bullet\vee})\to\mathcal{O}_{S^{[n_{1},n_{2}]}_{\beta}}^{p_{g}}.$

Proposition 2.7**.**

If the condition (21) is satisfied and $p_{g}(S)>0$, then,

[TABLE]

Proof.

Under the assumptions of the proposition, we showed above that the obstruction sheaf admits a surjection

[TABLE]

and hence the associated virtual class vanishes by [KL13, Theorem 1.1]. ∎

Definition 2.8**.**

The map $h^{1}(\mu)$ induces the morphism in derived category

[TABLE]

Dualizing gives a map $\mathcal{O}^{p_{g}}_{S^{[{\boldsymbol{n}}]}_{\beta}}[1]\to F^{\bullet}$. Define $F^{\bullet}_{\operatorname{red}}$ to be its cone.

We will show that under a slightly stronger condition than (21), $F^{\bullet}_{\operatorname{red}}$ gives rise to a perfect obstruction theory over $S^{[n_{1},n_{2}]}_{\beta}$. First note that the curve class $\beta\in H^{1,1}(S)\cap H^{2}(S,\mathbb{Z})$ defines an element of $H^{1}(\Omega_{S})$ and consider the natural pairing $T_{S}\otimes\Omega_{S}\to\mathcal{O}_{S}$. This condition is666This is condition (3) in [KT14].

[TABLE]

To show $F^{\bullet}_{\operatorname{red}}$ gives rise to a perfect obstruction theory, we use the beautiful idea of [KT14]. We sketch their method here and make some necessary changes; the reader can find the missing detail in [KT14]. $S$ is embedded as the central fiber of an algebraic twistor family $\mathcal{S}\to B$, where $B$ is a first order Artinian neighborhood of the origin in a certain $p_{g}$-dimensional family of the first order deformations of $S$ 777To simplify notation, we have used $\mathcal{S}$ instead of $\mathcal{S}_{B}$ that was used in [KT14].. Explicitly, let

[TABLE]

be a subspace over which $*\cup\beta$ in (22) restricts to an isomorphism, and let $\mathfrak{m}$ denote the maximal ideal at the origin $0\in H^{1}(T_{S})$. Then,

[TABLE]

and $\mathcal{S}$ is the restriction of a tautological flat family of surfaces with Kodaira-Spencer class the identity in

[TABLE]

The Zarisiki tangent space $T_{B}$ is naturally identified with $V$. By [KT14, Lemma 2.1], $\mathcal{S}$ is transversal to the Noether-Lefschetz locus of the $(1,1)$-class $\beta$, and as a result, $\beta$ does not deform outside of the central fiber of the family. Using this fact, as in [KT14, Proposition 2.3], one can show that

[TABLE]

where the right hand side is the relative nested Hilbert scheme of the family $\mathcal{S}\to B$. We use the same symbols

[TABLE]

as before to denote the universal objects over $\mathcal{S}\times_{B}(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}$, and we let $\pi$ be the projection to the second factor of $\mathcal{S}\times_{B}(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}$. The arguments of Section 2.1 can be adapted with no changes to prove that

[TABLE]

is the virtual tangent bundle of a perfect $B$-relative obstruction theory $G^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}/B}$, and

[TABLE]

is the associated absolute perfect obstruction theory. By the definitions of $F^{\bullet}$ and $G^{\bullet}_{\operatorname{rel}}$, and the isomorphism (23), we see that $F^{\bullet}\cong G^{\bullet}_{\operatorname{rel}}$. Now we claim that the composition

[TABLE]

is an isomorphism. By the definitions of $G^{\bullet}_{\operatorname{rel}}$ and $F^{\bullet}_{\operatorname{red}}$, to prove the claim, it suffices to show that

[TABLE]

is an isomorphism. By the Nakayama lemma we may check this at a closed point $P=(Z_{1},C,Z_{2},\phi)\in S^{[n_{1},n_{2}]}_{\beta}$. Define the reduced Atiyah class corresponding to $P$ as follows. Consider the natural homomorphisms of sheaf of graded algebras on $S$

[TABLE]

The degree 1 part of the transitivity triangle (see item 1 in Subsection 1.3) associated to the graded cotangent complexes gives the first arrow in

[TABLE]

Define $\and_{\operatorname{red}}(\phi)$ to be the composition of these two arrows.

After dualizing and using the identifications above the pullback of (24) to $P$ becomes

[TABLE]

Here as in [KT14], one needs to use a similar argument as [MPT10, Proposition 13] to deduce that the composition of $G^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}/B}$ and the Kodaira-Spencer map $\mathbb{L}^{\bullet}_{(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}/B}\to\Omega_{B}[1]$ for $(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}$ coincides with the cup product of $\and_{\operatorname{red}}(\phi)$ and the Kodaira-Spencer class for $S$. Similar to [ibid], this is achieved by relating the reduced Atiyah class of $\mathcal{S}\times_{B}(\mathcal{S}/B)^{[n_{1},n_{2}]}_{\beta}$ arising (as in Lemma 2.3) from the transitivity triangle associated to the natural maps of sheaves of graded algebras

[TABLE]

to the reduced Atiyah classes of each factors.

Lemma 2.9**.**

$h^{1}(\mu)_{P}\circ\and_{\operatorname{red}}(\phi)=\and(I_{Z_{2}}(C)),$* where*

[TABLE]

is the usual Atiyah class.

Proof.

Consider the commutative diagram of sheaves of graded algebras with all unlabelled arrows are the obvious natural maps:

[TABLE]

Taking the degree 1 part of the the transitivity triangles of the rows followed by a projection as in the definition of $\and_{\operatorname{red}}(\phi)$ above we get the commutative diagram

[TABLE]

proving the lemma. ∎

But by [BFl03, Prop 4.2],

[TABLE]

which by condition (22) is an isomorphism when restricted to $V\subset H^{1}(T_{S})$, and hence the claim is proven. We have shown

Proposition 2.10**.**

If the condition (22) is satisfied, then, $F^{\bullet}_{\operatorname{red}}$ is a perfect obstruction theory on $S^{[n]}_{\beta}$, and hence defines a reduced virtual fundamental class

[TABLE]

∎

2.4 Invariants

Let $\mathcal{P}_{1},\dots,\mathcal{P}_{s}$ be polynomials in the Chern classes of $\mathcal{I}^{[n_{1}]},\mathcal{I}^{[n_{2}]}$, $\mathcal{I}_{-\beta}$, $\pi^{*}T_{S^{[n_{1}]}}$, $\pi^{*}T_{S^{[n_{2}]}}$, etc. cupped with the pullback of a cohomology classes from $S$ then, we can define the invariant

[TABLE]

If the condition (22) is satisfied, we can define the reduced invariants

[TABLE]

Let $u:=c_{1}(\mathcal{O}(\mathcal{Z}_{\beta}))|_{x\times S^{[n_{1},n_{2}]}_{\beta}}$, where $x\in S$ is an arbitrary closed point. Define

[TABLE]

In Section 3 we will see that virtual classes $[S_{\beta}]^{\operatorname{vir}}$, $[S^{[0,n_{2}]}_{\beta}]^{\operatorname{vir}}_{\operatorname{red}}$ coincide respectively with the virtual classes constructed in [DKO07] and [KT14]. Therefore, by suitable choices of the integrands $\pi_{*}\mathcal{P}_{i}$, the invariants $\mathsf{N}^{\operatorname{red}}_{S}(0,n_{2},\beta;\mathcal{P}_{1},\dots,\mathcal{P}_{s})$ recover the reduced stable pair invariants of [KT14]. Similarly, the invariants $P_{S}(0,0,\beta)$ recover Poincaré invariants of [DKO07]. In [GSY17b], we express certain contributions to the reduced localized DT invariants of $S$ in terms of the invariants $\mathsf{N}_{S}(n_{1},n_{2},\beta;\mathcal{P})$ by making suitable choices of the integrand $\mathcal{P}$.888To clarify the potential confusion for the reader, we emphasize that the reduced localized DT invariants of [GSY17b] uses the non-reduced virtual class $[S^{[n_{1},n_{2}]}_{\beta}]^{\operatorname{vir}}$. We will study some of the invariants $\mathsf{N}_{S}(n_{1},n_{2},0;\mathcal{P})$ in Sections 4 and 5.

2.5 Proof of Theorem 1 (Proposition 2.2)

In Section 2.1 we proved Proposition 2.2 in the case $r=2$. We now use induction on $r$ to prove the theorem in general. For the simplicity of the notation, we show in detail how the result of Section 2.1 can be used to prove Proposition 2.2 in the case $r=3$. Other induction steps are completely similar and hence are omitted.

Suppose that ${\boldsymbol{n}}:=n_{1},n_{2},n_{3}$ is a sequence of nonnegative integers, and ${\boldsymbol{\beta}}:=\beta_{1},\beta_{2}$ is a sequence of effective curve classes in $H^{2}(S,\mathbb{Z})$. Define ${\boldsymbol{n}}^{\prime}:=n_{1},n_{2}$. Our goal is to prove the expression in Proposition 2.2 for $r=3$ is a perfect obstruction theory.

Consider the chain of natural forgetful morphisms and the associated exact triangle of cotangent complexes

[TABLE]

where $f:=f_{1}\circ f_{2}=\operatorname{pts}\circ\operatorname{pr}_{1}$, using the notation at the beginning of Section 2.

Proposition 2.4, provides the relative perfect obstruction theory for the morphism $f_{1}$, that we denote by

[TABLE]

Lemma 2.11**.**

There exists a relative perfect obstruction theory $F^{\bullet}_{f_{2}}\xrightarrow{\alpha_{2}}\mathbb{L}^{\bullet}_{f_{2}}$, where

[TABLE]

Proof.

The proof is along the line of the proof of Proposition 2.4 (see Step 2 of that proof for the corresponding expression in RHS of (27)). This time the obstruction theory is obtained by the deformation/obstruction theory theory of the universal map

[TABLE]

while the data $(\mathcal{I}^{[n_{1}]},\mathcal{I}^{[n_{2}]},\mathcal{Z}_{\beta_{1}},\Phi_{1})$ is kept fixed. ∎

Lemma 2.12**.**

The complexes $F^{\bullet}_{f_{1}}$ and $F^{\bullet}_{f_{2}}$ fit into the following commutative diagram:

[TABLE]

Proof.

Step 1: (Define the map $r$) All the maps in diagram (28) except $r$ are already defined above (see (25), (26), and Lemma 2.11). By the universal properties of the Hilbert schemes and using our convention in suppressing the pullback symbols from the universal ideal sheave, we can write

[TABLE]

Twisting by $\mathcal{O}(\mathcal{Z}_{\beta_{1}})$, we get

[TABLE]

and hence (27) can be written as

[TABLE]

The chain of maps $\mathcal{I}^{[n_{1}]}\xrightarrow{\Phi_{1}}\mathcal{I}^{[n_{2}]}_{\beta_{1}}\xrightarrow{\Phi_{2}(\mathcal{Z}_{\beta_{1}})}\mathcal{I}^{[n_{3}]}_{\beta_{2}}(\mathcal{Z}_{\beta_{1}})$ induces the natural exact triangle

[TABLE]

The maps $i_{2}[1]$ and $\Phi_{2}(\mathcal{Z}_{\beta_{1}})$ induce

[TABLE]

The map $r$ in diagram (28) is then defined by composition of two maps in (2.5).

Step 2: (Commutativity of diagram (28)) We start with the following diagram in which the columns are the exact triangles (31) and (25):

[TABLE]

We prove diagram (33) is commutative. For this, consider the following natural commutative diagrams of sheaf of graded algebras over $S\times S^{[{\boldsymbol{n}}]}_{{\boldsymbol{\beta}}}$ (following our convention we have suppress the symbols for pullbacks of theses via natural morphisms):

[TABLE]

and

[TABLE]

Applying $k^{1}(-)$ to the resulting commutative diagrams of the transitivity (see item 1 in Subsection 1.3) triangle of each row, we get the commutativity of the following two squares:

[TABLE]

and

[TABLE]

Now projecting to the first factors in the second columns of the last two diagrams, and using the definition of $\and_{\operatorname{red}}(-)$ (from the proof of Lemma 2.3), we obtain the commutativity of the bottom and middle squares of diagram (33). Since in diagram (33) both columns are exact triangles, the commutativity of the top square follows, and hence we have proven that the whole diagram (33) commutes.

Recall from Step 2 in the proof of Proposition 2.4, that the maps $\alpha_{i}:F^{\bullet}_{f_{i}}\to\mathbb{L}^{\bullet}_{f_{i}}$ are induced from the classes $\and_{\operatorname{red}}(\Phi_{1})$ and $\and_{\operatorname{red}}(\Phi_{2}(\mathcal{Z}_{\beta_{1}}))$. Therefore, by the definition of the map $r$ in Step 1 of the proof, the commutativity of diagram (28) is equivalent to the commutativity of the top square in diagram (33) proven above, and hence the proof of lemma is complete.

∎

As a result of Lemma 2.12 we get a commutative digram

[TABLE]

in which both rows are exact triangles (the commutativity of the right square was established in Lemma 2.12).

Proposition 2.13**.**

$\alpha_{3}:F^{\bullet}_{f}\to\mathbb{L}^{\bullet}_{f}$* is a relative perfect obstruction theory.*

Proof.

Since $\alpha_{1}$ and $\alpha_{2}$ are perfect obstruction theories, we know $F^{\bullet}_{f_{2}}[-1]$ is of perfect amplitude contained in $[0,1]$ and $\mathbf{L}f_{2}^{*}(F^{\bullet}_{f_{1}})$ is of perfect amplitude contained in $[-1,0]$, therefore $F^{\bullet}_{f}=\operatorname{Cone}(r)$ is of perfect amplitude contained in $[-1,0]$. It remains to show that $h^{0}(\alpha_{3})$ is an isomorphism and $h^{-1}(\alpha_{3})$ is surjective. From diagram (34) and the fact that $\alpha_{1}$ and $\alpha_{2}$ are perfect obstruction theories, we get the following commutative diagram in which both rows are exact:

[TABLE]

Applying 4-lemma once to the leftmost three squares and once to the rightmost three squares above prove the desired properties for $h^{0}(\alpha_{3})$ and $h^{-1}(\alpha_{3})$.

∎

Proof of Proposition 2.2 (for $r=3$).

First note that by construction, for $i=1,2$,

[TABLE]

Now define

[TABLE]

and consider the following two commutative diagrams

[TABLE]

in which all four rows are natural exact triangles and $q:B^{\bullet}_{2}\to\operatorname{Cone}(\Xi_{2})$ is the natural map. Taking cones of the columns of the right diagram gives

[TABLE]

Therefore, taking cones of the columns of the left diagram

[TABLE]

As in the proof of Proposition 2.5, the fact that $S^{[n_{1}]}$ is nonsingular can be used to show that

[TABLE]

is an absolute perfect obstruction theory for $S^{[{\boldsymbol{n}}]}_{\boldsymbol{\beta}}$, and then (using the expression above for $F^{\bullet\vee}_{f}$) to prove that

[TABLE]

where the arrow is as in Proposition 2.2. ∎

3 Special cases

In this section, we show that the virtual fundamental classes arising from the perfect obstruction theories $F^{\bullet}$ and $F^{\bullet}_{\operatorname{red}}$ of Propositions 2.5 and 2.10 specialize to several interesting and important cases such as the ones arising from the algebraic Seiberg-Witten theory and the reduced stable pair theory of surfaces. For the sake of brevity we do not try to match our perfect obstruction theories with these other cases, but rather we only match $K$-group classes of the underlying virtual tangent bundles. Since the virtual fundamental class only depends on the K-theory class of the virtual tangent bundle [S04], this is sufficient for the purpose of the following proposition:

Proposition 3.1**.**

The virtual fundamental class of Theorem 1 recovers the following known cases:

If $\beta=0$ and $n_{1}=n_{2}=n$ then $S^{[n,n]}_{\beta=0}\cong S^{[n]}$ and $[S^{[n,n]}_{\beta=0}]^{\operatorname{vir}}=[S^{[n]}]$ is the fundamental class of the Hilbert scheme of $n$ points. 2. 2.

If $\beta=0$ and $n=n_{2}=n_{1}-1$, as it is known

[TABLE]

is nonsingular **[L99, Section 1.2]**, then,

[TABLE]

for a line bundle $H$ on $\mathbb{P}(\mathcal{I}^{[n]})$. 3. 3.

If $\beta=0$ and $n_{2}=0$, then $S^{[n,0]}_{\beta=0}\cong S^{[n]}$ and

[TABLE]

where $\omega_{S}^{[n]}$ is the rank $n$ tautological vector bundle over $S^{[n]}$ associated to the canonical bundle $\omega_{S}$ of $S$. 4. 4.

If $n_{1}=n_{2}=0$ and $\beta\neq 0$, then $S^{[0,0]}_{\beta}=S_{\beta}$ is the Hilbert scheme of divisors in class $\beta$, and $[S^{[0,0]}_{\beta}]^{\operatorname{vir}}$ and $[S^{[0,0]}_{\beta}]^{\operatorname{vir}}_{\operatorname{red}}$ (in case $\beta$ satisfies condition (22)) coincide with the virtual cycles constructed in **[DKO07]**. 5. 5.

If $n_{1}=0$ and $\beta\neq 0$, then $S^{[0,n_{2}]}_{\beta}$ is the relative Hilbert scheme of points on the universal divisor over $S^{[0,0]}_{\beta}$, and by **[PT10]** is isomorphic to a moduli space of stable pairs; $[S^{[0,n_{2}]}_{\beta}]^{\operatorname{vir}}$ and $[S^{[0,n_{2}]}_{\beta}]^{\operatorname{vir}}_{\operatorname{red}}$ (in case $\beta$ satisfies condition (22)) are the same as the virtual fundamental classes of **[KT14]**.

Proof.

If $\beta=0$, the nested Hilbert scheme of points $S^{[n_{1}\geq n_{2}]}:=S^{[n_{1},n_{2}]}_{\beta=0}$ carries a virtual fundamental class

[TABLE]

Note that by [C98], $S^{[n_{1}\geq n_{2}]}$ is nonsingular only in the following two cases:

$\bullet$ $n_{1}=n_{2}$. In this case $S^{[n_{1}\geq n_{2}]}\cong S^{[n_{1}]}$ by definition, and $[S^{[n_{1}]}]^{\operatorname{vir}}=[S^{[n_{1}]}]$, because by Proposition 2.4, $F^{\bullet}_{\operatorname{rel}}\cong 0$, and so by Proposition 2.5, $F^{\bullet}\cong\Omega_{S^{[n_{1}]}}$. This gives part 1.

$\bullet$ $n_{1}=n_{2}+1$. In this case, since $S^{[n_{2}+1,n_{2}]}$ is nonsingular of dimension $2n_{2}$ (see [C98, L99]). The virtual dimension is $2n_{2}+1$, and hence the obstruction sheaf $H:=h^{1}(F^{\bullet\vee})$ is an invertible sheaf. Then, we can write

[TABLE]

where we have used [BF97, Proposition 5.6] to write $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ as the fundamental class capped with the Euler class of the obstruction bundle. We can express $c_{1}(H)$ in terms of other classes. We know that $S^{[n_{2}+1,n_{2}]}\cong\mathbb{P}(\mathcal{I}^{[n_{2}]})$ (see [L99, Section 1.2]), so in the $K$-group of $\mathbb{P}(\mathcal{I}^{[n_{2}]})$ we can write

[TABLE]

In taking the Chern class, we can ignore the trivial terms and hence we have

[TABLE]

This proves part 2.

If $n_{2}=0$ and $\beta=0$, then we get a perfect obstruction theory over the nonsingular Hilbert scheme of points $S^{[n_{1}]}$ that is arising from the natural obstruction theory of the Hilbert scheme. In fact in this case

[TABLE]

Note that

[TABLE]

Since $S^{[n_{1}]}$ is nonsingular of dimension $2n_{1}$, we see that the obstruction sheaf $h^{1}(F^{\bullet\vee})$ is a vector bundle of rank $n_{1}$, and hence by [BF97, Proposition 5.6]

[TABLE]

We were notified by Richard Thomas that the obstruction bundle \mathcal{E}xt^{1}_{\pi}\big{(}\mathcal{I}^{[n_{1}]},\mathcal{O}_{\mathcal{Z}_{1}}\big{)} can be identified with the dual of the tautological bundle $\omega_{S}^{[n]}:=\pi_{*}\left(\omega_{S}|_{\mathcal{O}_{\mathcal{Z}_{1}}}\right)$. This can be seen by applying \mathcal{H}om\big{(}\mathcal{O}_{\mathcal{Z}_{1}},-\big{)} to the short exact sequence

[TABLE]

over $S\times S^{[n_{1}]}$ to get the isomorphism

[TABLE]

Now pushing forward, we prove the claim

[TABLE]

where the second isomorphism is because of local to global spectral sequence (as $\mathcal{Z}_{1}$ is fiberwise 0-dimensional) and the third one is by Grothendieck-Verdier duality. This completes the proof of part 3.

If $n_{1}=n_{2}=0$, and $\beta\neq 0$, the perfect obstruction theory $F^{\bullet\vee}$ on $S_{\beta}=S^{[0,0]}_{\beta}$ specializes to

[TABLE]

studied by Dürr-Kabanov-Okonek [DKO07] in the course of algebraic Seiberg-Witten invariants (Poincaré invariants). Moreover, one can see by inspection that under condition (22), the $K$-group class of $(F^{\bullet}_{\operatorname{red}})^{\vee}$ coincides with the $K$-group class of the reduced virtual tangent bundle over $S_{\beta}$ constructed in [DKO07]. This is because by Definition 2.8 in the $K$-group $(F^{\bullet}_{\operatorname{red}})^{\vee}=F^{\bullet\vee}+\mathcal{O}_{S_{\beta}}^{p_{g}}$ and the same is true for the reduced virtual tangent constructed in [DKO07]. This proves part 4.

Finally, if $n_{1}=0$, $n_{2}\neq 0$ and $\beta\neq 0$, then by [PT10, Prop B.8],

[TABLE]

where $\text{Hilb}^{n_{2}}(\mathcal{Z}_{\beta}/S_{\beta})$ is the relative Hilbert scheme of points on the universal curve $\mathcal{Z}_{\beta}$, and $P_{-}(S,-)$ is the moduli space of stable pairs on $S$. Let $\mathcal{O}_{S\times P}\to\mathbb{F}$ be the universal stable pair over $S\times P_{n_{2}-\beta\cdot(\beta+K_{S})/2}(S,\beta)$, and let $\mathbb{I}^{\bullet}$ be the associated complex. In this case $F^{\bullet\vee}$ is given by

[TABLE]

Here by $\mathcal{I}_{\mathcal{Z}^{[n_{2}]}\subset\mathcal{Z}_{\beta}}$ we mean the push-forward of the ideal sheaf of $\mathcal{Z}^{[n_{2}]}$ as a subscheme of $\mathcal{Z}_{\beta}\subset S\times S^{[0,n_{2}]}_{\beta}$. The last isomorphism above follows from (91) in [KT14]. We have shown that in this case $F^{\bullet\vee}$ coincides with virtual tangent bundle of the stable pair moduli space $P_{n_{2}-\beta\cdot(\beta+K_{S})/2}(S,\beta)$. Moreover, by the same reasoning given for the proof of part 4, one can see by inspection that under condition (22), the $K$-group class of $(F^{\bullet}_{\operatorname{red}})^{\vee}$ coincides with the $K$-group class of the reduced virtual tangent bundle over $P_{n_{2}-\beta\cdot(\beta+K_{S})/2}(S,\beta)$ constructed in [KT14]. This finishes the proof of part 5.

∎

4 Punctual nested Hilbert schemes

We will discuss a few tools for evaluating the virtual fundamental class $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ constructed in Corollary 2.6. We first develop a localization formula (36) in the case that $S$ is toric along the lines of [MNOP06]. When $S$ is toric we express $\iota_{*}[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ as the top Chern class of a vector bundle over the product of Hilbert schemes $S^{[n_{1}]}\times S^{[n_{2}]}$ (see Proposition 4.4). We have not been able to prove such a formula for general projective surfaces. Instead, we prove a weaker statement for general projective surfaces in which the integral of certain cohomology classes against $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ is expressed in terms of integrals over $S^{[n_{1}]}\times S^{[n_{2}]}$. This is done by using degeneration and the double point relations (see Corollary 4.13, Proposition 4.14). Such integrals arise in all the applications that we have in mind, particularly, they are related to the localized DT invariants of $S$ discussed in [GSY17b]. In Section 5 we express some of these integrals against $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ in terms of Carlsson-Okounkov’s vertex operators and as a result obtain explicit product formulas for their generating series.

Recall that

[TABLE]

For simplicity in this section, we denote by $I_{i}$ the ideal sheaf $I_{Z_{i}}$ of $Z_{i}$. Hence for any closed point $(Z_{1},Z_{2})\in S^{[n_{1}\geq n_{2}]}$ we have $I_{1}\subseteq I_{2}$. Sometimes, we denote the closed point above by the pair $(I_{1},I_{2})$, or by $I_{1}\subseteq I_{2}$, when we want to emphasize the inclusion of subschemes. As before, we have the universal objects over $S\times S^{[n_{1}\geq n_{2}]}$:

[TABLE]

We will use the following simple lemma in Section 4.1:

Lemma 4.1**.**

If $(I_{1}\subseteq I_{2})\in S^{[n_{1},n_{2}]}$ is a closed point, then

[TABLE] 2. 2.

If $(I_{1},I_{2})\in S^{[n_{1}]}\times S^{[n_{2}]}\setminus S^{[n_{1},n_{2}]}$ is a closed point then $\operatorname{Hom}_{S}(I_{1},I_{2})=0$. 3. 3.

If $p_{g}(S)=0$ and if $(I_{1},I_{2})\in S^{[n_{1}]}\times S^{[n_{2}]}$ is a closed point then $\operatorname{Ext}^{2}_{S}(I_{i},I_{j})=0$.

Proof.

Applying the functor $\operatorname{Hom}(I_{1},-)$ to the short exact sequence $0\to I_{2}\to\mathcal{O}_{S}\to\mathcal{O}_{Z_{2}}\to 0$, we get the exact sequence

[TABLE]

where $u$ composes any map $I_{1}\to\mathcal{O}_{S}$ with the natural map $\mathcal{O}_{S}\to\mathcal{O}_{Z_{2}}$. In part 1 the inclusion $I_{1}\subseteq I_{2}$ gives a nonzero element of $\operatorname{Hom}(I_{1},I_{2})$ and hence the claim follows. In part 2, $u(I_{1}\subset\mathcal{O}_{S})\neq 0$ because $I_{1}\not\subset I_{2}$, and so the claim is proven. For part 3, applying the functor $\operatorname{Hom}(I_{j},-)$ to the short exact sequence $0\to I_{i}\otimes\omega_{S}\to\omega_{S}\to\mathcal{O}_{Z_{i}}\to 0$, we get

[TABLE]

and so the claim follows by Serre duality. ∎

As will become clear shortly, the following $K$-group element plays an important role in the rest of the paper:

Definition 4.2**.**

For any line bundles $M$ on $S$, let $\mathsf{E}_{M}^{n_{1},n_{2}}\in K(S^{[n_{1}]}\times S^{[n_{2}]})$ be the element of rank $n_{1}+n_{2}$ defined by

[TABLE]

where $p$ and $\pi$ are respectively the projections from $S\times S^{[n_{1}]}\times S^{[n_{2}]}$ to the first and the product of the last two factors. Similarly, we define the twisted tangent bundle as the rank $2n_{i}$ element of $K(S^{[n_{1}]}\times S^{[n_{2}]})$

[TABLE]

*Note that $\mathsf{T}^{\mathcal{O}_{S}}_{S^{[n_{i}]}}$ is the class of (pullback of) the usual tangent bundle of $S^{[n_{i}]}$. If $M=\mathcal{O}_{S}$, we sometimes drop it from the notation. *

4.1 Toric surfaces

Let $(\mathbb{C}^{2})^{[n_{1}\geq n_{2}]}$ be the nested Hilbert scheme of points on $\mathbb{C}^{2}=\text{Spec}(R)$, where $R=\mathbb{C}[x_{1},x_{2}]$. The $2$-dimensional torus $\mathbf{T}$ acts on $\mathbb{C}^{2}$. We denote by $t_{1},t_{2}$ the torus characters, such that the tangent space at $0\in\mathbb{C}^{2}$ has the $\mathbf{T}$-character $t_{1}^{-1}+t_{2}^{-1}$. The $\mathbf{T}$-fixed set

[TABLE]

is isolated, and is given by the inclusion of the monomial ideals $I_{1}\subseteq I_{2}$ or equivalently the corresponding nested partitions $\mu^{\prime}\subseteq\mu$. By Proposition 2.5 and Lemma 4.3, the virtual tangent space at the $\mathbf{T}$-fixed point $I_{1}\subseteq I_{2}$ is given by999This is obtained by taking the derived restriction of the complex $F^{\bullet}$ to the point $I_{1}\subseteq I_{2}$, and then taking the $K$-group class of the resulting complex. Also note that by slightly modifying the proof of part 1 of Lemma 4.1, $\operatorname{Hom}(I_{1},I_{1})=\operatorname{Hom}(I_{1},I_{2})=\operatorname{Hom}(I_{2},I_{2})=R$.

[TABLE]

where $\chi(-,-)=\sum_{i=0}^{2}(-1)^{i}\operatorname{Ext}_{R}^{i}(-,-)$. By the exact method as in [MNOP06, Section 4.6] using Taylor resolutions and Čech complexes, the $\mathbf{T}$-representation of $\mathcal{T}^{\operatorname{vir}}_{I_{1}\subseteq I_{2}}$ can be explicitly written down as a Laurent polynomial in $t_{1}$ and $t_{2}$. For the $\mathbf{T}$-fixed 0-dimensional subschemes $Z_{2}\subseteq Z_{1}\subset\mathbb{C}^{2}$ corresponding to the monomial ideals $I_{1}\subseteq I_{2}$ define

[TABLE]

Here, $P_{1}$, $P_{2}$ are the Poincaré polynomials associated to the monomial ideals $I_{1}$ and $I_{2}$ (defined using their Taylor resolutions, see [MNOP06, Section 4.7]), respectively. Also, define

[TABLE]

Putting these expressions into (35) and simplifying, we get

[TABLE]

Now if $S$ is a toric surface, then the set of $\mathbf{T}$-fixed points of $S^{[n_{1}\geq n_{2}]}\subset S^{[n_{1}]}\times S^{[n_{2}]}$ is again isolated (Lemma 4.3), and the $\mathbf{T}$-character of the virtual tangent space at any fixed point is obtained by summing over the expression (36) for all the $\mathbf{T}$-invariant open subsets of $S$. This finishes the proof of Theorem 2.

Lemma 4.3**.**

Suppose that $S$ is a nonsingular projective toric surface, and $Z_{2}\subseteq Z_{1}$ is a $\mathbf{T}$-fixed point of $S^{[n_{1}\geq n_{2}]}$, then $\operatorname{Ext}^{2}_{S}(I_{1},I_{1})=\operatorname{Ext}^{2}_{S}(I_{2},I_{2})=\operatorname{Ext}^{2}_{S}(I_{1},I_{2})=0$, the $\mathbf{T}$-representations

[TABLE]

contain no trivial sub-representations.

Proof.

The vanishings in the lemma follow from the fact that $p_{g}(S)=0$ for toric surfaces, and part 3 of Lemma 4.1. For any fixed point $\alpha\in S$, let $U_{\alpha}\cong\mathbb{C}^{2}$ be the $\mathbf{T}$-invariant open neighborhood of $\alpha$, and let $I_{i,\alpha}:=I_{i}|_{\alpha}$, and $\mathcal{O}_{i,\alpha}:=\mathcal{O}_{Z_{i}}|_{\alpha}$. By [ES87, Lemma 3.2], $\operatorname{Hom}_{U_{\alpha}}(I_{i,\alpha},\mathcal{O}_{i,\alpha})$ contains no trivial subrepresentations. Therefore,

[TABLE]

contains no trivial representations either (in the first isomorphism we used the vanishing $H^{1}(\mathcal{O}_{S})=0$ for toric surfaces).

Next, applying $\operatorname{Hom}_{S}(I_{1},-)$ to the natural short exact sequence $0\to I_{2}\to\mathcal{O}_{S}\to\mathcal{O}_{Z_{2}}\to 0$, we obtain the exact sequence

[TABLE]

To finish the proof it suffices to show that the 1st and the 3rd terms in (37) contain no trivial representations. The claim for the 1st term in (37) follows from the fact that for each $\alpha$, $\mathbf{T}$ acts with different weights on the $\mathbb{C}$-basis elements for $I_{1,\alpha}$ and $\mathcal{O}_{2,\alpha}$ that are given by the monomials (because of the inclusion $I_{1,\alpha}\subseteq I_{2,\alpha}$). The claim for the 3rd term in (37) also follows because, applying $\operatorname{Hom}_{S}(-,\mathcal{O}_{S})$ to the natural short exact sequence $0\to I_{1}\to\mathcal{O}_{S}\to\mathcal{O}_{Z_{1}}\to 0$, and using equivariant Serre duality, we get

[TABLE]

But since $Z_{1}$ is zero dimensional and $\mathbf{T}$-fixed

[TABLE]

For each $\alpha$, let $\mu_{\alpha}$ be the partition corresponding to $Z_{1}|_{U_{\alpha}}$, and suppose that the $\mathbf{T}$-character of $T_{\alpha}S$ is $t_{1}^{-1}+t_{2}^{-1}$ for some $\mathbf{T}$-characters $t_{1}$ and $t_{2}$, then, the fiber of $\omega_{S}$ at $\alpha$ has the $\mathbf{T}$-character $t_{1}t_{2}$, and therefore,

[TABLE]

has no trivial representations. ∎

4.2 Proof of Theorem 3

Suppose that $S$ is a toric surface, and $(I_{1},I_{2})\in S^{[n_{1}]}\times S^{[n_{2}]}$ is a closed point. By Lemma 4.3

[TABLE]

Therefore by basechange, the sheaves

[TABLE]

are vector bundles over $S^{[n_{1}\geq n_{2}]}$ of ranks $2n_{1},2n_{2},n_{1}+n_{2}$, respectively. Moreover, the virtual tangent bundle of Proposition 2.5, simplifies to the 2-term complex

[TABLE]

Recall that the rank of $F^{\bullet\vee}$ is equal to $n_{1}+n_{2}$, and recall the $K$-group elements $\mathsf{E}^{n_{1},n_{2}}$ and $\mathsf{E}^{n_{1},n_{2}}|_{(I_{1},I_{2})}$ of ranks $n_{1}+n_{2}$ from Definition 4.2. We have

[TABLE]

This is true because of basechange, the vanishing (38), the vanishing $H^{1}(\mathcal{O}_{S})=H^{2}(\mathcal{O}_{S})=0$, and that by Lemma 4.1,

[TABLE]

Note that this is consistent with the fact that the dimension of $\operatorname{Ext}^{1}(I_{1},I_{2})$ jumps by 1 on $S^{[n_{1},n_{2}]}\subset S^{[n_{1}]}\times S^{[n_{2}]}$, and that the dimension of $\mathsf{E}^{n_{1},n_{2}}|_{(I_{1},I_{2})}$ is constant over $S^{[n_{1}]}\times S^{[n_{2}]}$.

Now we are ready to express the main result of this section relating the push forward of $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ to the products of the fundamental classes of Hilbert scheme of points. The following proposition proves Theorem 3.

Proposition 4.4**.**

Suppose that $S$ is a nonsingular projective toric surface, then,

[TABLE]

where $\iota$ is the natural inclusion $S^{[n_{1}\geq n_{2}]}\hookrightarrow S^{[n_{1}]}\times S^{[n_{2}]}$.

Proof.

Let $i$ and $j$ be inclusion of the fixed point set in $S^{[n_{1}\geq n_{2}]}$ and $S^{[n_{1}]}\times S^{[n_{2}]}$, respectively. By (39) and Lemma 4.3, the virtual localization formula (see [GP99]) gives

[TABLE]

where the sum is over the isolated $\mathbf{T}$-fixed points, and $e(-)$ indicates the equivariant Euler class. By Lemma 4.3, the coefficient of $i_{*}[(I_{1}\subseteq I_{2})]$ in the last sum is the product of the pure nontrivial torus weights. On the other hand, by Lemma 4.3 and the Atiyah-Bott localization formula

[TABLE]

where the last equality is because of (40), and the fact that since $H^{0}(\mathcal{O}_{S})\cong\mathbb{C}$ is the trivial $\mathbf{T}$-representation, we have $e(H^{0}(\mathcal{O}_{S}))=0$. The proposition is proven by comparing the outcomes of both localization formulas above, and taking the non-equivariant limit at the end. ∎

4.3 Relative nested Hilbert schemes

In this section we sketch how the degeneration formula of Li and Wu can be applied to the case of nested Hilbert scheme of points. Let $(S,D)$ be a pair of nonsingular projective surface and a nonsingular effective divisor. Li and Wu [LW15] introduced the notion of a stable relative ideal sheaf. $I\in S^{[n]}$ is said to be relative to $D$ if the natural map

[TABLE]

is injective (see also [MNOPII]). This is equivalent to $\mathcal{O}_{S}/I$ having support disjoint from $D$. Relativity is an open condition in $S^{[n]}$. Li and Wu constructed a relative Hilbert scheme, denoted by $(S/D)^{[n]}$, by considering the equivalence classes of the stable relative ideal sheaves on the $k$-step semistable models $S[k]$ for $0\leq k\leq n$. Let $D_{0},\dots,D_{k-1}$ be the singular locus of $S[k]$ and $D_{k}\subset S[k]$ be the proper transform of $D$. $S[k]$ consists of $k+1$ irreducible components $\Delta_{0},\dots,\Delta_{k}$ with $\Delta_{0}=S$ and $D_{i}=\Delta_{i}\cap\Delta_{i+1}$ for $i=0,\dots,k-1$. A relative ideal sheaf $I$ on $S[k]$ satisfies (41) for $D=D_{0},\dots,D_{k}$. Two relative ideal sheaves $I$ and $I^{\prime}$ on $S[k]$ are equivalent if the quotients $\mathcal{O}_{S[k]}/I$ and $\mathcal{O}_{S[k]}/I^{\prime}$ differ by an automorphism of $S[k]$ covering the identity on $\Delta_{0}=S$. The stability of a relative ideal sheaf means that it has finitely many automorphisms as described above. $(S/D)^{[n]}$ is a smooth proper Deligne-Mumford stack of dimension $2n$.

Since by the relativity condition for any relative ideal sheaf $I$, $I|_{D_{j}}\cong\mathcal{O}_{D_{j}}$, the generalization of Li-Wu Hilbert schemes to the set up of the nested Hilbert schemes is straightforward. In other words, we can construct a proper Deligne-Mumford stack $(S/D)^{[n_{1}\geq n_{2}]}$ as the moduli space of relative ideal sheaves $I_{1}$ and $I_{2}$ with $I_{1}$ stable and $I_{1}\subseteq I_{2}$101010Note that if $Z_{i}\subset S[k]$ is the 0-dimension subscheme corresponding to $I_{i}$, then the number of the auto-equivalences of $Z_{2}\subseteq Z_{1}\subset S[k]$ is less than or equal to that of $Z_{1}\subset S[k]$, which is finite by the stability of $I_{1}$..

Notation**.**

Following [LW15, Secttion 2.3], let $\mathfrak{A}_{\diamond}$ be the Artin stack of expanded degenerations for the pair $(S,D)$, and let $\mathcal{S}\to\mathfrak{A}_{\diamond}$ be the universal family of surfaces over it. They fit into the fibered diagram

[TABLE]

Let $(S/D)^{[n_{1}\geq n_{2}]}\to\mathfrak{A}_{\diamond}$ be the natural morphism; it factors through the substack $\mathfrak{A}_{\diamond}^{[{\boldsymbol{n}}]}\subset\mathfrak{A}_{\diamond}$ corresponding to the numerical data ${\boldsymbol{n}}$ (see [LW15, Secttion 2.5] for the construction of these substacks111111Since we are dealing with zero dimensional subschemes their Hilbert polynomials (used in [LW15]) are simply the nonnegative integers $n_{1},n_{2}$.). $\mathfrak{A}_{\diamond}^{[{\boldsymbol{n}}]}$ is a smooth Artin stack of dimension 0. We use the same notation as in the absolute case to denote the inclusion of the universal objects over $\mathcal{S}\times_{\mathfrak{A}^{[{\boldsymbol{n}}]}_{\diamond}}(S/D)^{[{\boldsymbol{n}}]}$:

[TABLE]

Let $\pi$ be the projection to the second factor of $\mathcal{S}\times_{\mathfrak{A}^{[{\boldsymbol{n}}]}_{\diamond}}(S/D)^{[{\boldsymbol{n}}]}$, and $p$ be the projection to its first factor followed by the natural map $\mathcal{S}\to S$.

By the method of [MPT10, Section 3.9] and [LW15], one can see, after modifying our argument for the usual nested Hilbert schemes (Proposition 2.5), that there is a relative perfect obstruction theory $\mathcal{F}_{\operatorname{rel}}^{\bullet}\to\mathbb{L}^{\bullet}_{(S/D)^{[n_{1}\geq n_{2}]}/\mathfrak{A}^{[{\boldsymbol{n}}]}_{\diamond}}$ with the relative virtual tangent bundle

[TABLE]

Since the Artin stack $\mathfrak{A}_{\diamond}^{[{\boldsymbol{n}}]}$ is smooth of dimension 0 by [BF97, Section 7] there is a virtual fundamental class

[TABLE]

associated to this relative perfect obstruction theory.

Let $X:S\rightsquigarrow S_{0}:=S_{1}\cup_{D}S_{2}$ be a good degeneration of the surface $S$ along $D$ over a pointed curve $(C,0)$121212It means we have a nonsingular threefold $X$ over $C$, whose general fibers are isomorphic to $S$, and whose fiber over $0\in C$ is a normal crossing divisor $S_{1}\cup_{D}S_{2}$ consisting of two nonsingular surfaces $S_{1},S_{2}$ glued along a nonsingular divisor isomorphic to $D$ and contained in $S_{1}$ and $S_{2}$., and let

[TABLE]

be the universal family of surfaces over the stack of expanded degenerations $\mathfrak{C}$ (see [L01, L02],[LW15, Sections 2.1-2.2]). They fit into the fibered diagram

[TABLE]

Following the construction of Li and Wu [LW15], one can construct the nested Hilbert scheme of points, denoted by $\mathfrak{S}^{[n_{1}\geq n_{2}]}$ on the fibers of $\mathfrak{S}$. The Hilbert scheme $\mathfrak{S}^{[n_{1}\geq n_{2}]}$ is a proper Deligne-Mumford stack over $\mathfrak{C}$ and its structure morphism factors through the substack $\mathfrak{C}^{[{\boldsymbol{n}}]}\subset\mathfrak{C}$ corresponding to the numerical data ${\boldsymbol{n}}$ (see [LW15, Secttion 2.5] for the construction of these substacks). $\mathfrak{C}^{[{\boldsymbol{n}}]}$ is a smooth Artin stack of dimension 1. Let $\mathfrak{C}^{[{\boldsymbol{n}}]}_{0}\subset\mathfrak{C}^{[{\boldsymbol{n}}]}$ be the substack corresponding to $0\in C$. A non-special fiber of $\mathfrak{S}^{[n_{1}\geq n_{2}]}$ is isomorphic to $S^{[n_{1}\geq n_{2}]}$, whereas the special fiber of $\mathfrak{S}^{[n_{1}\geq n_{2}]}$, denoted by $S^{[n_{1}\geq n_{2}]}_{0}$, can be written as the (non-disjoint) union

[TABLE]

where ${\boldsymbol{n}}^{\prime}=(n^{\prime}_{1},n^{\prime}_{2})$ and ${\boldsymbol{n}}^{\prime\prime}=(n^{\prime\prime}_{1},n^{\prime\prime}_{2})$ with $n^{\prime}_{1}\geq n^{\prime}_{2}$ and $n^{\prime\prime}_{1}\geq n^{\prime\prime}_{2}$. Each component

[TABLE]

is the pull-back of a divisor $\mathfrak{C}^{[{\boldsymbol{n}}]}_{{\boldsymbol{n}}^{\prime},{\boldsymbol{n}}^{\prime\prime}}\subset\mathfrak{C}^{[{\boldsymbol{n}}]}$. Let $\mathfrak{L}^{[{\boldsymbol{n}}]}_{{\boldsymbol{n}}^{\prime},{\boldsymbol{n}}^{\prime\prime}}$ be the corresponding line bundle. We then have

[TABLE]

where $\mathfrak{L}_{0}$ is the line bundle associated to the pull back of the divisor $\{0\}\subset C$.

We denote the universal objects over $\mathfrak{S}\times_{\mathfrak{C}^{[{\boldsymbol{n}}]}}\mathfrak{S}^{[n_{1}\geq n_{2}]}$ by

[TABLE]

The restriction of $\Phi$ to the component $\mathfrak{S}\times_{\mathfrak{C}^{[{\boldsymbol{n}}]}}\left((S_{1}/D)^{[n^{\prime}_{1}\geq n^{\prime}_{2}]}\times(S_{2}/D)^{[n^{\prime\prime}_{1}\geq n^{\prime\prime}_{2}]}\right)$ is identified with the pair of universal maps

[TABLE]

We denote by $\pi$ the projection to the second factor of $\mathfrak{S}\times_{\mathfrak{C}^{[{\boldsymbol{n}}]}}\mathfrak{S}^{[n_{1}\geq n_{2}]}$, and by $p$ the projection to its first factor followed by the natural morphism to the total space of the good degeneration of $S$ over $C$. Again by the method of Section 2.1 and [MPT10, LW15], one can construct a relative perfect obstruction theory $\mathfrak{F}^{\bullet}_{\operatorname{rel}}\to\mathbb{L}^{\bullet}_{\mathfrak{S}^{[n_{1}\geq n_{2}]}/\mathfrak{C}^{[{\boldsymbol{n}}]}}$ with the relative virtual tangent bundle:

[TABLE]

Since $\mathfrak{C}^{[{\boldsymbol{n}}]}$ is a smooth Artin stack ($h^{-1}(\mathbb{L}^{\bullet}_{\mathfrak{C}^{[{\boldsymbol{n}}]}})=0$ but $h^{1}(\mathbb{L}^{\bullet}_{\mathfrak{C}^{[{\boldsymbol{n}}]}})\neq 0$) there exists an absolute perfect obstruction theory $\mathfrak{F}^{\bullet}\to\mathbb{L}^{\bullet}_{\mathfrak{S}^{[n_{1}\geq n_{2}]}}$ associated to $\mathfrak{F}_{\operatorname{rel}}^{\bullet}$ (see for example the argument after diagram (49) in [MPT10]). The restriction of $\mathfrak{F}_{\operatorname{rel}}^{\bullet}$ to $S_{0}^{[{\boldsymbol{n}}]}$ and its components $S_{0,{\boldsymbol{n}}^{\prime},{\boldsymbol{n}}^{\prime\prime}}^{[{\boldsymbol{n}}]}$ induce relative perfect obstruction theories

[TABLE]

respectively. As in [MPT10], they satisfy the following compatibilities:

[TABLE]

where each sequence is an exact triangle.

A decomposition $S_{0}[k_{1},k_{2}]:=S_{1}[k_{1}]\cup_{D}S_{2}[k_{2}]$ yields the natural exact sequence

[TABLE]

Suppose that $I_{1}\subseteq I_{2}$ is a nested pairs of relative ideal sheaves on $S_{0}[k_{1},k_{2}]$, and let $I_{i}^{\prime}:=I_{i}|_{S_{1}[k_{1}]}$ and $I^{\prime\prime}_{i}:=I_{i}|_{S_{2}[k_{2}]}$. Tensoring the short exact sequence above with the perfect complexes $\mathbf{R}\mathcal{H}om(I_{1},I_{1})$, $\mathbf{R}\mathcal{H}om(I_{2},I_{2})$, and $\mathbf{R}\mathcal{H}om(I_{1},I_{2})$ and applying $\mathbf{R}\Gamma$ we get the commutative diagram

[TABLE]

where each row is an exact triangle and the the first two vertical maps are induced from the natural inclusions $I_{1}\subseteq I_{2}$, $I^{\prime}_{1}\subseteq I^{\prime}_{2}$ and $I^{\prime\prime}_{1}\subseteq I^{\prime\prime}_{2}$ as in Section 2.1, and in the thrid column we have used the relativity condition of ideal sheaves i.e. $I_{i}|_{D}=\mathcal{O}_{D}$. As before the vertical maps factor through the trace free parts and hence, using the natural exact triangle

[TABLE]

this induces the following commutative diagram of the exact triangles

[TABLE]

Taking the cones we get the isomorphism

[TABLE]

One of the upshots is that following the construction of [MPT10, LW15], we are led by the isomorphism above to the following degeneration formula for the virtual integration over $S^{[n_{1}\geq n_{2}]}$ ([MPT10, Thm. 16], [LW15, Prop. 6.5, Thm. 6.6]). This is done by using the compatibilities (45) and relating the relative prefect obstruction theory $\mathfrak{F}_{\operatorname{rel}}^{\bullet}$ to the absolute perfect obstruction theories $F^{\bullet}$ (given in Proposition 2.5) and $\mathcal{F}^{\bullet}_{\operatorname{rel}}$ (given by (42)):

Proposition 4.5**.**

Let $\alpha$ be a cohomology class in the total space of $\mathfrak{S}^{[n_{1}\geq n_{2}]}$, then,

[TABLE]

∎

Remark 4.6**.**

In Proposition 4.5, if $n=n_{1}=n_{2}$ then $\mathfrak{S}^{[n_{1}\geq n_{2}]}\cong\mathfrak{S}^{[n]}$ constructed by [LW15], and by the same argument as in proof of Theorem 3.1 part 1, one can recover the usual degeneration formula for the Hilbert schemes of points used in [T12, LT14, GS16]:

[TABLE]

4.4 Double point relation

Let $(S,D)$ be a pair of nonsingular projective surface and a nonsingular effective divisor as in Section 4.3, and let $M$ be line bundle on $S$. Also, recall the definitions of

[TABLE]

Let $\mathcal{D}\subset\mathcal{S}$ be the proper transform of $D\subset S$ via $p$. Note that we use $\pi$ and $p$ for the similar natural morphisms from $\mathfrak{S}\times_{\mathfrak{C}^{[{\boldsymbol{n}}]}}\mathfrak{S}^{[n_{1}\geq n_{2}]}$ as well.

Definition 4.7**.**

Define the following element in $K((S/D)^{[n_{1}\geq n_{2}]})$ of rank $n_{1}+n_{2}$:

[TABLE]

Define the following generating series:

[TABLE]

If $D=0$ we drop it from the notation.

Lemma 4.8**.**

Given a good degeneration $S\rightsquigarrow S_{0}:=S_{1}\cup_{D}S_{2}$, and a choice of degeneration of line bundles

[TABLE]

we get the degeneration of the class $c(\mathsf{K}^{[n_{1}\geq n_{2}]}_{M})$ whose restriction to the component

[TABLE]

of the central fiber of $\mathfrak{S}^{[n_{1}\geq n_{2}]}$ is $c(\mathsf{K}^{[n^{\prime}_{1}\geq n^{\prime}_{2}]}_{M_{1}})\boxtimes c(\mathsf{K}^{[n^{\prime\prime}_{1}\geq n^{\prime\prime}_{2}]}_{M_{2}}).$

Proof.

Let $\mathcal{M}$ be the line bundle over the total space of the good degeneration of $S$ that gives the degeneration of $M$ as in the lemma. The derived pullbacks of the perfect complexes $\mathbf{R}\mathcal{H}om_{\pi}(\mathfrak{I}^{[n_{1}]},\mathfrak{I}^{[n_{2}]}\otimes p^{*}\mathcal{M})$ and $\mathbf{R}\pi_{*}p^{*}\mathcal{M}$ to the component $(S_{1}/D)^{[n^{\prime}_{1}\geq n^{\prime}_{2}]}\times(S_{2}/D)^{[n^{\prime\prime}_{1}\geq n^{\prime\prime}_{2}]}$ fits in the exact triangles

[TABLE]

and $\mathbf{R}\pi_{*}p^{*}\mathcal{M}\to\oplus_{i=1}^{2}\mathbf{R}\pi_{*}p^{*}M_{i}\to\mathbf{R}\pi_{*}p^{*}M|_{D}.$ Now taking the difference of the $K$-group classes from the exact triangles above, and applying the total Chern class, we conclude that $c(\mathsf{K}^{[n_{1}\geq n_{2}]}_{M})$ degenerates to a class whose restriction to the component

[TABLE]

is $c(\mathsf{K}^{[n^{\prime}_{1}\geq n^{\prime}_{2}]}_{M_{1}}\boxplus\mathsf{K}^{[n^{\prime\prime}_{1}\geq n^{\prime\prime}_{2}]}_{M_{2}})$.

∎

A direct corollary of Proposition 4.5 and Lemma 4.8 is

Proposition 4.9**.**

Given a good degeneration $S\rightsquigarrow S_{0}:=S_{1}\cup_{D}S_{2}$, and a choice of degeneration of line bundles

[TABLE]

we have

[TABLE]

∎

In the situation of Proposition 4.9, Let $\mathbb{P}$ be either of the projective bundles $\mathbb{P}(\mathcal{O}_{D}+N_{S_{1}/D})\cong\mathbb{P}(\mathcal{O}_{D}+N_{S_{2}/D})$, and let $M_{\mathbb{P}}$ be the pullback of $M|_{D}$ to $\mathbb{P}$. Applying Proposition 4.9 to the degeneration to the normal cone of $D\subset S_{i}$ gives

[TABLE]

Similarly, the degeneration to the normal cone of $D\subset\mathbb{P}$ gives

[TABLE]

Let $\mathcal{M}_{2,1}(\mathbb{C})^{+}$ be the group completion of the set of isomorphism classes of the pairs $(S,M)$, where $S$ is a smooth projective surface over $\mathbb{C}$ and $M$ is a line bundle on $S$ (see [LP12, Definition 3] ).

Corollary 4.10**.**

$Z_{\operatorname{nest}}(-,-)$* satisfies the relation131313This relation is the analog of the relation (0.10) in [LP09].*

[TABLE]

and hence it respects the double point relations in $\mathcal{M}_{2,1}(\mathbb{C})^{+}$. In other words, $Z_{\operatorname{nest}}(-,-)$ descends to a homomorphism

[TABLE]

where $\omega_{2,1}(\mathbb{C})$ is the double point cobordism theory for line bundles on surfaces obtained by taking the quotient of $\mathcal{M}_{2,1}(\mathbb{C})^{+}$ by all the double point relations.

Proof.

Relation (50) follows immediately from relations (47)-(49). ∎

It is known that $\omega_{2,1}(\mathbb{C})$ is generated by the following classes (see [T12, LP12])

[TABLE]

Let $B_{1},B_{2},B_{3},B_{4}$ be $Z_{\operatorname{nest}}(S,M)$ where $(S,M)$ is one of the pairs above respectively from left to right. Define

[TABLE]

Proposition 4.11**.**

Let $S$ be a nonsingular projective surface and $M$ be a line bundle on $S$. Let $A_{1},A_{2},A_{3},A_{4}\in\mathbb{Q}[[q_{1},q_{2}]]^{*}$ be defined as above (independent of $S$) then

[TABLE]

Proof.

By basechange,

[TABLE]

and so by Proposition 4.4, we have

[TABLE]

if $(S,M)$ is one of the generators (51). For a general $(S,M)$ as in the proposition we can express the class $[S,M]\in\omega_{2,1}(\mathbb{C})$ as a linear combination of the generators (51) [T12, Proposition 4.1]. The result then follows by applying the homomorphism $Z_{\operatorname{nest}}(-,-)$ of Corollary 4.10 and then rearranging the factors as in the proof of [T12, Proposition 4.1]. ∎

Corollary 4.12**.**

Let $S$ be a nonsingular projective surface and $M$ be a line bundle on $S$. Then the integral

[TABLE]

can be written as a degree $n_{1}+n_{2}$ universal polynomial in $M^{2},M\cdot K_{S},K_{S}^{2},c_{2}(S)$.

Proof.

The integral in the proposition is the coefficient of $q_{1}^{n_{1}}q_{2}^{n_{2}}$ in $Z_{\operatorname{nest}}(S,M)$. The result follows after expanding the right hand side of the formula in Proposition 4.11 and extracting the coefficient of $q_{1}^{n_{1}}q_{2}^{n_{2}}$. ∎

Corollary 4.13**.**

For any nonsingular projective surface $S$ and $M\in\operatorname{Pic}(S)$ let $\mathcal{P}:=c(\mathsf{K}^{[n_{1}\geq n_{2}]}_{M})$. Then

[TABLE]

Proof.

By Corollary 4.12 we know that the LHS of the statement above i.e.

[TABLE]

is a universal polynomial $P_{1}$ in $M^{2},M\cdot K_{S},K_{S}^{2},c_{2}(S)$. On the other hand, using Grothendieck-Riemann-Roch formula and the induction scheme of Ellingsrud, Göttsche, and Lehn (see [EGL99, Sections 3, 4] and [CO12, Section 3]), we can express the RHS of the corollary in terms of a universal polynomial $P_{2}$ in $M^{2},M\cdot K_{S},K_{S}^{2},c_{2}(S)$. But since the equality in the corollary holds for any $(S,M)$ in which $S$ is toric (by Proposition 4.4), we conclude that $P_{1}=P_{2}$, and the result follows.

∎

This corollary allows us to write integrlas against $[S^{[n_{1}\geq n_{2}]}]^{\operatorname{vir}}$ in terms of integrations over the product of Hilbert schemes of points. As said in the introduction the following generalization of this corollary is used in [GSY17b] in the context of reduced localized DT invariants. In fact the only property of the integrand $c(\mathsf{K}^{[n_{1}\geq n_{2}]}_{M})$ that was needed in the argument leading to Corollary 4.13 was its decomposition property under good degenerations of $S$ as stated in Lemma 4.8. We can therefore prove similar identities as in Corollary 4.13 for other integrands with such a decomposition property. For example, by a same proof as Lemma 4.8 one can see that the Chern class of the twisted tangent bundle (Definition 4.2), $c(\mathsf{T}^{M}_{S^{[n_{i}]}})$, also has this property, and so does any product/quotient of these Chern classes. In particular, we can extend Corollary 4.13 to the following more general statement:

Proposition 4.14**.**

Let $L_{1},\dots,L_{s}$, $L^{\prime}_{1},\dots,L^{\prime}_{s^{\prime}}$, $M_{1},\dots,M_{t}$, be some line bundles on the nonsingular projective surface $S$, and $l_{1},\dots,l_{s}$, $l^{\prime}_{1},\dots,l^{\prime}_{s^{\prime}}$, $m_{1},\dots,m_{t}$ be finite sequences of $\pm 1$. Define

[TABLE]

Then,

[TABLE]

∎

5 Vertex operator formulas and proof of Theorem 4

Let $(S,M)$ be a pair of a projective nonsingular surface $S$ and a line bundles $M$ on $S$. Let $\mathsf{F}=\oplus_{n}H^{*}(S^{[n]},\mathbb{Q})$. Carlsson and Okounkov defined the operator $W(M_{1})$ in $\operatorname{End}(\mathsf{F})[[z_{1},z_{1}^{-1}]]$ by

[TABLE]

where $\langle-,-\rangle$ is the Poincaré pairing, $\eta_{i}\in H^{*}(S^{[n_{i}]},\mathbb{Q})$, $p_{i}$ is the projection to the $i$-th factor of $S^{[n_{1}]}\times S^{[n_{2}]}$, and $\mathsf{E}^{n_{1},n_{2}}_{M_{1}}\in K(S^{[n_{1}]}\times S^{[n_{2}]})$ is as in Definition 4.2. In other words, using [F13, Definition 16.1.2], $W(M_{1})$ is the operator associated to the family of correspondences

[TABLE]

If $M_{2}$ is another line bundle on $S$, we define

[TABLE]

By [F13, Proposition 16.1.2],

[TABLE]

where $\eta_{i}\in H^{*}(S^{[n_{i}]},\mathbb{Q})$, and $p_{i}$ is the projection to the $i$-th factor of $S^{[n_{1}]}\times S^{[n_{2}]}\times S^{[n_{3}]}$.

Carlsson and Okounkov found an explicit formula for $W(-)$ in terms of vertex operators. Let $\alpha_{\pm}(-)$ denote Nakajima’s annihilation/creation operators.

Theorem 5.1**.**

(Carlsson-Okounkov [CO12])

[TABLE]

where

[TABLE]

∎

Note that the operators $\Gamma_{\pm}$ satisfy the commutation relations $[\Gamma_{\pm},\Gamma_{\pm}]=0$, and moreover,

[TABLE]

Let $\mathsf{c}:=(1-\frac{z_{1}}{z_{2}})^{\langle M_{1},M_{2}^{D}\rangle}$. Using these properties, we can write

[TABLE]

Let $\mathsf{N}$ be the number-of-points operator: $\mathsf{N}|_{\mathsf{F}_{n}}=n\operatorname{id}$. It satisfies

[TABLE]

Starting with $\operatorname{str}\left(q^{\mathsf{N}}\circ W(M_{1},M_{2})\right)$ and using the commutation relation of the super-trace $\operatorname{str}(A\circ B)=\operatorname{str}(B\circ A)$, we obtain

[TABLE]

Iterating this process, we get

[TABLE]

where for the last equality, we have used the fact that $\Gamma_{+}$ is a lower triangular operator, and Göttsche’s formula

[TABLE]

Define

[TABLE]

then we have shown

[TABLE]

On the other hand, by [F13, Example 16.1.3],

[TABLE]

where the last equality is because of Grothendieck-Verdier duality.

Notation**.**

If $Z=\sum_{r_{1},r_{2}\geq 0}a_{r_{1},r_{2}}q_{1}^{r_{1}}q_{2}^{r_{2}}$ is a formal series, we define

[TABLE]

The following proposition completes the proof of Theorem 4:

Proposition 5.2**.**

Let $(S,M)$ be a pair of a projective nonsingular surface $S$ and a line bundles $M$ on $S$, then,

[TABLE]

Proof.

By (53),

[TABLE]

The result now follows immediately from (52) and Corollary 4.13.∎