Generalized Macdonald Functions on Fock Tensor Spaces and Duality Formula for Changing Preferred Direction

TL;DR

This paper derives an explicit formula for generalized Macdonald functions on Fock spaces, proving their factorization property related to the DIM algebra and providing insights into the 5D AGT correspondence and $SL(2,Z)$ duality.

Contribution

It presents a new explicit formula for generalized Macdonald functions and proves their factorization property, linking algebraic structures to the 5D AGT correspondence.

Findings

Explicit formula for generalized Macdonald functions on Fock tensor spaces.

Proof of the factorization property of matrix elements of multi-valent intertwining operators.

Connection of the factorization formula to $SL(2,Z)$ duality and 5D AGT correspondence.

Abstract

An explicit formula is obtained for the generalized Macdonald functions on the $N$-fold Fock tensor spaces, calculating a certain matrix element of a composition of several screened vertex operators. As an application, we prove the factorization property of the arbitrary matrix elements of the multi-valent intertwining operator (or refined topological vertex operator) associated with the Ding--Iohara--Miki algebra (DIM algebra) with respect to the generalized Macdonald functions, which was conjectured by Awata, Feigin, Hoshino, Kanai, Yanagida and one of the authors. Our proof is based on the combinatorial and analytic properties of the asymptotic eigenfunctions of the ordinary Macdonald operator of $A$-type, and the Euler transformation formula for Kajihara and Noumi's multiple basic hypergeometric series. That factorization formula provides us with a reasonable algebraic description of the 5D (K-theoretic) Alday-Gaiotto-Tachikawa (AGT) correspondence, and the interpretation of the invariance under the preferred direction from the point of view of the $SL(2,\mathbb{Z})$ duality of the DIM algebra.

Equations

$$\left(1-\frac{x}{z}\right)X^{(i)}(z)\mathcal{V}(x)=\left(1-(t/q{)}^i\frac{x}{z}\right)\mathcal{V}(x)X^{(i)}(z)(i=1,\dots ,N),$$

$$⟨K_{\mathbf{\lambda }}(\mathbf{v})|\mathcal{V}(x)|K_{\mathbf{\mu }}(\mathbf{u})⟩=$$

$$\displaystyle(a;q)_{\infty}:=\prod_{n=1}^{\infty}(1-q^{n-1}a),\quad(a;q)_{m}:=\left\{\begin{array}[]{c l}\displaystyle\prod_{n=1}^{m}(1-q^{n-1}a)&(m\geq 1);\\ 1&(m=0);\\ \displaystyle\prod_{n=1}^{-m}(1-q^{-n}a)^{-1}&(m\leq-1),\end{array}\right.$$

$${\theta }_q(a):=(a;q{)}_{\infty }(q{a}^{-1};q{)}_{\infty }.$$

$$x^{\pm }(z)=\sum _{n\in \mathbb{Z}}{x}_n^{\pm }{z}^{-n},{\psi }^{\pm }(z)=\sum _{\pm n\in {\mathbb{Z}}_{\ge 0}}{\psi }_n^{\pm }{z}^{-n},$$

$$\begin{split}&\psi^{+}(z)x^{\pm}(w)=g(c^{\mp 1/2}w/z)^{\mp 1}x^{\pm}(w)\psi^{+}(z),\qquad\psi^{-}(z)x^{\pm}(w)=g(c^{\mp 1/2}z/w)^{\pm 1}x^{\pm}(w)\psi^{-}(z),\\ &\psi^{\pm}(z)\psi^{\pm}(w)=\psi^{\pm}(w)\psi^{\pm}(z),\qquad\psi^{+}(z)\psi^{-}(w)=\dfrac{g(c^{+1}w/z)}{g(c^{-1}w/z)}\psi^{-}(w)\psi^{+}(z),\\ &[x^{+}(z),x^{-}(w)]=\dfrac{(1-q)(1-1/t)}{1-q/t}\bigg{(}\delta(c^{-1}z/w)\psi^{+}(c^{1/2}w)-\delta(cz/w)\psi^{-}(c^{-1/2}w)\bigg{)},\\ &G^{\mp}(z/w)x^{\pm}(z)x^{\pm}(w)=G^{\pm}(z/w)x^{\pm}(w)x^{\pm}(z)\,,\end{split}$$

$$g(z)=\frac{G^{+}(z)}{G^{-}(z)},G^{\pm }(z)=(1-q^{\pm 1}z)(1-t^{\mp 1}z)(1-q^{\mp 1}{t}^{\pm 1}z),\delta (z)=\sum _{n\in \mathbb{Z}}{z}^n.$$

$$\begin{array}{rl}& \Delta (c^{\pm 1/2})=c^{\pm 1/2}\otimes c^{\pm 1/2},\\ & \Delta (x^{+}(z))=x^{+}(z)\otimes 1+{\psi }^{-}(c_{(1)}^{1/2}z)\otimes x^{+}(c_{(1)}z),\\ & \Delta (x^{-}(z))=x^{-}(c_{(2)}z)\otimes {\psi }^{+}(c_{(2)}^{1/2}z)+1\otimes x^{-}(z),\\ & \Delta ({\psi }^{\pm }(z))={\psi }^{\pm }(c_{(2)}^{\pm 1/2}z)\otimes {\psi }^{\pm }(c_{(1)}^{\mp 1/2}z),\end{array}$$

$$[a_m,a_n]=m\frac{1-q^{|m|}}{1-t^{|m|}}{\delta }_{m+n,0}{a}_0,$$

$$\begin{array}{rl}& c^{1/2}\mapsto (t/q{)}^{1/4},x^{+}(z)\mapsto u{z}^{-M}{q}^{-M/2}{t}^{M/2}\eta (z),x^{-}(z)\mapsto u^{-1}{z}^M{q}^{M/2}{t}^{-M/2}\xi (z),\\ & {\psi }^{+}(z)\mapsto q^{M/2}{t}^{-M/2}{\phi }^{+}(z),{\psi }^{-}(z)\mapsto q^{-M/2}{t}^{M/2}{\phi }^{-}(z),\end{array}$$

$$\begin{split}&\eta(z)=\exp\Big{(}\sum_{n=1}^{\infty}\dfrac{1-t^{-n}}{n}a_{-n}z^{n}\Big{)}\exp\Big{(}-\sum_{n=1}^{\infty}\dfrac{1-t^{n}}{n}a_{n}z^{-n}\Big{)},\\ &\xi(z)=\exp\Big{(}-\sum_{n=1}^{\infty}\dfrac{1-t^{-n}}{n}q^{-n/2}t^{n/2}a_{-n}z^{n}\Big{)}\exp\Big{(}\sum_{n=1}^{\infty}\dfrac{1-t^{n}}{n}q^{-n/2}t^{n/2}a_{n}z^{-n}\Big{)},\\ &\varphi^{+}(z)=\exp\Big{(}-\sum_{n=1}^{\infty}\dfrac{1-t^{n}}{n}(1-t^{n}q^{-n})q^{n/4}t^{-n/4}a_{n}z^{-n}\Big{)},\\ &\varphi^{-}(z)=\exp\Big{(}\sum_{n=1}^{\infty}\dfrac{1-t^{-n}}{n}(1-t^{n}q^{-n})q^{n/4}t^{-n/4}a_{-n}z^{n}\Big{)}.\end{split}$$

$${\rho }_{\mathbf{u}}^{(N,0)}:=({\rho }_{u_1}\otimes {\rho }_{u_2}\otimes \cdots \otimes {\rho }_{u_N})\circ {\Delta }^{(N)},$$

$$\displaystyle\mbox{where $\Delta^{(1)}:=\mathrm{id}$, $\Delta^{(N)}:=(\underbrace{\mathrm{id}\otimes\cdots\otimes{\rm id}}_{N-2}\otimes\Delta)\circ\cdots\circ(\mathrm{id}\otimes\Delta)\circ\Delta$ ($N\geq 2$)}.$$

$$a_n^{(i)}=\stackrel{i-1}{\overbrace{1\otimes \cdots \otimes 1}}\otimes a_n\otimes \stackrel{N-i}{\overbrace{1\otimes \cdots \otimes 1}},$$

$${\Lambda }^{(i)}(z):={\phi }^{-}({\gamma }^{1/2}z)\otimes \cdots \otimes {\phi }^{-}({\gamma }^{i-3/2}z)\otimes \stackrel{i\text{-th Fock space}}{\overbrace{\eta ({\gamma }^{i-1}z)}}\otimes 1\otimes \cdots \otimes 1.$$

$${\rho }_{\mathbf{u}}^{(N,0)}(x^{+}(z))=\sum _{i=1}^N{u}_i{\Lambda }^{(i)}(z).$$

$$X^{(k)}(z)=\sum _{n\in \mathbb{Z}}{X}_n^{(k)}{z}^{-n}=X^{(1)}({\gamma }^{2(1-k)}z)X^{(1)}({\gamma }^{2(2-k)}z)\cdots X^{(1)}(z)\in \mathrm{End}({\mathcal{F}}_{\mathbf{u}})[[z^{\pm 1}]].$$

$$X^{(k)}(z)=\sum _{1\le j_1<\cdots <j_k\le N}:{\Lambda }^{(j_1)}(z)\cdots {\Lambda }^{(j_k)}((q/t{)}^{k-1}z):u_{j_1}\cdots u_{j_k}.$$

$$|X_{\mathbf{\lambda }}(\mathbf{u})⟩:=X_{-{\lambda }_1^{(1)}}^{(1)}{X}_{-{\lambda }_2^{(1)}}^{(1)}\cdots X_{-{\lambda }_1^{(2)}}^{(2)}{X}_{-{\lambda }_2^{(2)}}^{(2)}\cdots X_{-{\lambda }_1^{(N)}}^{(N)}{X}_{-{\lambda }_2^{(N)}}^{(N)}\cdots |0⟩,$$

$$⟨X_{\mathbf{\lambda }}(\mathbf{u})|:=⟨0|\cdots X_{{\lambda }_2^{(N)}}^{(N)}{X}_{{\lambda }_1^{(N)}}^{(N)}\cdots X_{{\lambda }_2^{(2)}}^{(2)}{X}_{{\lambda }_1^{(2)}}^{(2)}\cdots X_{{\lambda }_2^{(1)}}^{(1)}{X}_{{\lambda }_1^{(1)}}^{(1)}.$$

$$\mathbf{\lambda }\stackrel{\ast }{>}\mathbf{\mu }\stackrel{\mathrm{def}}{⟺}$$

$$(|{\lambda }^{(1)}|,|{\lambda }^{(2)}|,\dots ,|{\lambda }^{(N)}|)\ne (|{\mu }^{(1)}|,|{\mu }^{(2)}|,\dots ,|{\mu }^{(N)}|).$$

$$(\varnothing ,\varnothing ,(2))\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$$X_0^{(1)}\prod _{j=1}^N{P}_{{\lambda }^{(j)}}(a_{-n}^{(j)})|0⟩={ϵ}_{\mathbf{\lambda }}\prod _{j=1}^N{P}_{{\lambda }^{(j)}}(a_{-n}^{(j)})|0⟩+\sum _{\mathbf{\mu }\stackrel{\ast }{<}\mathbf{\lambda }}{v}_{\mathbf{\lambda },\mathbf{\mu }}\prod _{j=1}^N{P}_{{\mu }^{(j)}}(a_{-n}^{(j)})|0⟩,$$

$$⟨0|\prod _{j=1}^N{P}_{{\lambda }^{(j)}}(a_n^{(j)})X_0^{(1)}={ϵ}_{\mathbf{\lambda }}^{\ast }⟨0|\prod _{j=1}^N{P}_{{\lambda }^{(j)}}(a_n^{(j)})+\sum _{\mathbf{\mu }\stackrel{\ast }{>}\mathbf{\lambda }}{v}_{\mathbf{\lambda },\mathbf{\mu }}^{\ast }⟨0|\prod _{j=1}^N{P}_{{\mu }^{(j)}}(a_n^{(j)}).$$

$${ϵ}_{\mathbf{\lambda }}(\mathbf{u})={ϵ}_{\mathbf{\lambda }}^{\ast }(\mathbf{u})=\sum _{k=1}^N{u}_k{e}_{{\lambda }^{(k)}},e_{\lambda }:=1+(t-1)\sum _{i\ge 1}(q^{{\lambda }_i}-1)t^{-i}.$$

$${ϵ}_{\mathbf{\lambda }}(\mathbf{u})\ne 0(\forall \mathbf{\lambda });$$

$${ϵ}_{\mathbf{\lambda }}(\mathbf{u})\ne {ϵ}_{\mathbf{\mu }}(\mathbf{u})(\mathbf{\lambda }\ne \mathbf{\mu });$$

$$u_i\ne q^n{t}^m{u}_j(n,m\in \mathbb{Z},i,j=1,\dots ,N).$$

$$\bullet |P_{\mathbf{\lambda }}(\mathbf{u})⟩=\prod _{i=1}^N{P}_{{\lambda }^{(i)}}(a_{-n}^{(i)})|0⟩+\sum _{\mathbf{\mu }\stackrel{\ast }{<}\mathbf{\lambda }}{u}_{\mathbf{\lambda },\mathbf{\mu }}\prod _{i=1}^N{P}_{{\mu }^{(i)}}(a_{-n}^{(i)})|0⟩,u_{\mathbf{\lambda },\mathbf{\mu }}\in \mathbb{C};$$

$$\bullet X_0^{(1)}|P_{\mathbf{\lambda }}(\mathbf{u})⟩={ϵ}_{\mathbf{\lambda }}(\mathbf{u})|P_{\mathbf{\lambda }}(\mathbf{u})⟩,{ϵ}_{\mathbf{\lambda }}(\mathbf{u})\in \mathbb{C}.$$

$$\bullet ⟨P_{\mathbf{\lambda }}(\mathbf{u})|=⟨0|\prod _{i=1}^N{P}_{{\lambda }^{(i)}}(a_n^{(i)})+\sum _{\mathbf{\mu }\stackrel{\ast }{>}\mathbf{\lambda }}{u}_{\mathbf{\lambda },\mathbf{\mu }}^{\ast }⟨0|\prod _{i=1}^N{P}_{{\mu }^{(i)}}(a_n^{(i)}),u_{\mathbf{\lambda },\mathbf{\mu }}^{\ast }\in \mathbb{C};$$

$$\bullet ⟨P_{\mathbf{\lambda }}(\mathbf{u})|X_0^{(1)}={ϵ}_{\mathbf{\lambda }}^{\ast }(\mathbf{u})⟨P_{\mathbf{\lambda }}(\mathbf{u})|,{ϵ}_{\mathbf{\lambda }}^{\ast }(\mathbf{u})\in \mathbb{C}.$$

$${ϵ}_{\mathbf{\lambda }}(\mathbf{u})=(1-t^{-1})\sum _{i=1}^N{u}_i\sum _{k=1}^{n_i}{q}^{{\lambda }_k^{(i)}}{t}^{1-k}+\sum _{i=1}^N{u}_i{t}^{-n_i}.$$

Full text

1. Introduction

Let $\mathcal{U}$ be the DIM algebra [DI, Mi]. As for the definition of the DIM algebra, see Definition 2.1. The central object in the present paper is the intertwining operator $\mathcal{V}(x)$ associated with some structure of the $\mathcal{U}$ modules. From the point of view of the geometric engineering, or topological vertex construction for the partition functions for the quantum supersymmetric gauge theories, we regard the $\mathcal{V}(x)$ as the (multi-valent) topological vertex operator. The intertwining operator $\mathcal{V}(z)$ is defined through a certain set of commutation relations with the $\mathcal{U}$-generators. Let $X^{(i)}(z)$’s ($i=1,\ldots,N$) be the generating currents constructed from the standard Drinfeld current of $\mathcal{U}$ (Definition 2.6), acting on the $N$-fold tensor space \mathcal{F}_{\boldsymbol{u}}=\ooalign{\textstyle\bigotimes\crcr\hss\raisebox{7.96527pt}{\scriptsize\curvearrowright}\hss}\rule[-2.58334pt]{0.0pt}{10.33327pt}_{j=1}^{N}\mathcal{F}_{u_{j}} of the Fock spaces (For the definition of

$\textstyle\bigotimes$

$\curvearrowright$ , see Notation 5.7).

Definition 1.1** (topological vertex).**

Let $\mathcal{V}(x):\mathcal{F}_{\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{v}}$ be a linear map satisfying the commutation relations

[TABLE]

and the normalization condition $\bra{\boldsymbol{0}}\mathcal{V}(x)\ket{\boldsymbol{0}}=1$. Here $\ket{\boldsymbol{0}}$ (resp. $\bra{\boldsymbol{0}}$) is the vacuum (resp. dual vacuum) state.

We refer to this operator as the Mukadé operator. Mukadé is a Japanese word which means a centipede. The operator $\mathcal{V}(x)$ can be realized by connecting the trivalent intertwining operators as Figure 1 in Section 5, and that figure is the reason why we call $\mathcal{V}(x)$ the Mukadé operator. Let $\boldsymbol{\lambda}=(\lambda^{(1)},\ldots,\lambda^{(N)})$ be an $N$-tuple of partitions. Denote by $\ket{K_{\boldsymbol{\lambda}}}=\ket{K_{\boldsymbol{\lambda}}(\boldsymbol{u})}$ the canonical form of the generalized Macdonald function on $\mathcal{F}_{\boldsymbol{u}}$ (Definition 3.37). The main result of this paper is to prove the following factorization formula conjectured in [AFHKSY1].

Theorem 1.2**.**

We have

[TABLE]

Here, $\gamma=(t/q)^{1/2}$, $e_{N}(\boldsymbol{u})=u_{1}\cdots u_{N}$ and $N_{\lambda,\mu}$ is the Nekrasov factor. For the definition of $g_{\lambda}$, see Definition 3.39.

In our proof, the first important thing is to construct the explicit formula for the generalized Macdonald functions $\ket{K_{\boldsymbol{\lambda}}}$. We construct this explicit formula by using some screened vertex operators (Theorem 3.26). Next, the operator $\mathcal{V}(x)$ can also be realized by a certain fusion of the screened vertex operators. Matrix elements of the composition of the screened vertex operators satisfy a difference equation and correspond to a multiple series $p_{n}$ (Macdonald functions of A-type, Definition 3.21). The analyticity of matrix elements of screened vertex operators can be clarified by the analyticity of the Macdonald functions $p_{n}$. The final step of our proof is attributed to a transformation problem with respect to this multiple series $p_{n}$, and this is proved by the Euler transformation formula for Kajihara and Noumi’s multiple basic hypergeometric series.

We remark that the case $N=1$ of Theorem 1.2 was proved by using Kajihara-Noumi’s Euler transformation formula and the Pieri rules for the Macdonald polynomials [AFHKSY1]. As for the detail of the proof, unfortunately, it remains unpublished yet [AFHKSY2]. Since the Pieri rules are known only for the standard Macdonald case ($N=1$), it is totally unclear how we extend that method to the case $N>1$, which drove the present authors to have the formulation presented in this paper.

This paper is organized as follows. In Section 2, we briefly remind the readers of the basic facts about the DIM algebra $\mathcal{U}$ and its module structure on the Fock space $\mathcal{F}$. In Section 3, we introduce the generalized Macdonald functions, and they are explicitly constructed by the screened vertex operators (Therorem 3.26). We also discuss the analyticity of the matrix elements of the composition of screened vertex operators (Theorem 3.33). In Section 4, we give a proof of the main theorem (1.2). In Section 5, the existence of $\mathcal{V}(x)$ is proved through the explicit construction in terms of the topological vertex (Proposition 5.9). As its application, we can show the invariance under changing the preferred direction of toric diagrams, which is the natural consequence of the dualities in the string theory (Theorem 5.11).

In Appendix A, the Macdonald functions $p_{n}$ are constructed in terms of intertwining operators. In Appendix B, we give a brief review of the definition of ordinary Macdonald functions and their basic facts. In Appendix C, some useful formulas used in paper are explained. Some proofs in the main text are given in Appendix D. In Appendix E, we revisit the proof of a certain formula for the Kac determinant with respect to $\ket{X_{\boldsymbol{\lambda}}}$ (Definition 2.9) given in [O]. In particular, we clarify the choice of integral cycles of screening operators. In Appendix F, we give some examples. At last, we present list of notations in Appendix List of Notations.

Notations

A partition $\lambda=(\lambda_{1},\lambda_{2},\ldots)$ is a sequence of non-negative integers satisfying $\lambda_{1}\geq\lambda_{2}\geq\cdots$ and containing finitely many nonzero elements. They are identified if their elements without [math] are the same. $\ell(\lambda)$ denotes the length of $\lambda$, i.e., the number of elements in $\lambda$ without [math]. For an integer $i>\ell(\lambda)$, we occasionally write $\lambda_{i}$. Note that it is just [math]. The partition in which all elements are [math] is denoted by $\emptyset$. Write $|\lambda|=\sum_{i\geq 1}\lambda_{i}$ and $n(\lambda)=\sum_{i\geq 1}(i-1)\lambda_{i}$. Partitions are identified with the Young diagrams. For example, if $\lambda=(4,4,2,1)$, the Young diagram of $\lambda$ is

$s$

.

The transpose of $\lambda$ is denoted by $\lambda^{\prime}$. The coordinate of the box in the $i$-th row and the $j$-th column is denoted by $(i,j)$, say the coordinate of the box $s$ in the above Young diagram is $(2,3)$. For a coordinate $(i,j)\in\mathbb{Z}^{2}$, the arm length and the leg length are defined by $a_{\lambda}(i,j)=\lambda_{i}-j$ and $\ell_{\lambda}(i,j)=\lambda^{\prime}_{j}-i$, respectively. $A(\lambda)$ (resp. $R(\lambda)$) is the set of coordinates where we can add (resp. remove) a box. For example, if $\lambda=(3,3,1)$, $A(\lambda)=\{(1,4),(3,2),(4,1)\}$ and $R(\lambda)=\{(2,3),(3,1)\}$. In this paper, we often use $N$-tuples of partitions $\boldsymbol{\lambda}=(\lambda^{(1)},\ldots,\lambda^{(N)})$. For an $N$-tuple of partitions $\boldsymbol{\lambda}$, we put $|\boldsymbol{\lambda}|=\sum_{k=1}^{N}|\lambda^{(k)}|$.

Further, we use the following factorials ($q$-Pochhammer symbol) and the theta function:

[TABLE]

2. Preliminaries

2.1. Ding–Iohara–Miki algebra $\mathcal{U}$

Recall briefly some basic facts about the DIM algebra $\mathcal{U}$ [DI, Mi]. Let $q$ and $t$ be nonzero complex parameters satisfying $|q|<1$ and $q^{\frac{n}{2}}t^{\frac{m}{2}}\neq 1$ for all integers $n,m$.

Definition 2.1**.**

Let $\mathcal{U}=\mathcal{U}_{q,t}$ be the unital associative algebra over $\mathbb{C}$ generated by the Drinfeld currents

[TABLE]

and the invertible central element $c^{1/2}$, satisfying the following defining relations:

[TABLE]

where

[TABLE]

Fact 2.2**.**

The $\mathcal{U}$ admits the (topological) Hopf algebra structure with the Drindeld coprocuct $\Delta$:

[TABLE]

where $c_{(1)}^{\pm 1/2}=c^{\pm 1/2}\otimes 1$ and $c_{(2)}^{\pm 1/2}=1\otimes c^{\pm 1/2}$. We omit the antipode and the counit.

2.2. Fock and Fock tensor modules

A $\mathcal{U}$-module is called of level-$(n,m)$, if the two central elements act as $c=(t/q)^{n/2}$ and $(\psi^{+}_{0}/\psi^{-}_{0})^{1/2}=(q/t)^{m/2}$. Let $M\in\mathbb{Z}$. The Fock module $\mathcal{F}$ of level-$(1,M)$ is constructed as follows. Let $\{a_{n}|n\in\mathbb{Z}\}$ be the Heisenberg algebra with the relation

[TABLE]

and act on the Fock space $\mathcal{F}$ with the vacuum $\ket{0}$ in the usual way (that is, $a_{n}\ket{0}=0$ for $n\in\mathbb{Z}_{>0}$). $\mathcal{F}^{*}$ is the dual space generated by $\bra{0}$ satisfying $\bra{0}a_{n}=0$ ($n<0$). The basis of $\mathcal{F}$ (resp. $\mathcal{F}^{*}$) is given by $\ket{a_{\lambda}}=a_{-\lambda_{1}}a_{-\lambda_{2}}\cdots\ket{0}$ (resp. $\bra{a_{\lambda}}=\bra{0}\cdots a_{\lambda_{2}}a_{\lambda_{1}}$) with a partition $\lambda=(\lambda_{1},\lambda_{2},\dots)$. The bilinear form $\mathcal{F}^{*}\otimes\mathcal{F}\rightarrow\mathbb{C}$ is given by $\braket{0}{0}=1$.

Fact 2.3** ([FHHSY]).**

Let $u$ be a nonzero complex parameter. The following algebra homomorphism $\rho_{u}:\mathcal{U}\to\mathrm{End}(\mathcal{F})$ endows the $\mathcal{U}$-module structure on $\mathcal{F}$:

[TABLE]

where

[TABLE]

We call $u$ the spectral parameter, and denote this $\mathcal{U}$-module by $\mathcal{F}^{(1,M)}_{u}$.

We can also define the dual $\mathcal{U}$-module structure $\mathcal{F}^{(1,M)*}_{u}$ on $\mathcal{F}^{*}$ through the same $\rho_{u}$ by regarding its image as in $\mathrm{End}(\mathcal{F}^{*})$.

Let $N\in\mathbb{Z}_{\geq 1}$, and let $\boldsymbol{u}=(u_{1},\ldots,u_{N})$ be an $N$-tuple of nonzero complex parameters. Set

[TABLE]

The $\rho_{\boldsymbol{u}}^{(N,0)}$ gives the level-$(N,0)$ module structure on the $N$-fold tensor space of $\mathcal{F}$. We denote it by $\mathcal{F}^{(1,0)}_{u_{1}}\otimes\cdots\otimes\mathcal{F}^{(1,0)}_{u_{N}}$. In what follows, we will use the shorthand notation $\mathcal{F}_{\boldsymbol{u}}=\mathcal{F}^{(1,0)}_{u_{1}}\otimes\cdots\otimes\mathcal{F}^{(1,0)}_{u_{N}}$ for simplicity. We also denote $\mathcal{F}^{*}_{\boldsymbol{u}}=\mathcal{F}^{(1,0)*}_{u_{1}}\otimes\cdots\otimes\mathcal{F}^{(1,0)*}_{u_{N}}$. Let $\ket{\boldsymbol{0}}$ (resp. $\bra{\boldsymbol{0}}$) be the tensor product of the vacuum (resp. dual vacuum) states $\ket{0}^{\otimes N}$ (resp. $\bra{0}^{\otimes N}$) in $\mathcal{F}_{\boldsymbol{u}}$ (resp. $\mathcal{F}^{*}_{\boldsymbol{u}}$). Set

[TABLE]

for simplicity.

Definition 2.4**.**

For $k=1,2,\dots,N$, set

[TABLE]

Fact 2.5** ([FHSSY]).**

On $\mathcal{F}_{\boldsymbol{u}}$, we have

[TABLE]

Definition 2.6**.**

Set $X^{(1)}(z)=\rho_{\boldsymbol{u}}^{(N,0)}(x^{+}(z))$. Introduce the set of generators $X^{(k)}_{n}$ ($k=1,\ldots,N,n\in\mathbb{Z}$) by performing the fusion of several $X^{(1)}$’s as

[TABLE]

Fact 2.7** ([AFHKSY1]).**

We have

[TABLE]

Here, $:\ast:$ denotes the usual normal ordering in the Heisenberg algebra.

Definition 2.8**.**

We denote by $\mathsf{U}(N)$ (the completion in the sence of the adic topology, of) the algebra $\langle X^{(i)}_{n}|n\in\mathbb{Z},i=1,\ldots N\rangle$ in $\mathrm{End}(\mathcal{F}_{\boldsymbol{u}})$. Namely, $\mathsf{U}(N)$ is the completion of the algebra generated by the set of operators $\left\{X^{(i)}_{n}\right\}$.

This algebra $\mathsf{U}(N)$ can be regarded as the tensor product of the deformed $W_{N}$ algebra and some Heisenberg algebra [FHSSY].

Definition 2.9**.**

For an $N$-tuple of partitions $\boldsymbol{\lambda}=(\lambda^{(1)},\lambda^{(2)},\ldots,\lambda^{(N)})$, we define the vectors $\Ket{X_{\boldsymbol{\lambda}}}=\Ket{X_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}_{\boldsymbol{u}}$ and $\Bra{X_{\boldsymbol{\lambda}}}=\Bra{X_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}_{\boldsymbol{u}}^{*}$ by

[TABLE]

In what follows, we omit the spectral parameter $\boldsymbol{u}$, as long as there is no confusion.

Fact 2.10** ([O]).**

The set $(\Ket{X_{\boldsymbol{\lambda}}})$ (resp. $(\Bra{X_{\boldsymbol{\lambda}}})$) forms a PBW-type basis of $\mathcal{F}_{\boldsymbol{u}}$ (resp. $\mathcal{F}^{*}_{\boldsymbol{u}}$), if $u_{i}\neq q^{s}t^{-r}u_{j}$ and $u_{i}\neq 0$ for all $i,j$ and $r,s\in\mathbb{Z}$.

In Appendix E, we reproduce the proof (presented in [O]) of Fact 2.10, with detailed treatments concerning the integration cycles.

3. Generalized Macdonald Functions

In [AFHKSY1], a sort of generalization of Macdonald functions are introduced on the level $(N,0)$ module $\mathcal{F}_{\boldsymbol{u}}$. In this section, we give a certain explicit formula for the generalized Macdonald functions.

3.1. Definition of generalized Macdonald functions

The generalized Macdonald functions are defined as eigenfunctions of $X^{(1)}_{0}$. In the theory of ordinary Macdonald functions, the fundamental existence theorem follows from the triangulation of Macdonald’s difference operator. In that triangulation, the order of a basis is given by the dominance partial ordering. In the $N=1$ case, $X^{(1)}_{0}$ corresponds to Macdonald’s difference operator [AMOS]. In the general $N$ case, the operator $X^{(1)}_{0}$ can be triangulated by the following partial ordering.

Definition 3.1**.**

The partial ordering $\mathop{\overset{*}{>}}$ on the set of $N$-tuples of partitions is defined by

[TABLE]

Remark 3.2**.**

This partial ordering is defined so that as we move boxes in a right Young diagram to a left one, it gets smaller. On the other hand, as boxes are moved from left to right, it gets larger. Note that if $N$-tuples of partitions $\boldsymbol{\lambda}$ and $\boldsymbol{\mu}$ have the same number of boxes in each Young diagram, then neither $\boldsymbol{\lambda}\mathop{\overset{*}{>}}\boldsymbol{\mu}$ nor $\boldsymbol{\lambda}\mathop{\overset{*}{<}}\boldsymbol{\mu}$.

Example 3.3**.**

If $N=3$ and the number of boxes is $2$, then

[TABLE]

Here $\boldsymbol{\lambda}\rightarrow\boldsymbol{\mu}$ stands for $\boldsymbol{\lambda}\mathop{\overset{*}{>}}\boldsymbol{\mu}$.

Regarding the products of ordinary Macdonald functions $\prod_{i=1}^{N}P_{\lambda^{(i)}}(a^{(i)}_{-n})\ket{\boldsymbol{0}}$ as a basis of $\mathcal{F}_{\boldsymbol{u}}$, we can triangulate the operator $X^{(1)}_{0}$. Here, $P_{\lambda}(a_{-n})$ is the abbreviation for $P_{\lambda}(a_{-1},a_{-2}\ldots)$, which is the Macdonald function obtained by replacing the power sum symmetric functions with generators of the Heisenberg algebra. For the definition and notations of the ordinary Macdonald functions see Appendix B. In Appendix B, we also explain some well-known facts for Macdonald functions.

Fact 3.4** ([AFO]).**

We have

[TABLE]

[TABLE]

Here, $v_{\boldsymbol{\lambda},\boldsymbol{\mu}},v^{*}_{\boldsymbol{\lambda},\boldsymbol{\mu}}\in\mathbb{C}$. The eigenvalues $\epsilon_{\boldsymbol{\lambda}}$ and $\epsilon^{*}_{\boldsymbol{\lambda}}$ are in the following form:

[TABLE]

Remark 3.5**.**

In this paper, we assume that the complex parameters $q$, $t$ and the spectra parameters $\boldsymbol{u}$ are generic in the following sense:

[TABLE]

Under this assumption, the eigenfunctions of $X^{(1)}_{0}$ can be characterized as follows.

Fact 3.6** (Existence and Uniqueness [AFO]).**

For an $N$-tuple of partitions $\boldsymbol{\lambda}$, there exists a unique vector $\Ket{P_{\boldsymbol{\lambda}}}=\Ket{P_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}_{\boldsymbol{u}}$ such that

[TABLE]

Similarly, there exists a unique vector $\Bra{P_{\boldsymbol{\lambda}}}=\Bra{P_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}_{\boldsymbol{u}}^{*}$ such that

[TABLE]

Again, we omit $\boldsymbol{u}$ unless mentioned otherwise.

Note that for integers $n_{i}$ ($i=1,\ldots N$) with $n_{i}\geq\ell(\lambda^{(i)})$, the eigenvalues can be rewritten as

[TABLE]

By this Fact, it is easily seen that the generalized Macdonald functions $\ket{P_{\boldsymbol{\lambda}}}$ form a basis of $\mathcal{F}_{\boldsymbol{u}}$. It is also known that the operator $X^{(1)}_{0}$ is a member of a certain family of infinitely many commutative operators, and $\ket{P_{\boldsymbol{\lambda}}}$ can be regarded as simultaneous eigenfunctions of them [FHHSY]. Moreover, $\ket{P_{\boldsymbol{\lambda}}}$ correspond to the torus fixed points in the instanton moduli space, which form a significant basis in the AGT correspondence. Note that this is a $q$-analogue of the AFLT basis [AFLT].

Fact 3.7** ([AFHKSY1]).**

$\ket{P_{\boldsymbol{\lambda}}}$* and $\bra{P_{\boldsymbol{\lambda}}}$ are orthogonal, and their inner products take the form*

[TABLE]

Here, we put

[TABLE]

Definition 3.8**.**

We introduce $\ket{Q_{\boldsymbol{\lambda}}}$ such that $\braket{P_{\boldsymbol{\lambda}}}{Q_{\boldsymbol{\mu}}}=\delta_{\boldsymbol{\lambda},\boldsymbol{\mu}}$, i.e., $\ket{Q_{\boldsymbol{\lambda}}}:=\prod_{i=1}^{N}\frac{c_{\lambda^{(i)}}}{c^{\prime}_{\lambda^{(i)}}}\ket{P_{\boldsymbol{\lambda}}}$.

3.2. Screening currents and vertex operators

This subsection and the next are devoted to the construction of an explicit formula for the generalized Macdonald functions $\ket{Q_{\boldsymbol{\lambda}}}$. Our explicit formula is made up of two ingredients. The one is a certain screened vertex operators on $\mathcal{F}_{\boldsymbol{u}}$, and the other is an explicit solution to the eigenfunction problem associated with the Macdonald $q$-difference operator of A-type. In this subsection, we introduce the first ingredient, screening currents and some vertex operators. They play an important role also in a realization of our main vertex operator $\mathcal{V}(x)$.

Notation 3.9**.**

For an $N$-tuple of parameters $\boldsymbol{u}=(u_{1},\ldots,u_{N})$, we write

[TABLE]

Definition 3.10**.**

Define the screening currents $S^{(i)}(y):\mathcal{F}_{t^{\alpha_{i}}\cdot\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{u}}$ by the following,

[TABLE]

with

[TABLE]

Remark 3.11**.**

In this paper, we do not attach zero modes to the screening currents. When discussing commutativity with the algebra $\mathsf{U}(N)$, we use a function satisfying some difference equation as in the following proposition.

Proposition 3.12**.**

Let $g_{i}(w)$ ($i=1,\ldots,N-1$) be a function which satisfies the difference equation $g_{i}(qw)=\frac{u_{i+1}}{tu_{i}}g_{i}(w)$. Then, the commutation relation between $S^{(i)}(w)g_{i}(w)$ and the algebra $\mathsf{U}(N)$ is a total $q$-difference:

[TABLE]

Here, $T_{q,w}$ is the difference operator such that $T_{q,w}:w\mapsto qw$.

The proof is given in Appendix D.1.

Remark 3.13**.**

The function $g_{i}(w)$ can be realized as $g_{i}(w)=\frac{\theta_{q}(t^{2}u_{i}w/u_{i+1})}{\theta_{q}(tw)}$, and integrals of the screening currents commute with $\mathsf{U}(N)$ if the spectral parameters $\boldsymbol{u}$ are degenerated (See Proposition E.4).

Definition 3.14**.**

Define the screened vertex operators $\Phi^{(k)}(x):\mathcal{F}_{t^{-\delta_{k+1}}\cdot\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{u}}$ ($k=0,1,\dots,N-1$) by

[TABLE]

and as the composition of operators,

[TABLE]

with an integral kernel

[TABLE]

Here, the contour of the integration $C$ is chosen so that $|t^{-1}|<|y_{j}/y_{i}|<|q|$ for $1\leq i<j\leq k$, and $|q/t|<|y_{i}/x|<1$ for $i\geq 1$.

Remark 3.15**.**

The integration contour is well-defined if $|t^{-1}|<|q^{N-2}|$. Hence, in what follows, we assume this condition in this paper without Appendix E.

These screened vertex operators can be obtained by combining a certain specialized intertwiner of the DIM algebra and replacing some Jackson integrals with contour ones. In Appendix A, we explain details of this construction and correspondence between Jackson integrals and contour ones.

Remark 3.16**.**

The screened vertex operator $\Phi^{(k)}(x)$ is normalized such that $\oint\frac{dx}{2\pi\sqrt{-1}x}\Phi^{(k)}(x)\ket{\boldsymbol{0}}=\ket{\boldsymbol{0}}$. Indeed, $\Phi^{(k)}(x)$ can be expanded as

[TABLE]

This expansion follows from the operator products (C.7)-(C.15) and Ramanujan’s ${}_{1}\psi_{1}$ summation formula (Fact 3.17). By (3.24), it can be shown that the coefficient of $y_{0}^{r_{1}}$ in the expansion of $\Phi^{(0)}(y_{0})\ket{\boldsymbol{0}}$ is given by a finite sum and the constant term with respect to $y_{0}$ is $1$.

Fact 3.17** ((5.2.1) in [GR]).**

[TABLE]

Remark 3.18**.**

Note that $g(x,y_{1},\ldots,y_{k})$ satisfies

[TABLE]

*This corresponds to the difference equation which the function $g_{i}(z)$ satisfies in Proposition 3.12. Moreover, the screening currents in $\Phi^{(k)}(x)$ commute with the algebra $\mathsf{U}(N)$ by taking the integrals. For more details, see the proof of the following lemma. *

The screened vertex operators $\Phi^{(k)}(x)$ satisfy the following relation with the algebra $\mathsf{U}(N)$.

Lemma 3.19**.**

For $k=0,1,\ldots,N-1$ and $r=1,\ldots,N$,

[TABLE]

where $\delta(z)=\sum_{n\in\mathbb{Z}}z^{n}$ is the formal delta function, and we defined

[TABLE]

Note that in particular $Y^{(1)}(z)=1$.

For the proof, see Appendix D.2

3.3. Explicit formula for generalized Macdonald functions

In this subsection, we explain an explicit solution to the eigenfunction equation of the Macdonald $q$-difference operator of A-type. By using its solution, we give an explicit formula for generalized Macdonald functions.

First, we introduce a modified Macdonald $q$-difference operator.

Definition 3.20**.**

Define the operators on $\mathbb{C}[[x_{2}/x_{1},x_{3}/x_{2}\ldots,x_{n}/x_{n-1}]]$ by

[TABLE]

where $\boldsymbol{s}=(s_{1},\dots,s_{n})$ are generic parameters, and $T_{q,x_{k}}$ is the difference operator defined by

[TABLE]

Though it is essentially equivalent, we also introduce the operator

[TABLE]

An explicit formula for the eigenfunction of $\widetilde{D^{1}_{n}}(\boldsymbol{s};q,t)$ was conjectured in [S]. Afterward, it was proved, and a solution to the bispectral problem was also given in [NS].

Definition 3.21**.**

Let $\boldsymbol{x}=(x_{i})_{1\leq i\leq n}$ be inderminates and $\boldsymbol{s}=(s_{i})_{1\leq i\leq n}$ be generic parameters. Define $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ and $f_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)\in\mathbb{C}[[x_{2}/x_{1},x_{3}/x_{2},\ldots,x_{n}/x_{n-1}]]$ by

[TABLE]

For convenience, we put $f_{0}=p_{0}=1$. Here $M_{n}$ is the set of all $n\times n$ upper triangular matrices with nonnegative integers, in which diagonal elements are [math]. $c_{n}(\theta;\boldsymbol{s}|q,t)$ are coefficients defined by the following recurrence relations

[TABLE]

with

[TABLE]

From the recurrence relations, we can compute the explicit form of $c_{n}$ as

[TABLE]

Fact 3.22** ([NS, S, BFS]).**

The function $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ is an unique formal solution to the eigenfunction equation

[TABLE]

up to scalar multiples. Equivalently, $f_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ is an unique function (up to scalar multiples) such that

[TABLE]

Remark 3.23**.**

The function $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ and $f_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ are dual to each other in the sense of Lemma C.2. Furthermore, it is known that the function $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ can also be regarded as a holomorphic function, and its analyticity is clarified in [NS]. We explain its analyticity in Section 3.4 (Fact 3.32).

In the beginning of this subsection, we stated to give an explicit formula for generalized Macdonald functions. For that purpose, let us introduce the following notations with respect to the screened vertex operators.

Notation 3.24**.**

For an $N$-tuple of nonnegative integers $\boldsymbol{n}=(n_{1},\ldots,n_{N})$, we write

[TABLE]

Definition 3.25**.**

Let $\boldsymbol{n}=(n_{1},\ldots,n_{N})$ be an $N$-tuple of nonnegative integers. Define the operator $V^{(\boldsymbol{n})}(x_{1},\ldots,x_{|\boldsymbol{n}|}):\mathcal{F}_{t^{-\boldsymbol{n}}\cdot\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{u}}$ by

[TABLE]

We occasionally write V^{(\boldsymbol{n})}(x_{1},\ldots,x_{|\boldsymbol{n}|})=V^{(\boldsymbol{n})}\Big{(}\begin{array}[]{c}\boldsymbol{u}\\ t^{-\boldsymbol{n}}\cdot\boldsymbol{u}\end{array}\,;x_{1},\ldots,x_{|\boldsymbol{n}|}\Big{)} in order that the spectral parameters of representations are clear.

Now, we obtain an explicit formula for the generalized Macdonald functions $\ket{Q_{\boldsymbol{\lambda}}}$ in terms of the screened vertex operators. The action of the screened vertex operators is controlled by the function $f_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$. Further, we can also obtain an expression of $p_{n}$ by using $\bra{P_{\boldsymbol{\lambda}}}$ and the operator $V^{(\boldsymbol{n})}$. These expressions of $\ket{Q_{\boldsymbol{\lambda}}}$ and $p_{n}$ are symmetric to each other in some sense.

Theorem 3.26**.**

Let $\boldsymbol{n}=(n_{1},\ldots,n_{N})$ be an $N$-tuple of integers satisfying that $n_{i}\geq\ell(\lambda^{(i)})$ for all $i$. Then under the identification $s_{[i,k]_{\boldsymbol{n}}}=q^{\lambda^{(i)}_{k}}t^{1-k}u_{i}$ ($1\leq k<n_{i},\,i=1,\ldots,N$), the followings hold:

[TABLE]

Here, we set

[TABLE]

$[\cdots]_{\boldsymbol{x},1}$* means to take the constant term in $\cdots$ with respect to $\boldsymbol{x}$, and $\mathcal{R}^{\boldsymbol{n}}_{\boldsymbol{\lambda}}(\boldsymbol{u})\in\mathbb{C}(u_{1},\ldots,u_{N})$ is some coefficient.*

Proof.

By Lemma 3.19 and the operator product (C.21) we can get the relation

[TABLE]

By taking the constant term of $z$, we have

[TABLE]

Noting the integrals of the total difference vanish, integrating by parts gives the following equality.

[TABLE]

Here, in the third equality, we made use of Fact 3.22. This equality identify the LHS of (3.45) with the generalized Macdonald functions up to the normalization.

From (3.54), we can also have

[TABLE]

Therefore, $\boldsymbol{x}^{-\boldsymbol{\lambda}}\bra{P_{\boldsymbol{\lambda}}}V^{(\boldsymbol{n})}\ket{\boldsymbol{0}}$ is an eigenfunction of $D^{1}_{|\boldsymbol{n}|}$. This implies (3.48). Since the constant terms in $p_{n}$ and $f_{n}$ are $c_{1}(-;s_{1}|q,t)=1$, it is clear that (3.45) and (3.48) have the same proportionality constant $\mathcal{R}^{\boldsymbol{n}}_{\boldsymbol{\lambda}}(\boldsymbol{u})$. ∎

The following proposition gives the explicit form of $\mathcal{R}^{\boldsymbol{n}}_{\boldsymbol{\lambda}}(\boldsymbol{u})$.

Proposition 3.27**.**

The coefficient $\mathcal{R}^{\boldsymbol{n}}_{\boldsymbol{\lambda}}(\boldsymbol{u})$ is of the form,

[TABLE]

The proof of Proposition 3.27 is in subsection D.3.

3.4. Analyticity of matrix elements of $V^{(\boldsymbol{n})}$

In this subsection, we describe the fact that the Macdonald functions $p_{n}$ can be treated as holomorphic functions. Moreover, we show the matrix elements of $V^{(\boldsymbol{n})}$ can meromorphically extend to the whole $\mathbb{C}^{|\boldsymbol{n}|-1}$, and thus we can deal with the variables $\boldsymbol{x}$ as those in that space, not as formal variables.

Definition 3.28**.**

Define the projection for the canonical coordinates $\pi_{n}:(\mathbb{C}^{*})^{n}\to\mathbb{C}^{n-1}$ such that for each $a=(a_{1},\dots,a_{n})\in(\mathbb{C}^{*})^{n}$,

[TABLE]

Notation 3.29**.**

In what follows in this subsection, we use the following notations.

[TABLE]

Definition 3.30**.**

Define the open subset $D_{w}\subset\mathbb{C}^{n-1}$ by

[TABLE]

so that

[TABLE]

Definition 3.31**.**

Define the subsets $U^{n}_{z}(r),B^{n}_{z}(r)\subset\mathbb{C}^{n-1}$ by

[TABLE]

so that

[TABLE]

In order to see the matrix element above can be analytically continued except for its singularities, we make use of the following fact, proved in [NS].

Fact 3.32** ([NS]).**

Let $\tau$ be a generic complex parameter. For $n=2,3,\dots$, we regard $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,\tau)$ as a formal power series in $z=(z_{1},\dots,z_{n-1})$ with coefficients in $\mathcal{O}(D^{n}_{w})$, the ring of holomorphic functions on $D^{n}_{w}$:

[TABLE]

Set $r_{0}=|q/\tau|^{\frac{n-2}{n-1}}$ if $|q/\tau|\leq 1$, and $r_{0}=|\tau/q|$ if $|q/\tau|\geq 1$. Then for any compact subset $K\subset D_{w}$ and for any $r<r_{0}$, this series 3.64 is absolutely convergent, uniformly on $B^{n}_{z}(r)\times K$. Hence $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,\tau)$ defines a holomorphic function on $U^{n}_{z}(r_{0})\times D^{n}_{w}$.

Then, we have the following theorem.

Theorem 3.33**.**

The operator $V^{(\boldsymbol{n})}(x_{1},\ldots,x_{|\boldsymbol{n}|})$ is well-defined on $\pi^{-1}_{|\boldsymbol{n}|}(U^{|\boldsymbol{n}|}_{z}(r_{0}))$ with $r_{0}=|t^{-1}|$, in the sense that its matrix elements are the holomorphic functions there.

We prepare the following lemma.

Lemma 3.34**.**

Let $\boldsymbol{u}=t^{-\boldsymbol{n}}\cdot\boldsymbol{v}$, $\boldsymbol{w}=t^{-\boldsymbol{m}}\cdot\boldsymbol{u}$ and $n_{i}\geq\ell(\lambda^{(i)})$ ($\forall i$). Then

[TABLE]

under the identification

[TABLE]

This lemma can be derived as a corollary of Theorem 3.26 by noting the normalization of the screened vertex operators (Remark 3.16).

*Proof of Theorem 3.33. * First, by Fact 3.32, the left hand side of (3.65) is a holomorphic function on $U^{|\boldsymbol{n}|+|\boldsymbol{m}|}_{\tilde{z}}(r_{0})$. Then, through the pull-back $\pi^{*}_{|\boldsymbol{n}|+|\boldsymbol{m}|}$, we regard it as the holomorphic function in $\pi^{-1}_{|\boldsymbol{n}|+|\boldsymbol{m}|}(U^{|\boldsymbol{n}|+|\boldsymbol{m}|}_{\tilde{z}}(r_{0}))$. Thus, once we fix $y\in(\mathbb{C}^{*})^{|\boldsymbol{m}|}$ such that $|y_{j}/y_{i}|<r^{j-i}_{0}\,\,(1\leq i<j\leq|\boldsymbol{m}|)$, it becomes a holomorphic function of $x\in\widetilde{\pi}^{-1}_{|\boldsymbol{n}|,y_{1}}(U^{|\boldsymbol{n}|}_{z}(r_{0}))$, where

[TABLE]

Second, by multiplying both sides of (3.65) by $\boldsymbol{y}^{-\mu}f_{|\boldsymbol{m}|}(\boldsymbol{y};\boldsymbol{s}|q,q/t)$ and taking the constant term in $\boldsymbol{y}$, the left hand side of (3.65) becomes \boldsymbol{x}^{-\boldsymbol{\lambda}}\bra{P_{\boldsymbol{\lambda}}}V^{(\boldsymbol{n})}\Big{(}\begin{array}[]{c}\boldsymbol{v}\\ t^{-\boldsymbol{n}}\cdot\boldsymbol{v}\end{array}\,;x_{1},\ldots,x_{|\boldsymbol{n}|}\Big{)}\ket{Q_{\boldsymbol{\mu}}} with the help of Theorem 3.26. Note that, in this case, taking the constant term means the usual contour integral in $y$, and the contour is chosen to separate $x$ and $y$. This procedure does not affect the holomorphicity in $x$. At this point, x^{-\boldsymbol{\lambda}}\bra{P_{\boldsymbol{\lambda}}}V^{(\boldsymbol{n})}\Big{(}\begin{array}[]{c}\boldsymbol{v}\\ t^{-\boldsymbol{n}}\cdot\boldsymbol{v}\end{array}\,;x_{1},\ldots,x_{|\boldsymbol{n}|}\Big{)}\ket{Q_{\boldsymbol{\mu}}} can holomorphically extend to $\pi^{-1}_{|\boldsymbol{n}|}(U^{|\boldsymbol{n}|}_{z}(r_{0}))$, because we can take arbitrary small $y_{1}$. As a result, Theorem 3.33 is proved. ∎

3.5. Integral forms

In this subsection, we define the renormalization of generalized Macdonald functions, which correspond to the integral form of ordinary Macdonald functions up to monomials. In this basis, the main theorem for the matrix elements of a main vertex operator can be written in smarter way than in the basis formed by $\ket{P_{\boldsymbol{\lambda}}}$, defined above.

Definition 3.35**.**

Define

[TABLE]

$N_{\lambda\mu}(u)$* is called the Nekrasov factor.*

This factor appears in the instanton partition functions in five dimensional $\mathcal{N}=1$ supersymmetric gauge theories.

Definition 3.36**.**

Set

[TABLE]

where $c_{\lambda}$ is defined in (3.14) and $n(\lambda)=\sum_{j\geq 1}(j-1)\lambda_{j}$.

Definition 3.37**.**

Define $\ket{K_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}_{\boldsymbol{u}}$ and $\bra{K_{\boldsymbol{\lambda}}(\boldsymbol{u})}\in\mathcal{F}^{*}_{\boldsymbol{u}}$ by

[TABLE]

This normalization arises from the following conjecture with respect to the expansion coefficient in the PBW-type basis $\ket{X_{\boldsymbol{\lambda}}}$ and $\bra{X_{\boldsymbol{\lambda}}}$.

Conjecture 3.38**.**

[TABLE]

In Appendix F.1, we give examples of the transition matrices $\alpha^{(\pm)}_{\boldsymbol{\lambda}\boldsymbol{\mu}}$.

Definition 3.39** ((Taki’s) flaming factors).**

Define

[TABLE]

Proposition 3.40**.**

[TABLE]

where we put $e_{N}(\boldsymbol{u}):=\prod_{i=1}^{N}u_{i}$.

Proof.

This follows from

[TABLE]

∎

4. Proof of Main Theorem

This section is organized as follows. In Section 4.1, before going into the details of the proof of the main theorem, we recall what is our main claim in this paper. In Section 4.2, we derive a transformation formula for the multiple basic hypergeometric series, which is one of the key ingredients in the proof in Section 4.3 of the main theorem.

4.1. Statement of main theorem

First, we define the vertex operator $\mathcal{V}(x)$ by relations with the algebra $\mathsf{U}(N)$. We call $\mathcal{V}(x)$ the Mukadé operator as explained in Introduction.

Definition 4.1**.**

Define the linear operator \mathcal{V}(x)=\mathcal{V}\Big{(}\begin{array}[]{c}\boldsymbol{v}\\ \boldsymbol{u}\end{array}\,;x\Big{)}:\mathcal{F}_{\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{v}} by the following relations:

[TABLE]

and $\bra{\boldsymbol{0}}\mathcal{V}(x)\ket{\boldsymbol{0}}=1$.

Then we have the following proposition.

Proposition 4.2**.**

The $\mathcal{V}(x)$ exists uniquely.

Proof.

If the operator $\mathcal{V}(x)$ exists, the uniqueness is clear by definition of $\mathcal{V}(w)$ and the fact that the vectors $\ket{X_{\boldsymbol{\lambda}}}$ form a basis (Fact 2.10). (See also Appendix F.2) The existence of $\mathcal{V}(w)$ will be shown late in the Proposition 5.9. ∎

Remark 4.3**.**

The realization of $\mathcal{V}(x)$ given in Section 5 shows that $\mathcal{V}(x)$ satisfies the relation with the algebra $\mathcal{U}$ introduced in [AFHKSY1, Definition 3.12] after some simple renormalization. However, if we adopt its relation as definition of the vertex operator, the uniqueness is nontrivial.

In Appendix F.2, we demonstrate the calculation of the matrix elements of $\mathcal{V}(x)$ by using the defining relation (4.1). The matrix elements with respect to generalized Macdonald functions are factorized and reproduce the Nekrasov factor:

Theorem 4.4**.**

[TABLE]

As a direct result of this theorem, we obtain an algebraic description of the 5D (K-theoretic) AGT correspondence [AGT, AY1, AY2]. Theorem 4.4 makes us possible to calculate the multi-point functions. In particular, the two-point function, which can be regarded as $q$-analogue of the four-point conformal block in 2D CFT 111 As $X^{(1)}_{0}\ket{\boldsymbol{0}}=(u_{1}+\cdots+u_{N})\ket{\boldsymbol{0}}$, the vectors $\bra{\boldsymbol{0}}$ and $\ket{\boldsymbol{0}}$ are not the ordinary vacuum states for the algebra $\mathsf{U}(N)$ but highest weight vectors of highest weight $\boldsymbol{u}$. Hence, the function $\bra{\boldsymbol{0}}\mathcal{V}(x_{1})\cdots\mathcal{V}(x_{n})\ket{\boldsymbol{0}}$ can be regarded as $q$-analogue of the $(n+2)$-point conformal block., corresponds to the Nekrasov partition function of the 5-dimensional $\mathcal{N}=1$ $U(N)$ gauge theory with $2N$ fundamental matters [AK1, AK2]. (See also Remark 3.14, 3.15 in [AFHKSY1] for more detail.):

[TABLE]

In what follows in this section, we prove Theorem 4.4.

4.2. Transformation formula

First of all, we show a certain transformation formula (Proposition 4.8) for multiple series with respect to Macdonald functions $p_{n}(x;s|q,t)$, which plays an essential role in our proof of the main theorem. That formula is based on the Euler transformation of Kajihara and Noumi’s multiple basic hypergeometric series, which is defined as follows.

Definition 4.5**.**

Define

[TABLE]

with

[TABLE]

Let us also mention that the elliptic analogue is studied in [KN]. Kajihara and Noumi gave the following $q$-Euler transformation formula.

Fact 4.6** ([K, KN]).**

[TABLE]

We prepare the following notation.

Definition 4.7**.**

For nonnegative integers $n$, $m$ and $\mu=(\mu_{i})_{1\leq i\leq m}\in\mathbb{Z}^{m}$, we introduce

[TABLE]

By using Fact 4.6, we can show the following formula which transforms the Macdonald function $p_{n+m}(\boldsymbol{x};\boldsymbol{s}|q,t)$ to another series that contains $p_{m}$ as inner summation.

Proposition 4.8**.**

Let $s_{i}$ ($i=1,\ldots,n+m$) be generic complex parameters and $|t|>|q|^{-(n-2)}$. We put $x_{i}=xt^{n-i}$ for $i=1,\ldots,n$ and $x_{n+k}=y_{k}$ for $k=1,\ldots,m$. Then

[TABLE]

Remark 4.9**.**

Throughout this subsection, $s_{i}$ are treated as generic parameters with $s_{i}\neq 0$ and $s_{i}\neq q^{r_{1}}t^{r_{2}}s_{j}$ ($r_{1}\in\mathbb{Z}$, $r_{2}=0,\pm 1$, $\forall i,j$). The above transformation formula holds even if $s_{i}$ are not specialized as (3.66) and (3.67).

In particular, if $m=0$, this proposition is nothing but the following specialization formula.

Fact 4.10** ([NS]).**

Let $|t|>|q|^{-(n-2)}$. Then

[TABLE]

For the proof of Proposition 4.8, we prepare two lemmas. They are tools for attributing our transformation formula (4.8) to the one of the multiple basic hypergeometric series. The proofs of these lemmas are by direct calculation of factorials (See Appendix D.4 and D.5).

Lemma 4.11**.**

Let $\sigma=(\sigma_{k})_{1\leq k\leq m-1}\in\mathbb{Z}^{m-1}$ and $\theta=(\theta_{i})_{1\leq i\leq n+m-1}\in\mathbb{Z}_{\geq 0}^{n+m-1}$. Then under the transformation $\rho_{k}=\sigma_{k}-\theta_{n+k}$, we have

[TABLE]

Here, $h$ is a parameter which goes to $1$ in the limit, satisfying $q^{r_{1}}t^{r_{2}}h\neq 1$ ($\forall r_{1},r_{2}\in\mathbb{Z}$). $\widetilde{\mathsf{N}}^{n,m}_{\mu}$ is defined by

[TABLE]

Remark 4.12**.**

If $\rho_{k}<0$ and $\rho_{k}+\theta_{n+k}>0$ for some $k$, then $\widetilde{\mathsf{N}}^{n,m-1}_{\rho}$ is [math] and $\phi^{n+m-1,m}_{\theta}$ diverges in the limit $h\rightarrow 1$. However, (4.11) converges as a result. In order to avoid the problem with respect to this divergence, we inserted the parameter $h$.

Lemma 4.13**.**

Let $\rho=(\rho_{k})_{1\leq k\leq m-1}\in\mathbb{Z}^{m-1}$ and $\nu=(\nu_{k})_{1\leq k\leq m}\in\mathbb{Z}_{\geq 0}^{m}$. Then

[TABLE]

under the transformation

[TABLE]

Now, we prove Proposition 4.8 by using these lemmas and the $q$-Euler transformation formula.

Proof of Proposition 4.8 The proof is done by induction on $m$. If $m=0$, (4.8) follows from Fact 4.10. By assumption that it holds for $m-1$, it can be shown that the left hand side of (4.8) is

[TABLE]

By the contribution of the factor $\prod_{k=1}^{m-1}1/(q;q)_{\sigma_{k}}$ in $\mathsf{N}^{n,m-1}_{\sigma}$, we can extend the range which $\theta$ and $\sigma$ ran over to

[TABLE]

Under this range, by applying Lemma 4.11, we can rewrite (4.2) as

[TABLE]

By using the Euler transformation formula for the multiple hypergeometric series (Fact 4.6), (4.2) can be written as

[TABLE]

Finally, Lemma 4.13 shows that (4.2) is equal to

[TABLE]

Noting that the summation is restricted to $\mu\in\mathbb{Z}_{\geq 0}^{m}$ by the contribution of the factor in $\mathsf{N}^{n,m}_{\mu}$, we can show that (4.71) coincides with the right hand side of (4.8). ∎

4.3. Proof of Theorem 4.4

We are now in a position to show Theorem 4.4. This subsection is devoted to its proof.

Since the defining relation of $\mathcal{V}(w)$ and the expansion coefficients of $\ket{K_{\boldsymbol{\lambda}}}$ in the basis of $\ket{X_{\boldsymbol{\lambda}}}$ are written by rational functions of $q$, $t$, $u_{i}$ and $v_{i}$, the matrix elements $\bra{K_{\boldsymbol{\lambda}}}\mathcal{V}(z)\ket{K_{\boldsymbol{\mu}}}$ determine the unique rational function of them by Proposition 4.2. Therefore, it suffices to prove (4.2) in the case $v_{i}=t^{n_{i}}u_{i}$ ($i=1,\ldots,N$) for all sufficiently large $n_{i}\in\mathbb{Z}$, i.e., $n_{i}\geq\ell(\lambda^{(i)})$ by analytic continuation.

Step1: Realization of $\mathcal{V}(x)$

Firstly, we give a realization of $\mathcal{V}(x)$ in the case $\boldsymbol{v}=t^{\boldsymbol{n}}\cdot\boldsymbol{u}$ as follows.

Definition 4.14**.**

Let $|t|>|q|^{-(n-2)}$. Define \widetilde{V}^{(\boldsymbol{n})}(x)=\widetilde{V}^{(\boldsymbol{n})}\Big{(}\begin{array}[]{c}\boldsymbol{v}\\ \boldsymbol{u}\end{array}\,;x\Big{)}:\mathcal{F}_{\boldsymbol{u}}\rightarrow\mathcal{F}_{\boldsymbol{v}} with $\boldsymbol{u}=t^{-\boldsymbol{n}}\cdot\boldsymbol{v}$ by

[TABLE]

where

[TABLE]

Remark 4.15**.**

*Note that $\sum_{i=1}^{N}\gamma^{(i-1)n}a^{(i)}_{n}$ is the boson corresponding to the Cartan part $\Delta^{(N)}(\psi^{+}(z))$. *

Proposition 4.16**.**

$\widetilde{V}^{(\boldsymbol{n})}(x)$* is well-defined on $\mathbb{C}^{*}$, i.e., its arbitrary matrix elements are holomorphic functions there.*

Before the proof, we prepare the following fact. This fact tells us the duality of the Macdonald functions under exchanging $t$ and $q/t$.

Fact 4.17** ([NS]).**

The formal series $p_{n}(\boldsymbol{x};\boldsymbol{s}|q,t)$ with the leading coefficient 1 satisfies the symmetry relation

[TABLE]

*Proof of Proposition 4.16. * We introduce the following product of currents,

[TABLE]

We have the following equality.

[TABLE]

Here, for brevity of notation, we set $x_{|\boldsymbol{n}|+i}=y_{i}$, and used Fact 4.17 and Lemma 3.34.

Then, by the same argument as the proof of Theorem 3.33, we can show the matrix elements of

[TABLE]

are the holomorphic functions on $\pi^{-1}_{|\boldsymbol{n}|}(U^{|\boldsymbol{n}|}_{z}(\bar{r}_{0}))$ with $\bar{r}_{0}=|q/t|^{\frac{|\boldsymbol{n}|-2}{|\boldsymbol{n}|-1}}$. Note that in this case, $\boldsymbol{y}^{-\boldsymbol{\mu}}f_{|\boldsymbol{m}|}(\boldsymbol{y};\boldsymbol{s}|q,t)$ is multiplied before the integration in $\boldsymbol{y}$. Under the assumption $|t|>|q|^{-(n-2)}$, $|t^{-1}|<\bar{r}_{0}$, and thus we can safely take the limit $x_{i}\to t^{|\boldsymbol{n}|-i}x$. This limit ends with the operator $\widetilde{V}^{(\boldsymbol{n})}$, and this completes the proof. ∎

This operator $\widetilde{V}^{(\boldsymbol{n})}(x)$ is a realization of $\mathcal{V}$ in the case $\boldsymbol{v}=t^{\boldsymbol{n}}\cdot\boldsymbol{u}$. This follows from the following relation which are the essentially same as (4.1).

Proposition 4.18**.**

For $r=1,\dots,N$, the \widetilde{V}^{(\boldsymbol{n})}\Big{(}\begin{array}[]{c}\boldsymbol{v}\\ \boldsymbol{u}\end{array}\,;x\Big{)} satisfies

[TABLE]

Proof.

By Lemma 3.19, we can get

[TABLE]

where $\widetilde{U}^{(\boldsymbol{n})}_{[i,k]_{\boldsymbol{n}}}$ is the operator obtained by replacing $\Phi^{(i)}(x_{[i,k]_{\boldsymbol{n}}})$ in $V^{(\boldsymbol{n})}$ with $Y^{(r)}(x_{[i,k]_{\boldsymbol{n}}})\Phi^{(i)}(qx_{[i,k]_{\boldsymbol{n}}})$, i.e.,

[TABLE]

Because of the $q$-difference of the operator $\Phi^{(i)}(x_{[i,k]_{\boldsymbol{n}}})$ in $\widetilde{U}^{(\boldsymbol{n})}_{[i,k]_{\boldsymbol{n}}}$, if $[i,k]_{\boldsymbol{n}}\neq 1$, the product of $\widetilde{U}^{(\boldsymbol{n})}_{[i,k]_{\boldsymbol{n}}}$ and $\prod\frac{(tx_{j}/x_{i};q)_{\infty}}{(qx_{j}/tx_{i};q)_{\infty}}$ vanishes under the principal specialization, i.e.,

[TABLE]

Furthermore, by the operator product (C.22), we have

[TABLE]

By (4.84), (4.87) and (4.88), it can be shown that

[TABLE]

By multiplying the both hand sides by $\left(1-t^{-|\boldsymbol{n}|}\frac{z}{x}\right)$, we have

[TABLE]

After some simple calculation, we obtain Proposition 4.18. ∎

Step2: Evaluation of the Matrix Elements of $\widetilde{V}^{(\boldsymbol{n})}(x)$

Next, we evaluate the matrix elements of $\widetilde{V}^{(\boldsymbol{n})}(x)$. Let $\boldsymbol{s}^{\prime}:=(s^{\prime}_{i})_{1\leq i\leq|\boldsymbol{m}|}$ with

[TABLE]

and $\boldsymbol{s}=(s_{i})_{1\leq i\leq|\boldsymbol{n}|+|\boldsymbol{m}|}$ be the same one given in (3.66), (3.67), i.e.,

[TABLE]

Further, we write $x_{|\boldsymbol{n}|+i}=y_{i}$. By the explicit formula for $\ket{Q_{\boldsymbol{\lambda}}}$ (Theorem 4.4) and Lemma 3.34, we have

[TABLE]

By virtue of the transformation formula of Proposition 4.8, the Macdonald function $p_{|\boldsymbol{n}|+|\boldsymbol{m}|}$ in (4.99) can be transformed to the summation of $p_{|\boldsymbol{m}|}$ as

[TABLE]

Here, we used Fact 4.17. Note that since $s_{|\boldsymbol{n}|+[i,k]_{\boldsymbol{m}}}=t^{1-n_{i}-k}v_{i}$, if $\nu$ does not satisfy $\nu_{[i,1]_{\boldsymbol{m}}}\geq\nu_{[i,2]_{\boldsymbol{m}}}\geq\cdots\geq\nu_{[i,m_{i}]_{\boldsymbol{m}}}$, i.e., cannot be regarded as an $N$-tuple of partitions, then $\mathsf{N}^{|\boldsymbol{n}|,|\boldsymbol{m}|}_{\nu}=0$. By Lemma C.2 and

[TABLE]

we obtain

[TABLE]

Here, we put

[TABLE]

Moreover, (4.102) leads to

[TABLE]

Note that

[TABLE]

Finally, it can be shown that the expression (4.104) coincides with the Nekrasov factors

[TABLE]

The coincidence between (4.104) and (4.106) can be proved by induction on $\boldsymbol{\lambda}$ and $\boldsymbol{\mu}$. We give some formulas to show this coincidence in Appendix C.4. The difference between (4.1) and (4.81) can be modified by the transformation

[TABLE]

Noting that the renormalization constant (3.70) are also modified, we can see that the equation (4.106) shows Theorem 4.4 in the case $\boldsymbol{v}=t^{\boldsymbol{n}}\cdot\boldsymbol{u}$. Therefore, by analytic continuation, Theorem 4.4 holds for the general case.

5. Refined Topological Vertex and Changing Preferred Direction

In [AFS], the intertwiners among the $\mathcal{U}$-modules are introduced, and their matrix elements are identical to the refined topological vertex of [IKV]. In this section, we compute the matrix elements of the ladder diagrams, which are obtained by gluing the intertwiners. The result shows the invariance under changing the preferred directions of the diagrams, which is a natural consequence of the S-duality in the string theory.

5.1. Trivalent intertwiner and refined topological vertex

We introduced the $\mathcal{F}^{(1,M)}$-module in Sec 2.2. In order to introduce the intertwiners, we need the other one, referred to as the $\mathcal{F}^{(0,1)}_{v}$.

Fact 5.1** ([FT, FFJMM]).**

Let $u$ be an indeterminate. We can endow a $\mathcal{U}$-module structure to $\mathcal{F}$ by setting

[TABLE]

with

[TABLE]

We denote this module as $\mathcal{F}^{(0,1)}$-module.

Then, we introduce the intertwiners among the modules defined above.

Fact 5.2** ([AFS]).**

When $w=-uv$, there exists a unique intertwiner which satisfies

[TABLE]

and the normalization condition $\bra{0}\Phi(1\otimes\ket{0})=1$. Moreover, its component $\Phi_{\lambda}$, defined by

[TABLE]

has the following realization,

[TABLE]

where

[TABLE]

Similarly, the following intertwiner exists uniquely,

[TABLE]

with normalization $\Phi^{\ast}(\ket{0})=\ket{0}\otimes 1+\cdots$, and its component, defined by

[TABLE]

is realized by

[TABLE]

where

[TABLE]

Notation 5.3**.**

In what follows, we mainly consider the $M=0$ case, and we introduce the simplified notations for the intertwiners.

[TABLE]

and their components,

[TABLE]

We also assign the trivalent diagrams to each intertwiner as follows. The arrows stand for the $\mathcal{F}^{(0,1)}$-modules, and we refer to this direction as the preferred direction, following the terminology of the refined topological vertex in [IKV].

**\Phi^{*}[u,v]$$u$$v$$-uv

**u$$v$$-uv$$\Phi[u,v]

These intertwiners can be identified with the refined topological vertex, invented in [IKV]. To see this, we define the refined topological vertex.

Definition 5.4**.**

The refined topological vertex is defined by

[TABLE]

where $c_{\lambda}$ is defined in (3.14), $||\lambda||^{2}=\sum_{i}\lambda_{i}^{2}$, $\rho=(-1/2,-3/2,-5/2,\dots)$ and $\kappa(\lambda)=\sum_{i}\lambda_{i}(\lambda_{i}+1-2i)$.

The following fact shows the intertwiners can be regarded as the refined topological vertex.

Fact 5.5** ([AFS]).**

[TABLE]

5.2. Changing preferred direction of ladder diagrams

In this subsection, we will show the matrix elements of the following ladder diagrams are identical up to some monomial factors.

Notation 5.6**.**

We introduce some notations $w_{i}$ and $w^{\prime}_{i}$ related to the spectral parameters $\boldsymbol{u},\boldsymbol{v}$ of the modules.

[TABLE]

Notation 5.7**.**

For integers $n\leq m$, we write

[TABLE]

Definition 5.8**.**

Define the map \mathcal{T}^{H}(\boldsymbol{u},\boldsymbol{v};w):\left(\ooalign{\textstyle\bigotimes\crcr\hss\raisebox{7.96527pt}{\scriptsize\curvearrowright}\hss}\rule[-2.58334pt]{0.0pt}{10.33327pt}_{i=1}^{N}\mathcal{F}^{(0,1)}_{v_{i}}\right)\to\left(\ooalign{\textstyle\bigotimes\crcr\hss\raisebox{7.96527pt}{\scriptsize\curvearrowright}\hss}\rule[-2.58334pt]{0.0pt}{10.33327pt}_{i=1}^{N}\mathcal{F}^{(0,1)}_{u_{i}}\right) by the following composition

[TABLE]

Here, $\cdots\otimes\ket{0}$ and $\ket{0}\otimes\cdots$ mean taking the vacuum expectation value at the level (1,0) modules. For simplicity, we introduce the following notation,

[TABLE]

We denote its matrix elements by

[TABLE]

Also, define the vertex operator $\mathcal{T}^{V}(\boldsymbol{u},\boldsymbol{v};w):\mathcal{F}_{\boldsymbol{u}}\to\mathcal{F}_{\boldsymbol{v}}$ by

[TABLE]

with

[TABLE]

and its matrix elements are denoted by

[TABLE]

The operator $\mathcal{T}^{V}$ guarantees the existence of the Mukadé operator, defined in Def. 4.1, and the following proposition completes the proof of Proposition 4.2.

Proposition 5.9**.**

For arbitrary $i\in\{1,2,\dots,N\}$, the operator $\mathcal{T}^{V}(\boldsymbol{u},\boldsymbol{v};w)$ satisfies

[TABLE]

Proof.

The proof is done by the direct calculation. For more details, see Sec 5.3. ∎

Remark 5.10**.**

The factor $\gamma^{-i}$ in the right hand side of (5.32) can be compensated by redefining the spectral parameters as $\gamma^{-1}u_{i}$. Under this redefinition, the matrix elements of $\mathcal{T}^{V}(\boldsymbol{u},\boldsymbol{v};w)$ agree with those of $\mathcal{V}(w)$.

Theorem 5.11**.**

The following equality between the two matrix elements holds:

[TABLE]

Here, $\sim$ means the both sides are identical up to some monomial factor and $\mathcal{G}$-factors $\mathcal{G}(z):=\prod_{i,j=0}^{\infty}(1-zq^{i}t^{-j})$, which appear from the normal orderings among $\Phi_{\emptyset}$’s and $\Phi^{*}_{\emptyset}$’s.

Proof.

We can show, up to the monomial factors and $\mathcal{G}$-factors, the both sides are equal to

[TABLE]

For the right hand side, it is obvious from Theorem 4.4 and Definition 3.37. For the left hand side, it can be shown by a direct computation, using some formulas in Appendix C. ∎

Remark 5.12**.**

This proposition implies the invariance under changing preferred directions of the refined topological vertex.

5.3. Proof of Proposition 5.9

First, we give the proof for $X^{(1)}(z)$. For $X^{(k)}(z)$ with $k>1$, the strategy of the proof is similar to that of $k=1$ case, and thus we just show the sketch of the proof.

$k=1$** case**

In order to simplify the notation, we rewrite the operator $\mathcal{T}^{V}(\boldsymbol{u},\boldsymbol{v};w)$ as

[TABLE]

with

[TABLE]

which appear from the normal orderings. Here, we put $\nu^{(0)}=\nu^{(N)}=\emptyset$ for convenience. We also omit the parameters $\boldsymbol{u},\boldsymbol{v}$ in $\mathcal{T}^{V}(\boldsymbol{u},\boldsymbol{v};w)$ in what follows.

We first compute the commutation relations between $\Lambda^{(j)},j=1,\dots,N$ and $\mathcal{T}^{V}(w)$. Putting $z_{j}=\gamma^{j-1}z$, we have

[TABLE]

where $A(\nu)$ and $R(\nu)$ is defined in Section 1. Through the computation, we used the equality

[TABLE]

Now, for $\nu^{(j)}\neq\emptyset$, fix a $y\in R(\nu^{(j)})$ and pick up the corresponding term from the above equation. Then, we rewrite $\nu^{(j)}-y$ as $\bar{\nu}^{(j)}$, and express the term using $\bar{\nu}^{(j)}$. This procedure ends with the following expression:

[TABLE]

While taking summation over $\boldsymbol{\nu}$, these terms are canceled by the terms which appear from the commutation relation between $\Lambda^{(j+1)}$ and $\mathcal{T}^{V}_{\boldsymbol{\nu}^{\prime}}(w)$ with $\nu^{(j)^{\prime}}=\bar{\nu}^{(j)}$,

[TABLE]

While summing up for $j$ from $1$ to $N$, the delta function which is not cancelled by the above mechanism exists at $A(\nu^{(0)})=A(\emptyset)$. This delta function vanishes when we multiply the both sides by $1-w_{1}/z$, and we finally obtain the main claim of Proposition 5.9 for the $k=1$ case.

$k>1$** case**

We use the following simplified notation,

[TABLE]

We can compute the commutation relations between $\Lambda^{(i_{1},\dots,i_{k})}(z)$ and $\mathcal{T}^{V}_{\boldsymbol{\nu}}(w)$ as,

[TABLE]

The delta functions related to $R(\nu^{(i_{j})})$ (for $j=1,\dots,k$) cancel those related to $A(\nu^{(i_{j})})$ which appear in the commutation relations between $\Lambda^{(\dots,i_{j}+1,\dots)}(z)$ and $\mathcal{T}^{V}_{\boldsymbol{\nu}}(w)$. This sequence of the cancellation begins when $i_{j}=i_{j-1}+1$ and terminates when $i_{j}=i_{j+1}-1$, because in those cases, the poles and zeros related to $\nu^{(i_{j})}$ cancel each other, and no delta functions related to them appear.

As a result, the only delta function which survives the above cancellation mechanism is that related to $A(\nu^{(0)})$. Similarly to the $k=1$ case, it vanishes when $1-w_{1}/z$ is multiplied, and we obtain the expected commutation relation between $X^{(k)}(z)$ and $\mathcal{T}^{V}(w)$.

Acknowledgment

The authors would like to thank H. Awata, B. Feigin, A. Hoshino, M. Kanai, H. Kanno, Y. Matsuo, M. Noumi and S. Yanagida for valuable discussions. The research of J.S. is supported by JSPS KAKENHI (Grant Numbers 15K04808 and 16K05186). Y.O. and M.F. are partially supported by Grant-in-Aid for JSPS Research Fellow (Y.O.: 18J00754, M.F.: 17J02745).

Appendix A Construction of Macdonald Symmetric Functions in terms of Topological Vertex

We can obtain Macdonald functions as matrix elements of some compositions of the intertwining operators. The similar derivation and its supersymmetric version are given in [Z]. Set

[TABLE]

where the overall factor is chosen for the later convenience. Though the prefactor consisting of the $\mathcal{G}$-factors gives zeros under the restriction of $\boldsymbol{v}$, but the $\mathcal{G}$-factors which appear from the normal orderings among $\Phi$’s and $\Phi^{*}$’s in $\mathcal{T}^{V}$, cancel those zeros, and thus the operator is well-defined. By operator products (C.26) and Lemma C.1, it can be seen that the Young diagrams are restricted to only one row when gluing intertwiners over the vertical representations (Fig. 2), i.e.,

[TABLE]

where we put $m_{0}=0$. Here,

$\textstyle\bigotimes$

$\curvearrowright$ is introduced in Notation 5.7. Then, we have $\widetilde{\mathcal{T}}_{1}(x)=\Phi^{(0)}(t^{-1}x)$. Note that the operators $\Phi^{(k)}(x)\,\,(k=0,\dots,N-1)$ in this section slightly differ from those in Sec. 3.2. In this section, $\Phi^{(k)}(x)$ is a map $\mathcal{F}_{t^{-\delta_{k+1}}\cdot\tilde{\boldsymbol{u}}}\to\mathcal{F}_{\boldsymbol{u}}$ with $\tilde{\boldsymbol{u}}=(\gamma^{-1}u_{1},\ldots,\gamma^{-1}u_{N})$, that is, the spectral parameter differs by the factor $\gamma^{-1}$.

We obtain the expression of Macdonald functions in terms of intertwiners of the DIM algebra. The following proposition says the vacuum expectation value of Fig. 3 gives the Macdonald function of the $A$ type.

Proposition A.1**.**

[TABLE]

Proof.

The operators $\widehat{\Phi}_{(m)}(z)$ and $\widehat{\Phi}^{*}_{(m)}(z)$ can be decomposed as

[TABLE]

where

[TABLE]

Then the tensor product of $\mathcal{A}(z)$ and $\mathcal{A}^{*}(z)$ corresponds to the screening current (Definition 3.10):

[TABLE]

Therefore, we have

[TABLE]

Here, we put

[TABLE]

As in the proof of Proposition 3.12 (Appendix D.1), $X^{(1)}(z)$ commutes with $\widetilde{S}^{(k)}(w)$ up to $q$-difference:

[TABLE]

Further, by the property of the operator products

[TABLE]

we obtain

[TABLE]

This leads to

[TABLE]

By this relation, we get

[TABLE]

Thus, the considered matrix elements are eigenfunctions of the difference operator $D_{N}^{1}$:

[TABLE]

∎

Remark A.2**.**

By using the expression (A.13), it is shown that the operator $\widetilde{\mathcal{T}}_{i}(x)$ also represents the screened vertex operator $\Phi^{(i-1)}(x)$. Whereas $\widetilde{\mathcal{T}}_{i}(x)$ is written by a sort of Jackson integrals, $\Phi^{(i-1)}(x)$ is introduced by counter integrals. In order to justify the equivalence of the two expressions, see the following argument.

We first note the following deformation of the formal series,

[TABLE]

Of course, this series does not converge for arbitrary $q$ and $\alpha$. However, by the following reason, we can choose some domains for $q$ and $\alpha$ so that the matrix elements of $\widetilde{\mathcal{T}}_{i}(x)$ converge there, and thus they are identical to those of $\Phi^{(i-1)}(x)$. We discuss the example of the $i=2$ case. The matrix elements of $\widetilde{\mathcal{T}}_{2}(x)$ are of the form,

[TABLE]

where $\alpha=u_{2}/u_{1}$, and $\sim$ means the both sides are equal up to some overall factors. The factor $\bra{P_{\boldsymbol{\lambda}}}:\Phi^{(0)}(x)\widetilde{S}^{(1)}(w):\ket{Q_{\boldsymbol{\mu}}}$ is some finite sum of the monomial of the form $x^{a}w^{b}$ with the maximums and minimums of $a$’s and $b$’s depending on $\boldsymbol{\lambda}$ and $\boldsymbol{\mu}$. Then, by taking the constant term in $w$ in the above, the running index $k$ is fixed as $k=n+b\geq b$. Therefore, when we choose the $q$ and $\alpha$ so that

[TABLE]

the deformation (A.22) is justified by putting $k<\mathrm{min}(b)$ terms zero, and thus

[TABLE]

This discussion can extend to the general $i$ case, and the equivalence between the Jackson integral and contour integral is justified.

Appendix B Basic Facts for Ordinary Macdonald Functions

In this section, we give the definition and basic facts for ordinary Macdonald functions. Let $\Lambda_{n}=\mathbb{C}[x_{1},\ldots,x_{n}]^{\mathfrak{S}_{n}}$ be the ring of symmetric polynomials, and $\Lambda=\lim_{\leftarrow}\Lambda_{n}$ be the projective limit in the category of graded rings, i.e., the ring of symmetric functions. $m_{\lambda}$ denotes the monomial symmetric function. We denote the power sum symmetric functions by $\mathsf{p}_{n}=\mathsf{p}_{n}(x)=\sum_{i\geq 1}x^{n}_{i}$. For a partition $\lambda$, set $\mathsf{p}_{\lambda}=\prod_{i\geq 1}\mathsf{p}_{\lambda_{i}}$. Macdonald functions are defined as orthogonal functions with respect to the following scalar product.

Definition B.1**.**

Define the bilinear form $\langle-,-\rangle_{q,t}:\Lambda\otimes\Lambda\rightarrow\mathbb{C}$ by

[TABLE]

where $m_{i}$ is the number of entries in $\lambda$ equal to $i$.

Fact B.2** ([Ma]).**

There exists an unique function $P_{\lambda}\in\Lambda$ such that

[TABLE]

The symmetric functions $P_{\lambda}$ are called Macdonald symmetric functions. It is known that

[TABLE]

Here, $c_{\lambda}$ and $c^{\prime}_{\lambda}$ are defined in (3.14). Set $Q_{\lambda}:=\langle P_{\lambda},P_{\lambda}\rangle_{q,t}^{-1}\cdot P_{\lambda}$, so that $\langle P_{\lambda},Q_{\lambda}\rangle_{q,t}=1$. In this paper, we regard power sum symmetric functions as variables of Macdonald functions and write $P_{\lambda}=P_{\lambda}(\mathsf{p}_{n})$, $Q_{\lambda}=Q_{\lambda}(\mathsf{p}_{n})$. These are abbreviation for $P_{\lambda}(\mathsf{p}_{1},\mathsf{p}_{2},\ldots)$ and $Q_{\lambda}(\mathsf{p}_{1},\mathsf{p}_{2},\ldots)$. We often substitute the generators $a_{n}$ in the Heisenberg algebra into Macdonald functions as $P_{\lambda}(a_{-n})$. Note that this substitution preserves the properties of Macdonald functions as follows. The space of symmetric functions $\Lambda$ and the Fock space $\mathcal{F}$ are isomorphic as graded vector spaces, and they can be identified by

[TABLE]

Further, the bilinear form on the Fock spaces preserves the structure of the scalar product $\langle-,-\rangle_{q,t}$ under this identification, i.e.,

[TABLE]

We describe some facts for Macdonald functions that are used in this paper.

Kernel function

The following function $\Pi(x,y;q,t)$ is called the kernel function:

[TABLE]

This function $\Pi(x,y;q,t)$ can be expanded in terms of dual bases parametrized by partitions as follows.

Fact B.3** ([Ma]).**

Let $u_{\lambda}$, $v_{\lambda}\in\Lambda$ be homogeneous symmetric functions of level $|\lambda|$ and $\{u_{\lambda}\}$, $\{v_{\lambda}\}$ form $\mathbb{C}$-bases on $\Lambda$. Then, the followings are equivalent:

[TABLE]

Note that especially, we have

[TABLE]

Pieri formula

Define the symmetric function $g_{r}\in\Lambda$ to be the expansion coefficient of $y^{r}$ in the following series:

[TABLE]

For a partition $\lambda$ and a coordinate $s$, we write

[TABLE]

Fact B.4** ([Ma]).**

We have the Pieri rules:

[TABLE]

Here, the summation is over the partitions $\lambda$ such that $\lambda/\mu$ is a horizontal $r$-strip, i.e., $\lambda/\mu$ has at most one box in each column. $R_{\lambda/\mu}$ (resp. $C_{\lambda/\mu}$) is the union of the rows (resp. columns) that intersect $\lambda-\mu$.

Note that by this formula, we can see that $Q_{\mu}$ ($\mu\ngeq\lambda$) does not appear in the expansion of the product $\prod_{i\geq 1}g_{\lambda_{i}}$ in the basis of Macdonald functions.

Another scalar product

There is another scalar product for which Macdonald functions $P_{\lambda}$’s are pairwise orthogonal. Let $L_{r}=\mathbb{C}[x_{1}^{\pm 1},\ldots,x_{r}^{\pm}]$ be the $\mathbb{C}$-algebra of Laurent polynomials in $r$ variables. For $f(x_{1},\ldots,x_{r})\in L_{r}$, we put $\bar{f}=f(x_{1}^{-1},\ldots,x_{r}^{-1})$.

Definition B.5**.**

The scalar product $\langle-,-\rangle^{\prime}_{r}$ on $L_{r}$ is defined by

[TABLE]

where

[TABLE]

Fact B.6** ([Ma]).**

Let P_{\lambda}^{(r)}=P_{\lambda}\big{|}_{x_{i}\rightarrow 0\,(i>r)},\,Q_{\lambda}^{(r)}=Q_{\lambda}\big{|}_{x_{i}\rightarrow 0\,(i>r)}\in\Lambda_{r} be the Macdonald symmetric polynomials in $r$ variables. Then

[TABLE]

where $\ell(\lambda),\ell(\mu)\leq r$.

Appendix C Some Useful Formulas

C.1. List of operator products

In this subsection, we list some formulas for the normal ordering among the various operators appeared in the main text. We have

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

and with $\mathcal{G}(z)=\prod_{i,j=0}^{\infty}(1-zq^{i}t^{-j})$,

[TABLE]

C.2. On Nekrasov factors

For a partitions $\lambda$ and non-negative integers $r,s\in\mathbb{Z}_{\geq 0}$, define $\mathsf{B}_{r,s}(\lambda)$ to be the partition obtained by removing the 1st to $s$-th rows and the 1st to $r$-th columns, i.e.,

[TABLE]

For example, if $\lambda=(5,5,4,4,4,1,1)$, then $\mathsf{B}_{2,1}(\lambda)=(3,2,2,2)$. We have the following vanishing condition for the Nekrasov factor.

Lemma C.1**.**

If $m\geq 0$ and $n\leq 0$, then

[TABLE]

If $m\leq-1$ and $n\geq 1$, then

[TABLE]

Note that in particular, $N_{\emptyset,\mu}(t)\neq 0$ if and only if $\mu=(m)$ for some $m\in\mathbb{Z}_{\geq 0}$.

C.3. Duality of functions $p_{n}(x;s|q,t)$ and $f_{n}(x;s|q,t)$

Lemma C.2**.**

Let $\boldsymbol{\lambda}$ and $\boldsymbol{\mu}$ satisfy $\ell(\lambda^{(i)}),\ell(\mu^{(i)})\leq n_{i}$.

[TABLE]

Here, $s(\boldsymbol{\lambda})=(s_{j}(\boldsymbol{\lambda}))_{1\leq j\leq|\boldsymbol{n}|}$, $s_{[i,k]_{\boldsymbol{n}}}(\boldsymbol{\lambda})=q^{\lambda^{(i)}_{k}}t^{1-k}u_{i}$.

Proof.

We denote the LHS by $F(\boldsymbol{\lambda}|\boldsymbol{\mu})$. Inserting the Macdonald operator $D_{n}^{1}(s;q,q/t)$ and integrating by parts give

[TABLE]

where $\bar{s}_{[i,k]_{\boldsymbol{n}}}=u_{i}t^{1-k}$. The LHS gives $\epsilon_{\boldsymbol{\mu}}F(\boldsymbol{\lambda}|\boldsymbol{\mu})$, and the RHS gives $\epsilon_{\boldsymbol{\lambda}}F(\boldsymbol{\lambda}|\boldsymbol{\mu})$. Thus we can show $F(\boldsymbol{\lambda}|\boldsymbol{\mu})=C(\boldsymbol{\lambda})\delta_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ with the coefficient $C(\boldsymbol{\lambda})=F(\boldsymbol{\lambda}|\boldsymbol{\lambda})$. We can show $C(\boldsymbol{\lambda})=1$ because both $f_{|\boldsymbol{n}|}(x;s(\boldsymbol{\lambda})|q,q/t)$ and $p_{|\boldsymbol{n}|}(x;s(\boldsymbol{\mu})|q,q/t)$ are in $\mathbb{C}(s_{1},\ldots,s_{|\boldsymbol{n}|})[[x_{2}/x_{1},x_{3}/x_{2},\ldots,x_{|\boldsymbol{n}|}/x_{|\boldsymbol{n}|-1}]]$ and so is their product. ∎

C.4. Some formulas to prove (4.106)

The coincidence between (4.104) and (4.106) can be identified with the following equation.

Proposition C.3**.**

[TABLE]

The following lemmas are formulas for proving (C.34).

Lemma C.4**.**

With $n=\ell(\lambda)$ and $m=\ell(\mu)$, the following holds:

[TABLE]

where

[TABLE]

The equality that we obtain by removing the factors of this type from the both sides of (C.34), can be shown by using following relations.

Lemma C.5**.**

Under the $\lambda_{i}\to\lambda_{i}+1$ or $\mu_{i}\to\mu_{i}+1$, we also have the following induction relations,

[TABLE]

where

[TABLE]

Notation C.6**.**

In what follows in this proof, we set

[TABLE]

Definition C.7**.**

We set

[TABLE]

Lemma C.8**.**

Under $\lambda^{(i)}_{k}\to\lambda^{(i)}_{k}+1$, that is, $s_{[i,k]_{\boldsymbol{n}}}\to qs_{[i,k]_{\boldsymbol{n}}}$, we have

[TABLE]

with $v_{ij}=v_{i}/v_{j}$, and

[TABLE]

Lemma C.9**.**

Similarly, under $\mu^{(i)}_{k}\to\mu^{(i)}_{k}+1$, that is, $\sigma_{[i,k]_{\boldsymbol{n}}}\to q\sigma_{[i,k]_{\boldsymbol{n}}}$, we have

[TABLE]

Combining these identities, we complete the proof of (C.34).

Appendix D Some Proofs of Lemmas and Propositions

D.1. Proof of Proposition 3.12

By the operator product formulas (C.1)-(C.6), the operator $\Lambda^{(j)}(z)$ with $j\neq i,i+1$ does not contribute in the commutation relation, and it can be shown that

[TABLE]

Hence, it is enough to consider the relation only with $u_{i}\Lambda^{(i)}(z)+u_{i+1}\Lambda^{(i+1)}(z)$.

We have

[TABLE]

and

[TABLE]

Therefore, by the property $g_{i}(qz)=\frac{u_{i+1}}{tu_{i}}g_{i}(z)$ with respect to the $q$-difference, we obtain

[TABLE]

∎

D.2. Proof of Lemma 3.19

First, we show the relation for $k=0$. In this proof, we write

[TABLE]

for $i_{1}<\cdots<i_{r}$. By the operator products (C.16)-(C.19) and the relation $:\Phi^{(0)}(w)\Lambda^{(1)}(tw):=\Phi^{(0)}(qw)\Psi^{+}(w)$, it can be shown that if $i_{1}=1$,

[TABLE]

If $i_{1}\geq 2$,

[TABLE]

Thus, we obtain the relation in the case $k=0$:

[TABLE]

Applying the screening operators to this relation from the right side, we have the case $k\neq 0$. Indeed, $\Psi^{+}(x)$ commutes with $S^{(i)}(y)$:

[TABLE]

Noting that we have

[TABLE]

and by virtue of (D.9) and commutativity of the screening operators, we can establish the relation for general $k$.

We can show the commutativity of the screening operators as follows. First, it is clear that

[TABLE]

By calculating the commutation relation as in the proof of Proposition 3.12, the RHS of (D.11) consists of terms as

[TABLE]

Note that $j_{\ell+2}\geq i+2$. Let us investigate the positions of poles in $y_{i}$. Combining the $\theta$-functions containing $y_{i}$ in $g(x,y_{1},\ldots,y_{k})$ and the operator products among screening currents and $\Phi^{(0)}(x)$, there appears the factor

[TABLE]

From $\Phi^{(0)}(x)$ and $\Lambda^{(i)}(ty_{1}/q)$ in $\Lambda^{(j_{1},\ldots,j_{\ell},i,j_{\ell+2}\ldots,j_{r})}$, we have

[TABLE]

Noticing that for $i\geq 2$, the operator product of $S^{(i-1)}(y_{i-1})$ and $\Lambda^{(i)}(ty_{i}/q)$ are

[TABLE]

we have the following set of poles of (D.11) in $y_{i}$:

[TABLE]

and

[TABLE]

If $r=1$, it gives us the all poles. In general, this list does not exhaust the possible poles. In case $r\geq 2$, from $S^{(j)}(y_{j})$ and $\Lambda^{(j_{m})}$ in $\Lambda^{(j_{1},\ldots,j_{\ell},i,j_{\ell+2}\ldots,j_{r})}$ with $m\neq\ell+1$, we have extra poles. From the operator product formulas for them, we have

[TABLE]

In addition, from $\Phi^{(0)}(x)$ and $\Lambda^{(j_{m})}$, the following factor arises:

[TABLE]

Summarizing these, we can show that the poles in $y_{i}$ are in the following positions (Though not all following points are poles, all poles should be in the followings or (D.16)-(D.20)). For $i\geq 1$,

[TABLE]

and for $i\geq 2$,

[TABLE]

For the given integration contour, the poles (D.17), (D.19), (D.20), (D.25) and (D.26) are in the disk $\{z;|z|<|qy_{i}|\}$. On the other hand, the poles (D.16), (D.18), (D.23) and (D.24) are in $\{z;|z|>|y_{i}|\}$. Therefore, the change of variable $y_{i}\rightarrow qy_{i}$ is not affected by these poles, and the commutation relation (D.11) becomes zero after the integrals. ∎

D.3. Proof of Proposition 3.27

By taking the constant terms of $V^{(\boldsymbol{n})}(x_{1},\ldots,x_{|\boldsymbol{n}|})\ket{\boldsymbol{0}}$ with respect to $x_{i}$, the proportional constant $\mathcal{R}^{\boldsymbol{n}}_{\boldsymbol{\lambda}}$ is calculated as the expansion coefficient in front of $\ket{Q_{\boldsymbol{\lambda}}}$ in the basis of generalized Macdonald functions. We first consider only the operators that contain the creation operators $a^{(N)}_{-n}$’s with respect to the $N$-th Fock space. That is, we take the constant terms of

[TABLE]

Here, we used the expansion formula (3.24), and $\widetilde{\mathrm{M}}=\mathrm{Mat}(n_{N},N-1;\mathbb{Z})$ is the set of $n_{N}\times(N-1)$ matrices with integral entries. We denote $x_{[N,i]_{\boldsymbol{n}}}$ by $y_{i,0}$ in this proof. Further we set

[TABLE]

Let $C(z)=\sum_{k\geq 0}C_{k}z^{k}$, $\widetilde{C}(z)=\sum_{k\geq 0}\widetilde{C}_{k}z^{k}$ and $C^{(\pm)}(z)=\sum_{k\geq 0}C^{(\pm)}_{k}z^{k}$ be the formal power series defined by

[TABLE]

These series correspond to the operator product formulas among $\Phi^{(0)}$ and $S^{(i)}$’s. Moreover, we write

[TABLE]

for $e=((e^{(m)}_{i,j})_{i,j=1}^{n_{N}})_{1\leq m\leq N-1},k=((k^{(m)}_{i,j})_{i,j=1}^{n_{N}})_{1\leq m\leq N-1}\in M_{n_{N}}^{N-1}$, and $\ell=((\ell^{(m)}_{i,j})_{i,j=1}^{n_{N}})_{0\leq m\leq N-2}\in M_{n_{N}}^{N-1}$. Here $M_{n}$ is the set of strictly upper triangular $n\times n$ matrices with nonnegative integral entries. With these notations, (D.27) can be rewritten as

[TABLE]

Here, $\widetilde{\mathrm{M}}_{\geq 0}=\mathrm{Mat}(n_{N},N-1;\mathbb{Z}_{\geq 0})$, $\Phi^{(0)}(z)=\sum_{n\in\mathbb{Z}}\Phi^{(0)}_{n}z^{-n}$, and $S^{(i)}(z)=\sum_{n\in\mathbb{Z}}S^{(i)}_{n}z^{-n}$. Since the integral gives us the constant terms in $y_{i,m}$, we have

[TABLE]

and

[TABLE]

Since $\sum_{i=1}^{n_{N}}E^{(m)}_{i}=0$ and $\sum_{i=1}^{n_{N}}(L_{i}^{(m)}+K_{i}^{(m)})=0$, it is shown that

[TABLE]

Therefore, $\sum_{i=1}^{n_{N}}\mathsf{a}_{i}^{(N-1)}$ takes its maximum value $|\lambda^{(N)}|$ when $\mathsf{a}^{(m)}_{i}=0$ for all $i$ and $m\leq N-2$. Since only the operators $:\prod_{i=1}^{n_{N}}S^{(N-1)}_{-\mathsf{a}^{(N-1)}_{i}}:$ have the creation operators acting on the $N$-th Fock space, it is clear that the maximum degree in the $N$-th Fock component is $|\lambda^{(N)}|$.

Before taking expansion coefficients in the basis of generalized Macdonald functions, we investigate the one in the basis of products of ordinary Macdonald functions $\prod_{i=1}^{N}Q_{\mu^{(i)}}(a^{(i)}_{-n})\ket{\boldsymbol{0}}$. Consider the terms of level $|\lambda^{(N)}|$ with respect to the $N$-th Fock space. Then, $\mathsf{a}^{(N-1)}_{i}$ satisfies

[TABLE]

Furthermore, by the form of $E^{(m)}_{i}$, $K^{(m)}_{i}$ and $L^{(m)}_{i}$, it can be seen that only the following vectors appear:

[TABLE]

By the Pieri formula (Fact B.4), we can write the terms of level $|\lambda^{(N)}|$ with respect to the $N$-th Fock space as

[TABLE]

Here, $g^{(N)}_{n}$ is defined by

[TABLE]

Therefore, there appears $Q_{\lambda^{(N)}}(a^{(N)}_{-n})\ket{\boldsymbol{0}}$ only in the case that $\mu=\lambda^{(N)}$ on (D.42), i.e., the case that

[TABLE]

for all $i,j,m$. Then

[TABLE]

for all $i,m$.

From the above discussion, we have

[TABLE]

Here $\mathcal{O}\left(P\right)$ expresses the terms $\prod_{i=1}^{N}Q_{\mu^{(i)}}(a^{(i)}_{-n})\ket{\boldsymbol{0}}$ with $\boldsymbol{\mu}$ satisfying the proposition $P$. By repeating the similar argument $N-1$ times, we obtain

[TABLE]

The existence theorem of generalized Macdonald functions (Fact 3.6) shows that the coefficient in front of $\ket{Q_{\boldsymbol{\lambda}}}$ in the basis of generalized Macdonald functions is the same as the one in front of $\prod_{i=1}^{N}Q_{\lambda^{(i)}}(a^{(i)}_{-n})\ket{\boldsymbol{0}}$ in (D.48). ∎

D.4. Proof of Lemma 4.11

First, it can be shown that

[TABLE]

and

[TABLE]

By (D.4) and (D.50), we have

[TABLE]

Furthermore, we can see

[TABLE]

and

[TABLE]

Combining (D.4), (D.57) and (D.4) yields Lemma 4.11. ∎

D.5. Proof of Lemma 4.13

Set for short

[TABLE]

Then $\phi^{m,n+m-1}_{\nu}=ABCD$. First, we can get

[TABLE]

The first product in (D.5) reproduces the factors in $d_{m}((\theta_{i});(q^{\mu_{i}}s_{n+i})|q,t)$, i.e., the first product in (3.36). Next, we have

[TABLE]

where

[TABLE]

$BE$ reproduces the factors in $d_{m}((\theta_{i});(q^{\mu_{i}}s_{n+i})|q,t)$:

[TABLE]

This corresponds to the second product in (3.36). The product of $\mathsf{N}^{n,m-1}_{\mu}$, $D$ and the remaining factor in (D.5) is

[TABLE]

Therefore, Lemma 4.13 follows. ∎

Appendix E Kac determinant revisited

The formula for the Kac determinant with respect to the vectors $\ket{X_{\boldsymbol{\lambda}}}$ has been discussed in [O]. That shows the fact that $\ket{X_{\boldsymbol{\lambda}}}$ form a basis on the Fock space (Fact 2.10). For the sake of reader’s convenience, we revisit the proof, clarifying the choice of the integral cycles. Here, we construct the $q$-invariance cycles by using the elliptic theta function.

Definition E.1**.**

Let $1\leq k\leq N-1$ and $u_{k}=q^{s}t^{-r}u_{k+1}$ ($r,s\in\mathbb{Z}_{>0}$ ). Define the vector $\ket{\chi^{(k)}_{r,s}}\in\mathcal{F}_{\boldsymbol{u}}$ by the integral

[TABLE]

Here and hereafter, we use the shorthand notation

[TABLE]

where the cycle is the $r$-dimensional torus $T:|z_{1}|=\cdots=|z_{r}|=1$. Note that $\boldsymbol{u}$ is the spectral parameter of the codomain of $S^{(k)}(z_{1})$.

Proposition E.2**.**

The vector $\ket{\chi^{(k)}_{r,s}}$ does not vanish. In particular, this is of level $rs$.

Let us prepare a lemma with respect to the symmetrization of theta functions. Set

[TABLE]

Lemma E.3**.**

[TABLE]

As for the proof of this lemma, see the proof of Lemma 4 in [JLMP].

Proof of Proposition E.2. First, by using the operator products of screening currents and Lemma E.3, we have

[TABLE]

Here, $\Delta(z)$ is defined in (B.15). Note that $:\prod_{i=1}^{r}S^{(k)}(z_{i}):\ket{\boldsymbol{0}}$ can be regarded as the kernel function for the Macdonald functions. Hence, it is expanded in terms of the Macdonald functions (See Fact B.3). Note that $\prod_{i=1}^{r}z_{i}^{-s}$ is the Macdonald polynomial with a rectangular Young diagram in $r$ variables. Therefore, (E.5) can be written as

[TABLE]

where $P_{\lambda}^{(r)}(z)$ and $Q^{(r)}_{\lambda}(z)$ denote the Macdonald polynomials in $r$ variables,

[TABLE]

and $\langle-,-\rangle^{\prime}_{r}$ denotes the scalar product defined in Appendix B. Since the Macdonald polynomials are pairwise orthogonal for $\langle-,-\rangle^{\prime}_{r}$ and the inner product $\langle P^{(r)}_{(s^{r})},Q^{(r)}_{(s^{r})}\rangle^{\prime}_{r}$ can be evaluated (Fact B.6) to be nonvanishing. Thus, $\ket{\chi^{(k)}_{r,s}}\neq 0$ and of level $rs$. ∎

We show the following commutativity with the algebra $\mathsf{U}(N)$. The proof is similar to the case corresponding to the Minimal model, given in [JLMP].

Proposition E.4**.**

Let $r,s\in\mathbb{Z}_{>0}$ and $k,j\in\{1,\ldots,r\}$. Further we assume $|t|<|q|$. Then

[TABLE]

where the spectral parameter of the codomain of $S^{(k)}(w_{1})$ is $\boldsymbol{u}$ with $u_{k}=q^{s}t^{-r}u_{k+1}$.

Proof.

As in the proof of Proposition 3.12, it suffices to consider the relation only with $\Lambda^{(k)}(z)+\Lambda^{(k+1)}(z)$. By (D.4), we have

[TABLE]

By symmetrizing the variables $w_{i}$’s, we have

[TABLE]

By Lemma E.3, this can be rewritten as

[TABLE]

In this expression, we have poles in $w_{l}$ at $w_{l}=0$, $w_{l}=q^{n}tw_{i}$ and $w_{l}=q^{-n+1}t^{-1}w_{i}$ ($i\neq l$, $n=1,2,\ldots$). Since $|t|<|q|$, they do not change the integral while we $q$-shift the cycle as $w_{l}\rightarrow qw_{l}$. Therefore, the integral (E.12) is zero. ∎

Proposition E.2 and Proposition E.4 show the existence of the singular vectors of the algebra $\mathsf{U}(N)$.

Corollary E.5**.**

The vector $\ket{\chi^{(k)}_{r,s}}$ is a singular vector of level $rs$, i.e.,

[TABLE]

for all $n>0$ and $i=1,\ldots,N$.

We revisit the proof of the following formula for the Kac determinant $\mathrm{det}_{n}:=\det\left(\Braket{X_{\boldsymbol{\lambda}}}{X_{\boldsymbol{\mu}}}\right)_{\boldsymbol{\lambda},\boldsymbol{\mu}\vdash n}$.

Proposition E.6**.**

We have

[TABLE]

Here $b_{\lambda}(q):=\prod_{i\geq 1}\prod_{k=1}^{m_{i}}(1-q^{k})$, $b^{\prime}_{\lambda}(q):=\prod_{i\geq 1}\prod_{k=1}^{m_{i}}(-1+q^{k})$. $P^{(N)}(n)$ is the number of $N$-tuples of Young diagrams of size $n$, i.e., \#\big{\{}\boldsymbol{\lambda}=(\lambda^{(1)},\ldots,\lambda^{(N)})\big{|}|\boldsymbol{\lambda}|=n\big{\}}. In particular, if $N=1$,

[TABLE]

Proof.

The inner product $\braket{X_{\boldsymbol{\lambda}}}{X_{\boldsymbol{\lambda}}}$ can be calculated by commutation relations of $X^{(i)}_{n}$. The parameters $u_{1},\ldots,u_{N}$ arise from the eigenvalues of $X^{(i)}_{0}$. Therefore, it can be seen that $\braket{X_{\boldsymbol{\lambda}}}{X_{\boldsymbol{\lambda}}}$ is a polynomial in

[TABLE]

over $\mathbb{Q}(q^{\frac{1}{2}},t^{\frac{1}{2}})$, and thus so does the $\mathrm{det}_{n}$. Define the action of the symmetric group $\mathfrak{S}_{N}$ (the Weyl group of type $A_{N-1}$) on polynomials in $u_{j}$ in the usual way. Since $m_{(1^{i})}(u_{1},\ldots,u_{N})$ is invariant with respect to this action, $\mathrm{det}_{n}$ is also invariant, i.e., a symmetric polynomial in $u_{j}$.

Furthermore, let us introduce the new parameters $u_{i}^{\prime}$ and $u^{\prime\prime}$ by

[TABLE]

Then $\braket{X_{\boldsymbol{\lambda}}}{X_{\boldsymbol{\mu}}}$ can be decomposed as

[TABLE]

Therefore, $\mathrm{det}_{n}$ can be written as

[TABLE]

where $F(u^{\prime}_{1},\ldots,u^{\prime}_{N})$ is some polynomial in $u^{\prime}_{i}$. Note that the maximum degree of $F(u^{\prime}_{1},\ldots,u^{\prime}_{N})$ with respect to each $u^{\prime}_{i}$ is $2\sum_{|\boldsymbol{\lambda}|=n}\sum_{k=1}^{N}\ell(\lambda^{(k)})$.

By Corollary E.5, it can be seen that for $r,s\in\mathbb{Z}_{>0}$ with $rs\leq n$, the Kac determinant $\mathrm{det}_{n}$ has the factors

[TABLE]

in the usual way. By the $\mathfrak{S}_{N}$ invariance, $\mathrm{det}_{n}$ has also the factor

[TABLE]

for $i\neq j$. Noticing the degree of $F(u^{\prime}_{1},\ldots,u^{\prime}_{N})$, we can see that

[TABLE]

Here, $g_{N,n}(q,t)\in\mathbb{Q}(q^{\frac{1}{2}},t^{\frac{1}{2}})$. Thus, we obtained the vanishing loci of the Kac determinant $\mathrm{det}_{n}$. The prefactor $g_{N,n}(q,t)$ has been evaluated in [O]. ∎

As a corollary of Proposition E.6, Fact 2.10 follows.

Appendix F Examples

F.1. Examples of $\ket{K_{\boldsymbol{\lambda}}}$

We present examples of the transition matrix $\alpha^{(\pm)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ from $\ket{K_{\boldsymbol{\lambda}}}$ to the PBW-type basis $\ket{X_{\boldsymbol{\lambda}}}$.

Examples of $\alpha^{(+)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ in the case $N=1$:

[TABLE]

[TABLE]

Examples of $\alpha^{(+)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ in the case $N=2$:

[TABLE]

[TABLE]

Examples of $\alpha^{(-)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ in the case $N=2$:

[TABLE]

[TABLE]

Examples of $\alpha^{(+)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ in the case $N=3$:

[TABLE]

Examples of $\alpha^{(-)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ in the case $N=3$:

[TABLE]

F.2. Examples of matrix elements $\braket{X_{\boldsymbol{\lambda}}}{\mathcal{V}(w)}{X_{\boldsymbol{\mu}}}$

In this section, we demonstrate how to calculate the matrix elements $\braket{X_{\boldsymbol{\lambda}}}{\mathcal{V}(w)}{X_{\boldsymbol{\mu}}}$ by the defining relation of $\mathcal{V}(w)$. Let us first explain it in general case. If $\boldsymbol{\lambda}\neq(\emptyset,\ldots,\emptyset)$, let $j=\min\{i|\lambda^{(i)}\neq\emptyset\}$. The defining relation gives

[TABLE]

The first term can be rewritten by matrix elements $\braket{X_{\boldsymbol{\nu}}}{\mathcal{V}(w)}{X_{\boldsymbol{\mu}}}$ satisfying $|\boldsymbol{\nu}|=|\boldsymbol{\lambda}|-1$ (particularly, in the case $\lambda^{(j)}_{1}-1<\lambda^{(j)}_{2}$), and the second term can be expanded by vectors $\ket{X_{\boldsymbol{\rho}}}$ of level $|\boldsymbol{\rho}|=|\boldsymbol{\mu}|-\lambda^{(j)}_{1}$ or $|\boldsymbol{\mu}|-\lambda^{(j)}_{1}+1$. (If $|\boldsymbol{\mu}|-\lambda^{(j)}_{1},|\boldsymbol{\mu}|-\lambda^{(j)}_{1}+1<0$, it means just [math] vectors.) If $\boldsymbol{\lambda}=(\emptyset,\ldots,\emptyset)$, moving negative modes $X^{(i)}_{-n}$ to the left side of $\mathcal{V}(w)$ by the defining relation, we have the expression in terms of Young diagrams of smaller size. In this way, the matrix elements $\braket{X_{\boldsymbol{\nu}}}{\mathcal{V}(w)}{X_{\boldsymbol{\mu}}}$ can be inductively and uniquely determined.

The followings are examples.

In $N=1$ case

First, by the defining relation of $\mathcal{V}(w)$, we have

[TABLE]

By $X^{(1)}_{0}\ket{0}=u_{1}\ket{0}$, $\bra{0}X^{(1)}_{0}=v_{1}\bra{0}$ and $X^{(1)}_{1}\ket{0}=0$, we get

[TABLE]

Similarly, it is easily seen that

[TABLE]

Moreover, we have

[TABLE]

By the direct calculation of the free field expression, we have

[TABLE]

By (F.48) and (F.49), we obtain

[TABLE]

The matrix element $\braket{X_{\emptyset}}{\mathcal{V}(w)}{X_{(1)}}$ is already calculated. Since $\ket{X_{(1)}}=\ket{K_{(1)}}$ and $\bra{X_{(1)}}=\bra{K_{(1)}}$ in this particular case, $\braket{X_{(1)}}{\mathcal{V}(w)}{X_{(1)}}$ is factorized and corresponds to the Nekrasov factor (the right hand side of (4.2)):

[TABLE]

(F.45),(F.2) and (F.51) are the simplest examples of our main theorem (Theorem 4.4).

We list other cases. Let us first prepare the formula for the action of the algebra $\mathsf{U}(N)$ on the PBW-type basis $\ket{X_{\boldsymbol{\lambda}}}$.

[TABLE]

[TABLE]

[TABLE]

By using these relation, the matrix elements for larger Young diagrams are inductively determined as follows:

[TABLE]

[TABLE]

[TABLE]

By combining these matrix elements and the examples of transition matrices $\alpha^{(\pm)}_{\boldsymbol{\lambda},\boldsymbol{\mu}}$ from $\ket{K_{\boldsymbol{\lambda}}}$ to $\ket{X_{\boldsymbol{\mu}}}$ in the last subsection, we can check Theorem 4.4.

In $N=2$ case

If there is only one box in all Young diagrams, it is clear that

[TABLE]

As in the $N=1$ case, we prepare the formula for the action of $X^{(i)}_{n}$.

[TABLE]

where

[TABLE]

By these, the matrix elements can be written as

[TABLE]

Furthermore, the direct calculation gives

[TABLE]

[TABLE]

Here,

[TABLE]

[TABLE]

Thus, we obtain

[TABLE]

In $N=3$ case

Also in the representations of higher level, we can calculate the matrix elements similarly:

[TABLE]

List of Notations

General Notations

[TABLE]

Algebras and Representations

[TABLE]

Vertex Operators associated with Generalized Macdonald functions

[TABLE]

Symmetric functions and vectors in the $N$-fold tensor Fock spaces

[TABLE]

Factors in the Macdonald functions and hypergeometric series

[TABLE]

where $n$, $m$ are nonnegative integers and $\mu=(\mu_{i})_{1\leq i\leq m}\in\mathbb{Z}^{m}$.

Mukadé Operator and its Relatives

[TABLE]