Feature Augmentations for High-Dimensional Learning

TL;DR

This paper introduces a simple PCA-based feature augmentation technique that improves the performance and interpretability of high-dimensional supervised learning models, especially in financial NLP tasks.

Contribution

The paper proposes a novel, computationally efficient feature augmentation method using factor extraction to enhance high-dimensional learning models.

Findings

Boosts prediction accuracy across various algorithms and datasets.

Improves interpretability and numerical stability of models.

Effective in financial news and stock return prediction tasks.

Abstract

High-dimensional measurements are often correlated which motivates their approximation by factor models. This holds also true when features are engineered via low-dimensional interactions or kernel tricks. This often results in over parametrization and requires a fast dimensionality reduction. We propose a simple technique to enhance the performance of supervised learning algorithms by augmenting features with factors extracted from design matrices and their transformations. This is implemented by using the factors and idiosyncratic residuals which significantly weaken the correlations between input variables and hence increase the interpretability of learning algorithms and numerical stability. Extensive experiments on various algorithms and real-world data in diverse fields are carried out, among which we put special emphasis on the stock return prediction problem with Chinese financial news data due to the increasing interest in NLP problems in financial studies. We verify the capability of the proposed feature augmentation approach to boost overall prediction performance with the same algorithm. The approach bridges a gap in research that has been overlooked in previous studies, which focus either on collecting additional data or constructing more powerful algorithms, whereas our method lies in between these two directions using a simple PCA augmentation.

Figures

Tables

Table 1. Table 1: Portfolio performances from 2015 to 2019 with transaction fees using VW strategies

	$\left(𝑿\right)$		$({\widehat{𝑭}}_0,\widehat{𝑼})$		$({\widehat{𝑭}}_{\mathrm{inter}},{\widehat{𝑭}}_0,\widetilde{𝑼})$		$({\widehat{𝑭}}_{\mathrm{rbf}},{\widehat{𝑭}}_0,\widetilde{𝑼})$		$({\widehat{𝑭}}_{\mathrm{poly}},{\widehat{𝑭}}_0,\widetilde{𝑼})$		$({\widehat{𝑭}}_{\mathrm{fnn}},{\widehat{𝑭}}_0,\widetilde{𝑼})$
Portfolio	APR %	SR	APR %	SR	APR %	SR	APR %	SR	APR %	SR	APR %	SR
Classification
L+S	21.4	1.89	25,4	2.14	19.3	1.71	25.9	2.12	20.2	1.75	27.3	2.23
L	35.1	3.39	36.2	3.50	33.5	3.26	37.8	3.56	31.6	3.16	36.0	3.48
S	-10.2	-1.80	-8.0	-1.36	-10.8	-1.83	-8.72	-1.42	-8.74	-1.46	-6.49	-1.08
Regression
L+S	7.28	0.626	17.5	1.34	21.7	1.61	18.8	1.37	22.6	1.61	31.4	2.14
L	28.2	2.47	35.6	2.93	39.9	3.19	38.6	3.03	40.6	3.14	45.4	3.38
S	-16.3	-2.61	-13.3	-2.23	-13.0	-2.20	-14.2	-2.33	-12.8	-2.21	-9.65	-1.75

Equations

$${\mathbf{X}}_{\mathrm{inter}}:=(\mathsf{vec}(\{x_{1i}{x}_{1j}{\}}_{1\le i<j\le p}),\dots ,\mathsf{vec}(\{x_{ni}{x}_{nj}{\}}_{1\le i<j\le p}){)}^{\top },$$

$${\mathbf{z}}_i=\mathbf{a}+\mathbf{B}{\mathbf{f}}_i+{\mathbf{u}}_i,$$

$$\widehat{\bm{B}}=\big{(}{\widehat{\lambda}_{1}}^{1/2}\widehat{\bm{\xi}}_{1}\ldots,{\widehat{\lambda}_{K}}^{1/2}\widehat{\bm{\xi}}_{K}\big{)},\text{ and }\widehat{\bm{f}}_{i}=\mathrm{diag}\big{(}\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{K}\big{)}^{-1}\widehat{\bm{B}}^{\top}\bm{z}_{i},$$

$$\hat{K}:=\underset{k_{\min }\le j\le k_{\max }}{\mathrm{argmax}}{\hat{\lambda }}_j/{\hat{\lambda }}_{j+1}.$$

$$\widehat{\bm{f}}_{i}:=\bm{W}^{\top}\bm{z}_{i}/p,\text{ and }\widehat{\bm{B}}=\sum_{i=1}^{n}\bm{z}_{i}\widehat{\bm{f}}_{i}^{\top}\big{(}\sum_{i=1}^{n}\widehat{\bm{f}}_{i}\widehat{\bm{f}}_{i}^{\top}\big{)}^{-1}.$$

$$\mathbf{X}$$

$${\hat{\mathbf{B}}}_{\mathsf{inter}}$$

$${\hat{\mathbf{U}}}_{\mathsf{inter}}$$

$$y_i=g({\hat{\mathbf{f}}}_{\mathrm{inter},i},{\hat{\mathbf{u}}}_{\mathrm{inter},i})+{\epsilon }_i,$$

$$y_i=\tilde{g}({\hat{\mathbf{f}}}_{0,i},{\hat{\mathbf{f}}}_{\mathrm{inter},i},{\hat{\mathbf{u}}}_{\mathrm{inter},i})+{\epsilon }_i.$$

$${\hat{\mathbf{f}}}_{\mathrm{new}}$$

$$\displaystyle=\operatorname{diag}\big{(}\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{K}\big{)}^{-1}\widehat{\bm{B}}_{\mathsf{inter}}^{\top}\bm{x}_{\mathrm{new},\mathsf{inter}}.$$

$${\hat{\mathbf{f}}}_{\mathrm{new}}:=\frac{1}{p}{\mathbf{W}}^{\top }{\mathbf{x}}_{\mathrm{new},\mathsf{inter}}.$$

$$\widehat{\theta}_{j}=\operatorname*{arg\,min}_{\bm{\gamma}\in\mathbb{R}^{\hat{K}},\theta\in\mathbb{R}}L_{n}\big{(}\bm{y},\widehat{\bm{F}}_{\mathsf{inter}}\bm{\gamma}+\widehat{\bm{u}}_{j}\theta),$$

$$\hat{\mathbf{\theta }}\in \underset{\mathbf{\theta }\in {\mathbb{R}}^{p+K+1}}{\mathrm{argmin}}\left\{\frac{1}{n}\Vert \mathbf{y}-{\hat{\mathbf{Q}}}_{\mathsf{inter}}\mathbf{\theta }{\Vert }^2+{\gamma }_1\Vert \mathbf{\theta }{\Vert }_1\right\},$$

$$\hat{\mathbf{\theta }}\in \underset{\mathbf{\theta }\in {\mathbb{R}}^{p+K+1}}{\mathrm{argmin}}\left\{\frac{1}{n}\Vert \mathbf{y}-{\hat{\mathbf{Q}}}_{\mathsf{inter}}\mathbf{\theta }{\Vert }^2+{\gamma }_2\Vert \mathbf{\theta }{\Vert }_2^2\right\},$$

$$\widehat{\bm{\theta}}=\big{(}\widehat{\bm{Q}}_{\mathsf{inter}}^{\top}\widehat{\bm{Q}}_{\mathsf{inter}}+\gamma_{2}\bm{I}_{p+K+1}\big{)}^{-1}\widehat{\bm{Q}}_{\mathsf{inter}}^{\top}\bm{y}.$$

$${\mathbf{h}}^{(L+1)}={\mathbf{A}}^{(L+1)}\circ \sigma \circ {\mathbf{A}}^{(L)}\circ \dots \circ \sigma \circ {\mathbf{A}}^{(1)}(\mathbf{x}),$$

$${\mathbf{A}}^{(\ell )}(\mathbf{x})={\mathbf{W}}^{(\ell )}\mathbf{x}+{\mathbf{b}}^{(\ell )},\text{ for }i=1,\dots ,L,$$

$$R^{2}=1-\frac{\sum_{t=m+1}^{n_{\mathrm{total}}}\big{(}y_{t}-\widehat{y}_{t}\big{)}^{2}}{\sum_{t=m+1}^{n_{\mathrm{total}}}\big{(}y_{t}-\overline{y}_{t}\big{)}^{2}}.$$

$${\mathrm{Return}}_{it}=\sum _{p=-13}^{14}{\beta }_p{\mathrm{Day}}_{ip}+{\delta }_i+{\mu }_t+{\epsilon }_{it},$$

$$\mathrm{ERR}=\frac{1}{n_{\mathrm{new}}}\sum_{i=1}^{n_{\mathrm{new}}}I\big{(}\widehat{y}_{i}^{0}\neq y_{i}^{0}\big{)},$$

$$R^{2}=1-\frac{\sum_{t=1}^{n_{\mathrm{new}}}\big{(}y^{0}_{t}-\widehat{y}^{0}_{t}\big{)}^{2}}{\sum_{t=1}^{n_{\mathrm{new}}}\big{(}y^{0}_{t}-\bar{y}_{t}\big{)}^{2}},$$

$$L=R\times 299/1000+G\times 587/1000+B\times 114/1000.$$

Full text

Factor Augmentations,

Principal Components,

Feature Interactions,

Kernel Features,

Prediction,

, and

1 Introduction

Supervised learning has been an active area of research over decades, aiming to reveal underlying patterns in big data to enhance prediction accuracy. Previous studies primarily focus on two key areas: collecting extensive data from various sources and developing advanced algorithms to leverage the data. While substantial progress has been made in these areas, an intermediary aspect has received limited attention: the potential for enriching data features before feeding them into learning models. Note that various features can be extracted from the data and subsequently augmented to increase the prediction accuracy, provided that the augmented signals dominate the noises due to variable additions. This approach is especially useful when a systematic method is available with minimal additional effort compared to the potential gain in model performance. It is highly versatile and can be applied independently of other methods or algorithms.

Considering the pair of $(\bm{x},y)$, the response variable $y$ can be viewed as a part of the covariates $\bm{x}$ since $\bm{x}$ is collected to contribute to estimate $y$. Additionally, in practice, correlation effects often exist among observed features. Hence, there exist some latent common factors carrying the dependent structure of $\bm{x}$ and shared patterns between $\bm{x}$ and $y$. The most critical factors among $\bm{x}$ are also important to $y$. Meanwhile, since all the features are correlated with $y$, as the number of features increases to high-dimensional realms, the factors that influence $y$ become more significant. Various approaches have been explored to estimate an approximate factor model (see, e.g., Fama and French (1992); Stock and Watson (2002); Bai and Ng (2002); Bai (2003)), and factor models have proven beneficial in various contexts, enhancing variable selection and model performance both theoretically and empirically (Wang, Liu and Chen, 2019; Stock and Watson, 2002). From the methodological perspective, the Factor-Adjusted Regularized Model Selection (FarmSelect) approach proposed by Fan, Ke and Wang (2020) addresses sparse regression-related problems for model selection using latent factors to reduce variable dependence. Similarly, for prediction purposes, Zhou et al. (2023) proposes the Factor-Augmented Regularized Model for Prediction (FarmPredict) to analyze house usage using the integration of nightlight data and land planning data. Moving from linear to nonlinear prediction, Fan and Gu (2024) proposes the Factor Augmented Sparse Throughput (FAST) model that utilizes factor models for nonparametric variable selection and regression with deep ReLU neural networks.

To our knowledge, although being frequently used, factor models are so far only applied to the design matrix/tensor directly. No systematic studies have been made on the potential of information gain of such an approach that leads to systematic improvements in prediction power. On the contrary, we propose to extract nonlinear factors from transformed versions of the design matrix, like low-dimensional interactions and kernel tricks. These transformations are likely to contain different patterns and valuable information compared to the original data. Augmenting features with factors derived from these transformations elevates the feature space to higher dimensions, enhancing approximation power. This approach not only produces a better interpretation of the data but also aids variable selection in high-dimensional cases by decreasing feature correlations. Meanwhile, since only several more features are added (say, 5 to 10) via principal component analysis, the variance they bring about is almost negligible compared to the number of features in the problems. These two aspects together lead to the benefits of the proposed method in statistical prediction, as to be demonstrated.

Building on this methodological foundation, we investigate its performance in a real-world financial prediction setting involving Chinese news text data. Natural Language Processing (NLP) is gaining increasing prominence nowadays in diverse applications.Recent studies have highlighted the value of textual information in financial modeling and decision-making (Zhou, Fan and Xue, 2024; Ke, Kelly and Xiu, 2019; Goldstein, Spatt and Ye, 2021; Wang et al., 2024; Loughran and McDonald, 2016). These works primarily concentrate on the collection and preprocessing of informative text data, the transformation of textual content into structured, machine-readable inputs, and the development of predictive models to address tasks such as asset pricing and risk assessment. In many cases, especially in financial studies, the data is tedious to collect and expensive to buy. Therefore, it is very useful and helpful to be able to extract some latent and informative features, preferably in an easy way that does not require much additional computation, and achieve better estimation performance.

To present that, we leverage the large-scale Chinese financial news dataset compiled by Zhou, Fan and Xue (2024) and study the stock market problems. The original paper made great efforts on collecting and processing the news data, after which the authors applied the Factor-Adjusted Regularized Model Selection (Fan, Ke and Wang, 2020) with linear regression on the stock return, and analyzed thoroughly, focusing on the sentiment scores and portfolio returns, from a finance perspective. In contrast, our focus lies in demonstrating the simple and general statistical augmentation technique of extracting latent non-linear factors from the high-dimensional embedded textual data, and showing how such augmentations can enhance stock return prediction in financial market studies. We apply this approach across five widely used machine learning algorithms—Lasso, Ridge Regression, Random Forests, Gradient Boosted Trees, and Neural Networks—and consistently observe improved performance, underscoring the robustness and flexibility of the method. Building on these estimation results, we further conduct an event study and a portfolio analysis to illustrate the practical value of the proposed feature augmentation framework in the context of financial investment.

Furthermore, to highlight the versatility of the proposed method, we supplement our primary case study with additional empirical evaluations spanning a variety of domains and problem. These supplementary experiments confirm that the augmentation framework consistently enhances predictive accuracy in diverse applications, reinforcing its general applicability in high-dimensional learning contexts. In addition, the results offer practical guidance for selecting appropriate transformation methods tailored to different problem settings.

The remainder of the paper is organized as follows. Section 2 introduces the dataset of Chinese financial news and the associated stock returns. Section 3 outlines the proposed methodology, including matrix transformations, factor estimation, feature augmentation, variable screening, and the learning algorithms. Section 4 presents the empirical analysis of the Chinese News data, and as a supplement, Section 5 provides additional empirical studies across diverse datasets, illustrating the versatility and effectiveness of the method. Finally, Section 6 concludes the paper and discusses potential directions for future research.

Some notations used are as follows. Bold uppercase letters, bold lowercase letters, and unbold lowercase letters represent matrices, vectors, and scalars, respectively. Letters with hats are estimators. $\bm{I}_{n}\in\mathbb{R}^{n\times n}$ is the identity matrix, and $\mathbf{1}_{n}\in\mathbb{R}^{n}$ represents the all-one vector.

2 Data Description and Pre-processing

We utilize the large-scale Chinese financial news dataset constructed by Zhou, Fan and Xue (2024), collected from Sina Finance, one of the leading financial news platforms in China. Detailed information about the dataset can be found therein; we provide a brief summary and pre-processing procedure below.

The dataset spans from 2000 to 2019 and includes at most 300 articles per day, resulting in a final corpus of approximately 914,000 articles. Each news article was crawled along with its associated publication time and corresponding stock information, and paired with the effective beta-adjusted return of the linked stock on the day of publication, which serves as the target response variable in our study. More specifically, the matching stock is decided based on a combination of html and article content, and the time range of daily return is set as the close-to-close return covering the article’s publish time. The title and main text of each article are segmented into lists of words and phrases using the Jieba Python package (Sun, 2017), resulting in a bag of words representation for each article and collecting a vocabulary of approximately 1,181,000 unique words and phrases. Therefore, for each article, a vector of the same length ($1,181,000$) is generated, with each element being the word count of the corresponding word that appears in the article.

We adopt a six-month-ahead rolling window estimation strategy, whereby for each window, ten years of data are used for model training, followed by a six-month testing period. Given the high dimensionality and sparsity of the text data, we implement a two-step dimension reduction procedure in each training window. First, we retain only the 3,000 most frequent words in the training set. Then, we apply a screening technique, as detailed in Section 3.4, to further reduce the feature set to a comprehensive subset of 300 words and phrases for subsequent modeling.

3 Methodology

In this section, we present the general framework of feature augmentation with nonlinear latent factors and specify the application on the news text dataset.

3.1 Data Transformation

One often applies nonlinear transforms to create additional features and results in overparametrization and requires some dimensionality reduction. The factor model extracts information from a large matrix and creates low-dimensional features. Unlike prior literature that applies the factor model directly to the data matrix, we extend it to the simple transformations of the data. Since the process of nonlinear transformations may reveal additional nonlinear features, it is expected that extracted factors can capture distinct aspects of the data compared to those obtained directly from the original data. There are many kinds of nonlinear data transformations. We showcase three methods: interaction matrices, kernel matrices, and the intermediate outputs of neural networks.

**Pairwise interactions: ** Let $\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{n})^{\top}$ be the original feature matrix with $\bm{x}_{\!i}\!\!=\!(\!x_{i1},\!\ldots\!,x_{ip})^{\!\top}$. Then we augment the original feature matrix $\bm{X}$ with $\bm{X}_{\mathrm{inter}}\in\mathbb{R}^{n\times p(p+1)/2}$ which is defined as

[TABLE]

where $\mathsf{vec}(\cdot)$ is the vectorization operator. This transformation reveals latent semantic relationships between words. For example, while the individual counts of "artificial" and "intelligence" may not be highly informative alone, their co-occurrence can strongly indicate a specific topic, and factors extracted therein enrich the feature space to capture concept combinations that are not visible through single-word frequencies alone. Interaction terms are also useful in many other problems, such as the study of gene synergy, protein interactions, image pixels interplay, or word meanings, among others (Wu et al., 2016; Balli and S$\bm{o}$rensen, 2013; Zhang et al., 2021). Thus, it is reasonable to expect that factors derived from the interaction matrices are able to provide additional information, enhancing the capabilities of machine learning models.

Kernel methods: Kernel methods allow us to examine higher order interactions through a similarity kernel. Gaussian and polynomial kernels are considered herein, whose similarity matrix are denoted respectively by $\bm{X}_{\mathrm{rbf}}$ and $\bm{X}_{\mathrm{poly}}$. Since the number of samples $n$ is extremely large in each training window (approximately 300,000 in our main study), we randomly select $n_{0}$ columns, resulting in a reduced kernel matrix of size $n\times n_{0}$, to learn latent factors. This significantly reduces the computational burden while preserving the essential geometry of the feature space induced by the kernel.

By implicitly projecting the original feature vectors into richer, nonlinear spaces, these methods allow us to uncover complex relationships that are not visible in the raw data. In the context of Chinese news text data, kernel-based transformations can capture nuanced semantic patterns—for example, articles using different expressions like “policy easing” and “interest rate cuts” may be recognized as discussing similar monetary events. Such nonlinear representations are capable of identifying topic clusters that transcend surface word overlap, capturing sentiment shifts expressed through unusual or indirect phrasing, and modeling higher-order interactions between terms that indicate subtle narrative themes. These latent semantic structures enhance the model’s ability to generalize and improve predictive accuracy beyond what is achievable through linear representations alone.

Neural networks: As for neural networks, we draw the last hidden layer of an FNN applied to $\bm{X}$ and obtain ${\bm{X}}_{\mathrm{fnn}}$. Shallow neural networks with one or two layers of convolution with ReLU activation with a wide enough last hidden layer output is enough to be combined by PCA to derive the factors. In the context of news text data for stock prediction, neural networks can capture complex interactions between terms—such as "interest rates," "central bank," and "market volatility"—that individually provide limited information but together reveal important market signals. By modeling such nonlinear word combinations, neural networks help generate richer feature representations that better align with the underlying drivers of stock returns. Moreover, as the method is widely applicable, when considering image data, we may also use convolutional neural networks (CNN), and extract ${\bm{X}}_{\mathrm{cnn}}$ from its last hidden layer. Considering the widespread empirical success of neural networks in uncovering intricate and nonlinear data structures, this transformation is expected to produce powerful and informative features that complement the original representation.

3.2 Factor Model

Suppose that there are $n$ samples $\{\bm{z}_{i}\}^{n}_{i=1}$, $\bm{z}_{i}\in\mathbb{R}^{p}$ generated from the factor model

[TABLE]

where $\bm{B}\in\mathbb{R}^{p\times K}$ is a factor loading matrix, and the latent factors $\bm{f}_{i}\in\mathbb{R}^{K}$ are zero-mean random variables, uncorrelated with the idiosyncratic components ${\bm{u}}_{i}\in\mathbb{R}^{p}$. The intercept $\bm{a}$ is estimated by the average of samples, i.e. $\widehat{\bm{a}}=\overline{\bm{z}}=\sum_{i=1}^{n}\bm{z}_{i}/n$. Without loss of generality, we assume that $\widehat{\bm{a}}=\mathbf{0}$. Denote the design matrix by $\bm{Z}=(\bm{z}_{1},\ldots,\bm{z}_{n})^{\top}\in\mathbb{R}^{n\times p}$.

3.2.1 Factor Estimation via PCA

A standard way of extracting factors from a matrix is to apply PCA to the whole data matrix. For identifiability purposes, we assume that $\operatorname{cov}(\bm{f}_{i})=\bm{I}_{K}$ and ${\bm{B}}^{\top}{\bm{B}}$ is diagonal. By applying PCA on the sample covariance of $\left\{\bm{z}_{i}\right\}_{i=1}^{n}$, the factors can be estimated by (Bai, 2003)

[TABLE]

where $\widehat{\lambda}_{1},\ldots,\widehat{\lambda}_{K}$ and $\widehat{\bm{\xi}}_{1},\ldots,\widehat{\bm{\xi}}_{K}$ are the top $K$ eigenvalues of the sample covariance matrix and their associated eigenvectors. Let $\widehat{\bm{F}}:=(\widehat{\bm{f}}_{1},\ldots,\widehat{\bm{f}}_{n})^{\top}$ be the matrix of estimated latent factors. This is the same as setting the columns of $\widehat{\bm{F}}/\sqrt{n}$ as the eigenvectors of $\bm{Z}\bm{Z}^{\top}$ corresponding to the top $K$ eigenvalues and letting $\widehat{\bm{B}}=n^{-1}\bm{Z}^{\top}\widehat{\bm{F}}$.

There are several methods to determine $K$, the number of factors, such as the adjusted eigenvalues thresholding method (Fan, Guo and Zheng, 2022) and the eigenvalue ratio (eigen-ratio) estimator (Tsai et al., 2009; Lam and Yao, 2012; Ahn and Horenstein, 2013), to list just a few. Note that our down stream prediction task is not very sensitive to the slight over estimation of $K$. Here, we adopt the eigen-ratio method with $k_{\max}=\lfloor(p\wedge n)/3\rfloor$ suggested by Ahn and Horenstein (2013) and $k_{\min}=\operatorname{max}\{\lfloor(p\wedge n)/10\rfloor,2\}$:

[TABLE]

Note that the lower and upper bound is heuristic, and it may not be essential in some cases where some conditions are met (Zhang, Zhou and Wang, 2022). Also, if $p\wedge n$ is large, we can set $k_{\min}$ to, for example, $5$, just to avoid degenerate outcomes (Wainwright, 2019; Marchenko and Pastur, 1967).

3.2.2 Factor Estimation via Diversified Projection

Though powerful and popular over the years, the principal component estimation of factors has potential drawbacks. One big issue is that it is computationally expensive for high-dimensional features and also requires a large sample size to get accurate estimates. This computational scalability issue becomes very severe in text data, where the dimension of the input matrix can be very large. To sidestep this problem, Fan and Liao (2022) propose to learn latent factors using diversified projections by weighted averages with predetermined weights. Later on, Fan and Gu (2024) propose a data-driven pre-trained weight matrix, calculated by applying PCA to a separate set of $n^{\prime}$ examples. It is formally derived there that $n^{\prime}$ can be much smaller than the original sample size $n$: it suffices to have $n^{\prime}\asymp K^{2}\log p$. More specifically, let $\{\bm{z}^{\prime}_{i}\}_{i=1}^{n^{\prime}}\subseteq\mathbb{R}^{p}$ be another $n^{\prime}$ samples that are independent with $\{\bm{z}_{i}\}_{i=1}^{n}$. Then PCA is applied to get the top-$K^{\prime}$ eigenvectors $\widehat{\bm{\xi}}^{\prime}_{1},\ldots,\widehat{\bm{\xi}}^{\prime}_{K^{\prime}}$ of the sample covariance matrix of $\{\bm{z}^{\prime}_{i}\}_{i=1}^{n^{\prime}}$, where $K^{\prime}\geq K$. Choose the diversified weight matrix as $\bm{W}\!=\!\sqrt{p}\,(\widehat{\bm{\xi}}^{\prime}_{1},\ldots,\widehat{\bm{\xi}}^{\prime}_{K^{\prime}})$. With the pre-training in hand, we can proceed as if the factors are known to us, and the loading matrix $\bm{B}$ can thereafter be estimated by least squares:

[TABLE]

Besides the aforementioned advantages, diversified projection offers two additional benefits for real-world applications, especially in problems like the studied textual data example, where both the input dimension and sample sizes are very large. First, when the dimension $p$ is large, using a small subset of data with a reduced sample size $n^{\prime}$ can significantly accelerate the estimation of the number of factors, since the nonzero eigenvalues of $\bm{Z}\bm{Z}^{\top}$ and $\bm{Z}^{\top}\bm{Z}$ are identical, and their corresponding eigenvectors are related through simple linear transformations. Second, when the sample size $n$ (or $n_{0}$) is ultra-large, applying PCA to the full kernel matrix becomes computationally infeasible, whereas diversified projection remains computationally tractable and effective.

3.3 Feature augmentation

Denote by $\widehat{\bm{F}}_{0}$, $\widehat{\bm{F}}_{\mathsf{inter}}$, $\widehat{\bm{F}}_{\mathsf{poly}}$, $\widehat{\bm{F}}_{\mathsf{rbf}}$, and $\widehat{\bm{F}}_{\mathsf{fnn}}$the latent factors extracted respectively from $\bm{X}$ and the transformed matrices $\bm{X}_{\mathsf{inter}}$, $\bm{X}_{\mathsf{poly}}$, $\bm{X}_{\mathsf{rbf}}$, and $\bm{X}_{\mathsf{fnn}}$introduced in Section 3.1. We demean $\bm{X}$ and all the transformed matrices before further processing. In what follows, we take $\widehat{\bm{F}}_{\mathsf{inter}}$ as an example, while the method can also be applied to other factors.

We augment the original feature space $\bm{X}$ by adding $\widehat{\bm{F}}_{\mathsf{inter}}$. To decorrelate $\bm{X}$ from $\widehat{\bm{F}}_{\mathsf{inter}}$ while retaining all the information, we project $\bm{X}$ onto the subspace spanned by the factors $\widehat{\bm{F}}_{\mathsf{inter}}$, i.e. fit the model

[TABLE]

and obtain the estimates

[TABLE]

The subspace spanned by the new independent variables $(\widehat{\bm{F}}_{\mathsf{inter}},\widehat{\bm{U}}_{\mathsf{inter}})\in\mathbb{R}^{n\times(K+p)}$ is the same as that of $(\widehat{\bm{F}}_{\mathsf{inter}},\bm{X})$. Note that when the factors are estimated by PCA, we have $\widehat{\bm{F}}_{\mathsf{inter}}^{\top}\widehat{\bm{F}}_{\mathsf{inter}}=\bm{I}$, and therefore $\widehat{\bm{U}}_{\mathsf{inter}}=[\bm{I}_{n}-\widehat{\bm{F}}_{\mathsf{inter}}\widehat{\bm{F}}_{\mathsf{inter}}^{\top}]\bm{X}$.

Let $y_{i}$ be the variable to be predicted based on $\bm{x}_{i}$. Then, the general regression model takes the form

[TABLE]

where $\varepsilon_{i}$ is the idiosyncratic noise. Any statistical machine learning model can be employed here, such as Lasso Regression, Ridge Regression, Random Forest, Gradient Boosted Tree, and Neural Networks. Moreover, different factors may be combined together to boost performance. Specifically, one can add $\widehat{\bm{F}}_{0}$ to $(\widehat{\bm{F}}_{\mathsf{inter}},\widehat{\bm{U}}_{\mathsf{inter}})$ for further augmentation, which gives

[TABLE]

Note that we expand the original features to use principal component (factor) directions. This significantly reduces possible modeling biases while not dramatically increasing the number of variables.

Under the proposed model, the prediction for a given new feature vector $\bm{x}_{\mathrm{new}}$ consists of two steps. First, we compute the interactions for $\bm{x}_{\mathrm{new}}$ and denote it by $\bm{x}_{\mathrm{new}\!,\mathsf{inter}}$. With the estimated factor loading matrix $\widehat{\bm{B}}_{\mathsf{inter}}$, estimating the latent factors corresponding to $\bm{x}_{\mathrm{new}}$ can be equivalently viewed as regressing $\bm{x}_{\mathrm{new},\mathsf{inter}}$ on $\widehat{\bm{B}}_{\mathsf{inter}}$ based on (1), which gives

[TABLE]

Alternatively, if the factors are estimated by diversified projection, we have

[TABLE]

Thus, by (2), the new idiosyncratic component is $\widehat{\bm{u}}_{\mathrm{new}}=\bm{x}_{\mathrm{new}}-\widehat{\bm{B}}_{\mathsf{inter}}\widehat{\bm{f}}_{\mathrm{new}}$, and accordingly, $\widehat{y}_{\mathrm{new}}=\widehat{g}(\widehat{\bm{f}}_{\mathrm{new}},\widehat{\bm{u}}_{\mathrm{new}})$, where $\widehat{g}$ is the fitted model.

3.4 Decorrelated Variable Screening

To improve computational efficiency, conditional correlation screening can be applied to $\widehat{\bm{u}}_{\mathsf{inter}}$ to expeditiously select its useful components if the dimension $p$ is ultrahigh. The screening process involves reducing the dimension by only selecting the features that are the most strongly correlated with the response variable. This technique was first proposed by Fan and Lv (2008) and later improved by Fan, Ke and Wang (2020).With the latent factors given, features of the residual are ranked according to their conditional marginal contributions. Let $L_{n}(\bm{y},\widehat{\bm{y}})$ be an empirical convex loss function under consideration for model (5).The augmented marginal regression considers the following low-dimensional fit:

[TABLE]

where $\widehat{\bm{F}}_{\mathsf{inter}}$ can be replaced by any latent factors introduced before, and $\widehat{\bm{u}}_{j}$ is the $j$-th column of $\widehat{\bm{U}}$ that is standardized. Then \big{\{}\widehat{\theta}_{j}\big{\}}_{j=1}^{p} are sorted in their absolute values, and the screening deletes the variables corresponding to the smallest marginal marginal contributions. The selected $\widehat{\bm{u}}_{j}$ and the factors are then fed into a statistical machine learning algorithm to train a model. This procedure can also be applied to the original $\bm{X}$, which learns the common factors of $\bm{X}$ and important $\widehat{\bm{u}}_{j}$ for prediction.

3.5 Statistical Machine Learning Methods

Five statistical machining learn methods are considered as examples to verify if factor augmentation can improve the accuracy of learning algorithms.

Lasso Regression: Lasso is a popular regression analysis method for its convexity and ability to produce sparse estimators. It imposes $\ell_{1}$-norm regularization on linear regression. Take the pairwise interaction matrix for example, let $\widehat{\bm{Q}}_{\mathsf{inter}}:=(\mathbf{1}_{n},\widehat{\bm{F}}_{\mathsf{inter}},\widehat{\bm{U}}_{\mathsf{inter}})\in\mathbb{R}^{n\times(p+K+1)}$ and $\bm{\theta}\in\mathbb{R}^{p+K+1}$. Lasso imposes $\ell_{1}$-norm regularization on linear regression and the model yields

[TABLE]

where $\bm{y}=\left(y_{1},\ldots,y_{n}\right)^{\top}$, and $\gamma_{1}$ is the tuning parameter.

Ridge Regression: Similar to Lasso, Ridge adds $\ell_{2}$-norm penalization on linear regression. However, Ridge does not produce sparsity. Instead, it deals with collinearity issues via

[TABLE]

where $\gamma_{2}$ is the tuning parameter. It admits the explicit form

[TABLE]

Random Forest (RF): Random forest is an ensemble learning method used for both classification and regression problems (Breiman, 2001). It involves aggregating the results of multiple independent decision trees, each of which uses tree representation to solve problems. Random forest uses bootstrap to generate the collection of decision trees and uses a randomly selected subset of predictors at each split to construct each tree. The outcome of RF is the majority vote of all the trees for classification and average of all tree predictions.

Gradient Boosted Trees (GBT): Gradient Boosted Trees are also an ensembling method that iteratively performs regression or classification by fitting classification and regression trees to the pseudo residuals of the previous fit and aggregate the outcomes of the prediciton. We omit the detail.

Neural Networks: Neural networks have achieved tremendous success in recent years thanks to the availability of big data. Fully-connected Neural Networks (FNN) is one of the most basic learning models. For given layer widths $n_{0},\ldots,n_{L+1}$ and a simple nonlinear function $\sigma$, the architecture of FNN with depth $L$ takes the form

[TABLE]

where $\circ$ denotes the composition of two functions, and $\mathbf{A}^{\ell}:\mathbb{R}^{n_{\ell-1}}\rightarrow\mathbb{R}^{n_{\ell}}$ is an affine function defined by

[TABLE]

with $\mathbf{W}^{(\ell)}$ and $\mathbf{b}^{(\ell)}$ being the weight matrix and the intercept, respectively, associated with the $\ell$-th layer. Common choices for the activation function $\sigma$ are $\operatorname{ReLU}(t)=\max\{0,\,t\}$ and $\operatorname{tanh}(t)$ among others. Besides, to prevent overfitting, Hinton et al. (2012) proposed to randomly drop out subsets of hidden units. We only consider shallow FNN for classifications, for example, two layers of ReLU activation followed by Dropout layers, respectively.

4 Empirical Analysis on Chinese Text Data

We apply the aforementioned feature augmentation methods to the Chinese news data to predict stock returns and showcase how we can benefit from the proposed nonlinear factors in diverse machine learning algorithms. All experiments are done on a 4-rack Intel compute cluster with NVIDIA A100 GPUs.

4.1 Number of Factors

Before presenting the results, we discuss the way of determining the number of factors $K$ for each transformation of the Chinese text data. We use the diversified projection method, illustrated in Section 3.2.2, and set the pre-training set size to be $n^{\prime}=1000$ for each training window. We estimate $K$ based on the pertaining set data using the eigen-ratio method illustrated at the end sof Section 3.2.1. Figure 1 displays the scree plots of the top 10 to 30 eigenvalues, eigenvectors, and eigen-ratios of the covariance of $\bm{X}$ (the original matrix) and $\bm{X}_{\mathrm{inter}}$ (the interaction matrix), of the first training window. By the eigen-ratio method, the numbers of factors are chosen as 12 and 15, respectively, for $\bm{F}_{0}$ and $\bm{F}_{\mathrm{inter}}$. The same procedure is performed for all windows and models. With the number of factors determined, we present the results and findings as follows.

4.2 Tuning and Testing

Five machine learning techniques are employed for the classification problems, including Lasso, Ridge, RF, GBT, and FNN. The hyperparameters for all the algorithms and factor models are selected by cross-validation (CV) with predefined grids. To be more specific, the maximum number of iterations to converge in Lasso and Ridge, the number of trees in RF, and the number of boosting rounds in GBT are all set to 100. The tuning parameter $\gamma_{1}$ for Lasso is chosen from $15$ numbers from $10^{-10}$ to $10^{-3}$ with even space on a log scale, and $\gamma_{2}$ for Ridge is chosen as a geometric sequence from $10$ numbers from $10^{-3}$ to $10^{3}$. In RF, the maximum depth of trees and the minimum sample number required to split an internal node are both chosen from $(4,8,16)$. In GBT, the maximum depth of trees is chosen from the same set, and the learning rate is chosen from $(0.2,0.02)$.

The performance is evaluated using the out-of-sample $R^{2}$. Let $(\bm{x}_{t},y_{t})$, $t=1,\ldots,n_{\mathrm{total}}$ be the time series data, where $n_{\mathrm{total}}$ is the total number of time points. Note that $\bm{x}_{t}$ are features known before time $t$. Consider a rolling window prediction with a window size $m$. Each window consists of a consecutive observation of $m$ data points $(\bm{x}_{t-m},y_{t-m}),\ldots,(\bm{x}_{t-1},y_{t-1})$, and they are taken as the training set to predict $y_{t}$ based on $\bm{x}_{t}$. To save the computation, the model is updated only once every $h$ predictions so that $y_{t},\ldots,y_{t+h-1}$ are predicted based on $\bm{x}_{t},\ldots,\bm{x}_{t+h-1}$, using the same trained model. After each group of predictions, the training window is shifted forward by $h$ time points to retrain the model, and the next $h$ out-of-sample time points are predicted. Let $\widehat{y}_{t}$, $t=m+1,\ldots,n_{\mathrm{total}}$ be the prediction of rolling windows. Let $\overline{y}_{t}$ be the sample mean of the training set corresponding to the data point $t$. For example, $\overline{y}_{m+i}=\sum_{j=1}^{m}y_{j}/m$ for $i=1,\ldots,h$. Then, we can define the out-of-sample $R^{2}$ in the same way as above, which is

[TABLE]

4.3 Prediction Results

We apply the proposed feature augmentation methods to the Chinese news text data for the purpose of predicting the returns of associated stocks. The primary objective is to assess the extent to which stock return prediction can benefit from the incorporation of augmented features. To ensure a fair comparison, all experimental settings are held constant except for one aspect: whether or not factor-based augmentations are included. Accordingly, we establish benchmark models that are applied directly to the original feature matrix $\bm{X}$ and compare their performance against augmentations with the proposed factors. We consider five types of augmentation factors—$\widehat{\bm{F}}_{\mathsf{inter}}$, $\widehat{\bm{F}}_{\mathsf{poly}}$, $\widehat{\bm{F}}_{\mathsf{rbf}}$, and $\widehat{\bm{F}}_{\mathsf{fnn}}$—as introduced in Section 3.1. Hereafter, when we talk about $(\widehat{\bm{F}},\widehat{\bm{U}})$ for a factor $\widehat{\bm{F}}$, the matrix $\widehat{\bm{U}}$ is always the residual of $\bm{X}$ on $\widehat{\bm{F}}$, and we will not specify the corresponding factor type for simplicity.

All experiments are repeated 20 times to reduce the influence of randomness in selecting the subsample of size 1000 to create different diversified projections, and the average results are reported. Figure 2 demonstrates the relative performance of the feature augmentation methods. All the out-of-sample $R^{2}$ are divided by $R^{2}(\bm{X})$, the out-of-sample $R^{2}$ of the benchmark model. Greater values indicate better performance. The blue bars are for $(\widehat{\bm{F}}_{0},\widehat{\bm{U}})$. For each of the newly proposed factors $\widehat{\bm{F}}$, we consider adding factors $\widehat{\bm{F}}_{0}$ to further augment the feature space $(\widehat{\bm{F}},\widehat{\bm{U}})$. Two ways of aggregation are carried out. One is to directly use $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widehat{\bm{U}})$ for prediction, and the other is to decorrelate $\widehat{\bm{F}}_{0}$ from $\widehat{\bm{U}}$ by consider $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widetilde{\bm{U}})$, where $\widetilde{\bm{U}}:=[\bm{I}_{n}-\widehat{\bm{F}}_{0}(\widehat{\bm{F}}_{0}^{\top}\widehat{\bm{F}}_{0})^{-1}\widehat{\bm{F}}^{\top}]\widehat{\bm{U}}.$ These two methods are equivalent under linear regression but may yield different results under other models. In Figure 2, we present the best outcomes among the three augmentations $(\widehat{\bm{F}},\widehat{\bm{U}})$, $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widehat{\bm{U}})$, and $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widetilde{\bm{U}})$.

The results consistently show that feature augmentation with latent factors enhances predictive performance relative to the benchmark. In many cases, nonlinear factors perform comparably to or significantly better than the linear factor $\widehat{\bm{F}}_{0}$. The relative gains vary across different machine learning models. For instance, under the simpler models — Lasso, Ridge, and RF — the neural network-based factor $\widehat{\bm{F}}_{\mathrm{fnn}}$ outperforms all other types, underscoring the strength of neural networks in capturing complex semantic information from text. Note that the FNN factor actually uses some information about the response variable (supervised), while all other methods do not (unsupervised). This explains why FNN outperforms and suggests improvements of other methods using supervised feature augmentation through, for example, sure independence screening of Fan and Lv (2008). This advantage diminishes under more sophisticated models like GBT and FNN, where other factor types, nevertheless, achieve superior performance. This observation suggests that while some of the latent information captured by $\widehat{\bm{F}}_{\mathrm{fnn}}$ may be more or less redundant with what is learned by complex models, other augmented factors encode additional predictive structure not readily extracted by the base learners. Notably, across all five models, the Gaussian kernel factor $\widehat{\bm{F}}_{\mathrm{rbf}}$ consistently ranks among the top performers, highlighting the effectiveness of extracting nonlinear latent representations through kernel transformations for this problem.

The proposed augmentation method proves useful not only in regression settings but also in classification tasks, such as sentiment estimation. In the following two subsections, we further examine the stock market by defining sentiment scores, which are then used to conduct event studies and portfolio analyses. These evaluations serve as supplementary assessments of the estimation performance of the proposed feature augmentation methods. The goal of these experiments is to evaluate the effectiveness of the augmentation strategies when applied to news text data in the context of financial investment. To this end, we compare the performance of estimations based on factors derived from various transformations, following the financial market analysis framework established in Zhou, Fan and Xue (2024).

4.4 Event Study

Recall that our dependent variable is the beta-adjusted return of a stock on the day of the associated news publication. The sign of the return can serve as a proxy for the sentiment conveyed by the news regarding the corresponding stock. Based on these sentiment scores, we conduct an event study to assess how individual stocks exhibit return responses over time in reaction to the identified sentiment weeks before and after the event. To illustrate the flexibility of the proposed feature augmentation framework, we present two approaches for generating sentiment scores, both leveraging the augmented features.

First, as described in the previous subsection, we have estimated continuous-valued returns using regressions based on different feature sets. To ensure consistency across rolling windows, we re-center the predicted values by subtracting the mean of the training data (which is very close to zero) and adding 0.5, so that the output remains interpretable as a centered score. Due to space constraints, we report results only for the score estimations obtained using Random Forest. Alternatively, since the event study focuses on the direction of stock returns, we can adopt a classification-based approach that simplifies the estimation task and enhances robustness. Specifically, we dichotomize the target variable and, as an example, train a three-layer feedforward neural network (FNN) with two hidden layers of widths 16 and 4, each followed by a dropout layer with a rate of 0.2. The model outputs the estimated probability of a positive return, which we interpret as a sentiment score. This approach is less sensitive to noise and outliers and aligns naturally with the binary nature of sentiment. For both methods, we compare sentiment scores derived from six different feature sets: (i) the original feature matrix $\bm{X}$; (ii) the linear factor and its associated idiosyncratic component $(\widehat{\bm{F}}_{0},\widehat{\bm{U}})$; and (iii–vi) four sets of nonlinear factors, each combined with the linear factor and decorrelated idiosyncratic component: $(\widehat{\bm{F}}_{\mathrm{inter}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, $(\widehat{\bm{F}}_{\mathrm{rbf}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, $(\widehat{\bm{F}}_{\mathrm{fnn}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, and $(\widehat{\bm{F}}_{\mathrm{poly}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$.

Based on these sentiment scores, we conduct an event study to examine whether individual stocks exhibit significant return responses over time in reaction to the identified sentiment. Each news publication is regarded as an “event”, with the sign of the sentiment score serving as a proxy for the sentiment of the news—positive scores indicate favorable news, while negative scores suggest adverse sentiment. The magnitude of the scores indicates the extent of positive or negative. We examine how individual stock returns respond to such sentiment signals. To mitigate the influence of noise, we only focus on the extreme events with the top 5% of positive or negative scores, as determined by the absolute magnitude of their fitted returns.

For positive news, we define an event for stock $i$ on day $t$ if there are news articles about stock $i$ published between the market close (3:00 pm) of day $(t{-}1)$ and the market close of day $t$, and its associated score falls within the top 5% quantile of all positive scores. If there is more than one news article about stock $i$ on day $t$, we take the average of the scores. We then estimate the following event-study regression:

[TABLE]

where $\mathrm{Return}_{it}$ denotes the realized beta-adjusted return of stock $i$ on day $t$; $\mathrm{Day}_{ip}$ is an indicator of $p$ days after the event; and $\delta_{i}$ and $\mu_{t}$ are stock and day fixed effects, respectively, to account for unobserved heterogeneity. An analogous regression is conducted for negative news events.

Figure 3 presents the results of the event-study regressions for positive and negative news events during the period 2015–2019. The top two panels correspond to sentiment scores obtained via Random Forest regression, while the bottom two are based on scores generated through binary classification using an FNN. The findings are consistent across both methods. Scores derived from feature augmentation yield sharper event curves compared to the baseline, where scores are computed directly from the original feature matrix $\bm{X}$ without augmentation. This suggests that the augmented features provide a more accurate estimation of sentiment, particularly for extreme positive and negative events. For the top 5% of positive sentiment cases, beta-adjusted returns—as captured by the estimated coefficients $\beta_{p}$—begin to rise approximately seven days before the event, accelerate two days prior, and peak on the day of the news release. One possible cause is that, due to possible new leakage or anticipation, investors start to buy stocks before the positive news. In contrast, negative sentiment events trigger return responses that are more concentrated around the event day, with generally lower magnitudes. This asymmetry suggests that negative news has a more limited impact on stock prices than positive news in the Chinese market. This is a consequence of short-sale restrictions, which limit investors’ ability to profit from negative information. Even if investors got negative news in the Chinese market, they can not easily make profits due to the short constraints. Shareholders gradually hear the news and add selling pressure a few days after the event. This behavior contrasts with that observed in more liberalized markets such as the United States.

4.5 Portfolio Performance

We further construct stock portfolios based on the scores from 2015 to 2019 and evaluate the effectiveness of the proposed feature augmentation methods. We compare the performance of different augmentation strategies in a practical investment setting and adopt a long-short portfolio strategy: among news published each day, buy the top 50 stocks with the highest sentiment scores and sell the bottom 50 with the lowest scores. If fewer than 50 stocks have scores larger (or smaller) than 0.5 on a given day, the unallocated capital is held as cash and earns no interest. We consider the value-weighted (VW) strategies, where portfolio weights are proportional to the stocks’ total market capitalization on the preceding trading day. Additionally, we account for the daily transaction costs in the Chinese equity market. Transaction costs are applied each day when the portfolio changes. These include stamp duty, transfer fees, and trading commissions, following typical practices in the Chinese stock market. We assume a total transaction cost of 13 basis points for each round-trip trade (buy and sell). Portfolio returns are adjusted for turnover to reflect trading costs accurately.

Figure 4 presents the results of the portfolio analysis based on score estimations obtained using binary classification with FNN. The top two panels show results based on regression-based score estimators, while the bottom two correspond to scores derived from binary classification. The panels on the left display portfolio performance without trading costs, whereas the right panels incorporate transaction costs. Across all settings, portfolios constructed using scores derived from feature augmentation consistently outperform the baseline model that uses only the original feature matrix. In particular, the factors extracted via the FNN-based transformation, $(\widehat{\bm{F}}_{\mathrm{fnn}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, yield the best performance, in line with the findings reported in Section 4.3.

Table 1 reports the detailed performance of the combined long-short portfolio, as well as its long and short components. We focus on the setting that incorporates the transaction costs. It shows that scores estimated from augmented features yield higher annualized percentage returns (APR) and Sharpe ratios compared to the baseline. Furthermore, factors extracted from nonlinear transformations of the original design matrix enhance the informativeness of the scores and further improve portfolio performance. Consistent with the results in Figure 3, the majority of portfolio gains are realized from the long leg rather than the short leg. Actually, the returns gained from the short legs are negative, and this is because of the way we construct the portfolio and that we consider the transaction cost.

While we consider two score estimation methods—Random Forest regression and FNN-based binary classification—the classification-based scores generally lead to superior portfolio performance. That is likely due to the fact that the regression-based estimates tend to be less robust and more sensitive to noise, and thus tend to underperform compared to classification-based sentiment estimates. However, we observe in Table 1 that regression-based scores using Random Forest with feature sets such as $(\widehat{\bm{F}}_{\mathrm{inter}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, $(\widehat{\bm{F}}_{\mathrm{poly}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$, and $(\widehat{\bm{F}}_{\mathrm{fnn}},\widehat{\bm{F}}_{0},\widetilde{\bm{U}})$ actually outperform their classification-based counterparts. This highlights the value of incorporating nonlinear feature transformations, which can significantly strengthen the predictive capacity of the feature set.

5 More Empirical Analysis on Diverse Datasets

We emphasize that the proposed feature augmentation framework is broadly applicable across diverse problem domains and learning algorithms. To demonstrate its general effectiveness in improving estimation performance, we complement the main study on Chinese news text data with a series of extensive experiments spanning both classification and regression tasks. These tasks are motivated by real-world applications in image recognition, biology, finance, etc. For each example and each learning method, we evaluate the performance of the proposed methods and compare them to those without feature augmentations. Besides, typical performances of various factors across datasets and methods are also analyzed. The objective of these experiments is twofold: to assess whether feature augmentation consistently enhances estimation accuracy across settings, and to gain better insights into which types of factors perform best under different data structures and learning contexts.

To accommodate the characteristics of these auxiliary datasets, we introduce several adjustments to the experimental setup. For image-based problems, we incorporate the factor $\bm{F}_{\mathrm{cnn}}$, extracted from the last hidden layer of a convolutional neural network (CNN) applied to the original feature matrix $\bm{X}$, capitalizing on CNNs’ proven capabilities in visual representation learning. Additionally, we employ task-appropriate error metrics for classification and non-temporal regression settings. Since most of these datasets are substantially smaller than the Chinese news corpus, we are able to utilize finer cross-validation grids for hyperparameter tuning, which we keep fixed across all settings for clarity. We consider all five machine learning techniques (Lasso, Ridge, RF, GBT, and FNN). Since Neural Networks are generally not easy to tune for regression problems if the input data is structured (organized in rows and columns with clearly defined, curated features) (Grinsztajn, Oyallon and Varoquaux, 2022; Shwartz-Ziv and Armon, 2022), only the first four methods are applied to the regression datasets.

Moreover, another interesting idea considered here is to explore the potential of combining different augmentation methods to further enhance model performance. Two approaches are proposed for this purpose: aggregating different factors and combining factors with other feature augmentation techniques. We have already illustrated and used the first approach in the previous section, and we will do the same for all the rest datasets. The second approach is adopted mainly in binary classification problems in which log-likelihood ratios are considered as another augmentation method (Fan et al., 2016). Let $f_{1j},\ f_{2j}$ be the densities of $j^{th}$ feature in class 1 and 2. The log-likelihood ratio for the $j^{th}$ feature is defined as $\operatorname{log}\frac{f_{2j}(s_{j})}{f_{1j}(s_{j})}$, for $j=1,...,p$, and it is the best classifier if only $j^{th}$ feature is used. As pointed out by Fan et al. (2016), naive Bayes uses the summation of these features without any training, and inputting them in the training algorithms usually leads to improvements. The densities can be estimated by kernel density estimation using the training sample, and for the stability of the created features, we set an estimated marginal density to above some threshold $\epsilon$ (say $10^{-2}$) if it is less than $\epsilon$.

In the remainder of this section, we first describe the modified tuning and evaluation procedures, then detail the datasets and associated tasks, and finally present the empirical results — alongside those from the Chinese news data — to further validate the efficacy and adaptability of the proposed augmentation strategies.

5.1 Tuning and Testing

In the cross-validation for hyperparameters, we keep the maximum number of iterations to converge in Lasso and Ridge, the number of trees in RF, and the number of boosting rounds in GBT to be 100. For all the problems, the tuning parameter $\gamma_{1}$ (resp. $\gamma_{2}$) for Lasso (resp. Ridge) is chosen from $20$ numbers from $10^{-3}$ to $10^{3}$ with even space on a log scale. In RF, for classification problems, the maximum depth of trees is chosen from $(5,10,15,20,25,30)$, and the minimum sample number required to split an internal node is chosen from $(1,3,5,8)$; for the regression problems, the maximum depth of trees is chosen from $(3,6,9,12,15,18,21,24)$ and the minimum sample number required to split an internal node is chosen from $(1,2,4,8,16)$. In GBT, both classification and regression have the same parameter grids. The maximum depth of trees is chosen from $(5,10,15,20,25)$, the minimum sum of instance weight needed in a child is chosen from $(1,3,5,8)$, and the learning rate is chosen from $(0.1,0.3,0.5)$.

The performance of classification is measured by the classification error in the testing set, which is the rate of wrongly classified numbers to the size of the testing set. Let $\bm{x}_{1}^{0},\ldots,\bm{x}_{n_{\mathrm{new}}}^{0}$ be the samples in the testing sets with the corresponding true labels $y_{1}^{0},\ldots,y_{n_{\mathrm{new}}}^{0}$, where $n_{\mathrm{new}}$ is the sample size of the testing set. Denote the predicted labels by $\widehat{y}_{1}^{0},\ldots,\widehat{y}_{n_{\mathrm{new}}}^{0}$. Let $I(\cdot)$ be the indicator function. Then, the classification error (ERR) is defined as

[TABLE]

The performance of regression (without rolling-window) is evaluated using the out-of-sample $R^{2}$, which is defined as

[TABLE]

where $\widehat{y}^{0}_{t}$ is the predicted $y^{0}_{t}$, and $\bar{y}_{t}$ is the sample mean of the response values in the training set.

5.2 Data and pre-processing

Seven classification datasets and four groups of regression datasets are studied. To start with, we introduce the basic information of the datasets, their main features, and the preprocessing procedures.

Classification datasets

The MNIST database of handwritten digits (LeCun, 1998): The MNIST database is widely used for training various image processing systems and machine learning methods. It contains 60,000 training images and 10,000 testing images, associated with 10 labels. The images consist of black and white numbers from 0 to 9, each contained within a $28\times 28$ pixel bounding box. To speed up the process, we randomly choose $20\%$ of the training and testing sets respectively in each iteration. This results in a training set of size $12000\times 784$ and a testing set of size $2000\times 784$. 2. 2.

The Fashion-MNIST dataset of Zalando’s article images (Xiao, Rasul and Vollgraf, 2017): The scale of Fashion-MNIST is the same as that of MNIST, consisting of 60,000 training images and 10,000 testing images. Each example is a black and white image assigned to one of the ten clothing labels, including T-shirt/top, Trouser, Pullover, and so on. In each repetition, $20\%$ samples are randomly chosen from the training and testing sets respectively, which leads to a $12000\times 784$ training set and a $2000\times 784$ testing set. 3. 3.

Kaggle Cats and Dogs dataset (DogCat): This is a binary sentiment color image classification problem, consisting of images of cats and dogs. The dataset is available through TensorFlow, with 1738 corrupted images dropped. The original images have different sizes and shapes, so we first standardize the dimensions to $32\times 32$ pixels bounding box. To speed up the experiments, all images are converted to black and white. These procedures will cost the loss of information, which is acceptable in this experiment as we are focused on the comparative performance between the models. Recommendation 601 from ITU-R is used to convert the color images to grayscale, where

[TABLE]

After preprocessing, we get a $16283\times 1024$ training set and a $6979\times 1024$ testing set. 4. 4.

The CIFAR-10 dataset (Krizhevsky et al., 2009): The CIFAR-10 dataset consists of 60,000 $32\times 32$ color images in 10 classes, with 6000 images per class. The 10 labels range from animals to vehicles, including airplanes, automobiles, birds, cats, and so on. Similar to the DogCat dataset, all the images are converted into grayscale using Recommendation 602 from ITU-R. The dataset is divided into five training batches and one test batch, with the test batch containing exactly 1000 randomly selected images from each class. We randomly take $20\%$ of the data in each repetition, resulting in a $10000\times 1024$ training set and a $2000\times 1024$ testing set. 5. 5.

Reuters text categorization dataset: This is a dataset of 11,228 news articles from Reuters, labeled over 46 topics. The data is originally generated by parsing and preprocessing the classic Reuters-21578 datasets and can be accessed through TensorFlow. Each article is encoded as a list of word indexes, sorted by the words’ frequency of appearance in the training set. The data matrix is created such that elements are set to 1 if the word corresponding to the element’s index appears in the article, and 0 otherwise. For the purposes of this experiment, we selected the top 20-300 most frequent words to create an $8082\times 280$ training set and a $2246\times 280$ testing set. 6. 6.

Mice Protein Expression Dataset (Higuera, Gardiner and Cios, 2015): The dataset consists of the expression levels of 77 proteins/protein modifications that produced detectable signals in the nuclear fraction of the cortex. There are in total 72 mice, with 34 of them being trisomic and the rest serving as the control group. For each protein per mouse, 15 measurements are registered, which leads to a total of $72\times 15=1080$ measurements. Each measurement can be considered as an independent sample, so the dataset is $1080\times 77$ in size. The mice are divided into eight classes based on their genotype, behavior, and treatment. We randomly select $80\%$ of the data as the training set, with the remaining data serving as the testing set. 7. 7.

Cervical Cancer Behavior Risk Dataset (Machmud et al., 2016): The dataset contains 18 attributes regarding the risk of CA cervix behavior, with class labels of 1 and 0 indicating respondents with and without CA cervix, respectively. There are 72 samples, and $75\%$ of the data are randomly selected for training the model, with the remaining samples set as the testing set.Given the limited sample size, it is deemed unreliable and unnecessary to use the diversified projection method. As such, only the PCA method is used to estimate the factors.

For the Reuters dataset and all the image classification datasets, in addition to the augmented factors, we use $100$ screened features to reduce the computational complexity.

Regression datasets

Prediction of the U.S. bond risk premia: The response variable is the monthly risk premia in U.S. government bonds with maturities of 2 to 5 years between January 1980 and December 2003, containing 288 data points. The $t$-year risk premium is calculated as the excess log return as the subtraction of the log holding period return of a $t$-year bond and the log yield of the 1-year discount bond (Cochrane and Piazzesi, 2005). The dataset of 2 to 5-year discount bond prices is available from the supplement for Cochrane and Piazzesi (2005). The covariates are the 127 monthly U.S. macroeconomic variables in the FRED-MD database (McCracken and Ng, 2016), which contains observations that mimic the coverage of datasets used in the literature. The covariates are highly correlated, as shown by Fan, Ke and Wang (2020). The FRED-MD database contains NAs due to data availability. To maintain consistency and reliability, columns containing more than $5\%$ NAs are dropped, and the remaining 122 features are used. We apply a one-month ahead rolling window prediction with a window size of 240 months. In each rolling window, we scale the data and perform CV for all algorithms. 2. 2.

Prediction of taxi demand near Terminal 5, New York: The dataset is made publicly available by Rodrigues, Markou and Pereira (2019)and contains 16 features, including lagged observations, weather information, and information about the presence of events. The response variable is the number of individual taxi pickups that took place within a bounding box of $\pm 0.003$ geographical decimal degrees around Terminal 5. The samples are grouped in a daily pattern, with one year of observations (2013) being used for training, two years of data (2014-2015) for validation, and data from January 2016 to June 2016 (six months) for testing. 3. 3.

Prediction of daily new cases of COVID-19 in the UK, the US, Singapore, and Switzerland, respectively: The COVID-19 dataset and related information are provided by Our World in Data by Ritchie et al. (2020). We choose 10 features, including new deaths attributed to COVID-19, number of COVID-19 patients in ICUs, government response stringency index, real-time estimate of the effective reproduction rate, total number of COVID-19 vaccination doses administered, total number of people who received at least one vaccine dose and who received all doses, daily number of people receiving their first vaccine dose, total number of COVID-19 vaccination booster doses administered, and new COVID-19 vaccination doses administered. The data contains observations from January 21, 2021, to July 13, 2022, consisting of 504 samples. We carry out a one-day-ahead rolling window prediction with a window size of 365 days. Similarly to the FRED dataset, in each rolling window, we scale the data and perform CV for all algorithms. 4. 4.

Prediction of the house prices for Zillow: Zillow is an online real estate database company that affords a lot of data about housing prices in the US. The covariates consist of 17 attributes, including features of the house, for example, the number of bedrooms and bathrooms, the area of the living room, the year the house was built, the area the house is located (zip code), and so on. The training set consists of 15129 cases, and the testing set consists of 6484 cases. We first regress the housing price on their zip codes and then regress the rest 16 features on the residual. The performance is still measured based on the estimated housing price.

Since the training set has a size of around $10^{5}\sim 10^{6}$, it is impossible to conduct PCA on the whole dataset and to estimate the factors, especially for the two kernel factors. This is one example where diversified projection comes into play and makes factor models still applicable.

Note: If the diversified projection is used for factor estimation, we randomly take out a tiny subset of the training set to estimate the diversified projection weight matrix and the number of factors.

5.3 Results

This section displays the overall results of our experiments. Our goal is to verify the claim that simple feature augmentations can boost the performance of various learning methods across a wide range of datasets. Comparisons between the feature augmentation methods are also presented. In addition, we will summarize some consistent conclusions across different methods and provide guidance on how to select factors based on the nature of the problems.

All experiments are repeated 20 times to reduce the influence of randomness, and the averages are reported. Figure 5 demonstrates the relative performance of the feature augmentation methods. Plots on the left column are for classifications while those on the right are for regressions. Only two representative results for each kind of problem are shown, and the full images can be found in Figure 9 and Figure 10 at the end. All the classification errors are divided by $\mathrm{ERR}(\bm{X})$, the classification error without feature augmentation. Smaller values indicate better performance. Similarly, all the out-of-sample $R^{2}$ are divided by $R^{2}(\bm{X})$, the out-of-sample $R^{2}$ of the benchmark model. Greater values indicate better performance. The histograms are results for applying PCA on the whole dataset to estimate factors (cf. Section 3.2.1) while the dashed lines with ‘+’ points are the corresponding results for estimating factors by diversified projection (cf. Section 3.2.2). The blue bars are for $(\widehat{\bm{F}}_{0},\widehat{\bm{U}})$, and for each of the newly proposed factors $\widehat{\bm{F}}$, we present the best outcomes among the three augmentations $(\widehat{\bm{F}},\widehat{\bm{U}})$, $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widehat{\bm{U}})$, and $(\widehat{\bm{F}}_{0},\widehat{\bm{F}},\widetilde{\bm{U}})$.

Strength of Factor Models

Factor augmentations have been shown to provide significant improvements over direct estimation using $\bm{X}$ in most cases, particularly for image classification tasks, which is evident in Figure 5 and Figure 9 (at the end), where almost all the bars and points are lower than $1$, indicating improved classification performance with factor augmentations. Although the results for regressions are not as consistent across algorithms and datasets as classifications, for each dataset and algorithm, there exist augmentation methods that improve the estimation. Thus, broadly speaking, factor augmentation can provide performance boosts for both types of problems.

Besides the performance benefits over the original model without feature augmentation, we find that the latent factors obtained from transformed matrices also provide additional insights beyond $\bm{F}_{0}$. For almost all datasets and under all models, there always exist some transformed factors that beat the model augmented with only $\bm{F}_{0}$, showing the additional prediction power of extracted features from nonlinear transformations. Below are additional comments on the performance of specific factors and datasets.

• •

As has been shown by Figure 9, for classifications, factors extracted from neural networks, $\widehat{\bm{F}}_{\mathrm{fnn}}$ and $\widehat{\bm{F}}_{\mathrm{cnn}}$, are the most powerful ones. For example, in the MNIST database using Lasso, while directly estimating with $\bm{X}$ has a high testing error of around 0.16, most factor models decrease the errors to around 0.1. Furthermore, $\widehat{\bm{F}}_{\mathrm{fnn}}$ and $\widehat{\bm{F}}_{\mathrm{cnn}}$ decrease the errors to lower than 0.04. This phenomenon can be attributed to the success of neural networks on unstructured data (e.g. raw images, audio signals, and text sequences).

In contrast, FNN factors do not always stand out prominently in regressions. Two possible reasons for this are (a) the training set is too small to train a reasonable neural network and (b) there are too many parameters to tune on neural networks.

• •

The interaction factor and kernel factors also sometimes beat $\widehat{\bm{F}}_{0}$ for both classification and regression problems. Specifically, for regressions, our results first confirm the conclusion drawn in Fan, Ke and Wang (2020) regarding the bond risk premia prediction experiments that $(\widehat{\bm{F}}_{0},\widehat{\bm{U}}_{0})$ outperforms $\bm{X}$ under Lasso. Moreover, we find that $\widehat{\bm{F}}_{\mathrm{inter}}$ and $\widehat{\bm{F}}_{\mathrm{poly}}$ generally outperform $\widehat{\bm{F}}_{0}$ in feature augmentation, particularly for the two penalized linear models Lasso and Ridge Regression. However, different factors may stand out in different algorithms and for different datasets. For instance, as shown in Section 4.3, $\bm{F}_{\mathrm{fnn}}$ outperforms all the other factors for Chinese-text learning under Lasso, Ridge, and RF while the polynomial kernel factors $\bm{F}_{\mathrm{rbf}}$ are the best under GBT and FNN. Therefore, based on the data and algorithms, it is recommended to try different factors to achieve the best performance.

PCA and Diversified Projection

Though only using a tiny subset of data ($1\%-5\%$ of the training samples) to conduct factor estimation, the diversified projection produces results comparable to those obtained through PCA on the entire training set. This can be seen in Figure 5 where most black ‘+’ points (for diversified projection) are at similar vertical positions of the corresponding colorful bars (for PCA). With the efficiency of diversified projection guaranteed, it is possible to handle cases where the sample size or dimension is ultra-large. The Chinese news text data serves as an example of the effectiveness of diversified projection on datasets with an extremely large number of samples (around $10^{6}$). As shown by Figure 2, it is obvious that under all the studied models, there exists two to five augmentation method achieved by diversified projection that significantly improves prediction accuracy.

Although diversified projection performs comparably to applying PCA on the entire training set, there are instances where the estimation is less accurate. One such example can be seen in the lower-left corner of Figure 5, where diversified projection is used on the MNIST dataset with an FNN model. The reason is that diversified projection only requires the diversified weight matrix $\bm{W}$ to have angles with the loading matrix $\mathbf{B}$ so that $\mathbf{B}$ is not diversified away. This is a trade-off between accuracy and computational efficiency, and by increasing the parameter $n^{\prime}$, the performance of diversified projection can be improved, as this brings it closer to PCA.

Feature Augmentation Aggregation

As has been mentioned in Section 2.3, different augmentation features may carry different information, so they may cooperate to enhance estimation performance. To see this, we first examine if different factors can cooperate to further boost the performance by adding $\widehat{\bm{F}}_{0}$ to the feature space $(\widehat{\bm{F}},\bm{U})$. Figure 6 compares the performance of factor augmentations with and without $\bm{F}_{0}$. On the left-hand side, for each factor model and algorithm, we take the average over all the studied classification problems. If the point is below the one horizon line, it means that the factor contains different information from $\bm{F}_{0}$, and that $\bm{F}_{0}$ and the factor can cooperate to enhance the classification performance. Similarly, on the right-hand side, we look at the out-of-sample $\mathbb{R}^{2}$ averaging over all the studied regression problems for each factor model and algorithm. If the point is above the one horizon line, it means that $\bm{F}_{0}$ can cooperate with the corresponding factor to boost the regression performance. It is presented that further augmenting with $\widehat{\bm{F}}_{0}$ can most of the time result in a little performance improvement, with an average reduction in classification error and an average increase in regression out-of-sample $R^{2}$. Besides adding $\bm{F}_{0}$, it is also possible to use alternative or additional factors to aggregate information and enhance performance.

Furthermore, likelihood ratios (LRs) are added as another feature augmentation technique to DogCat, the binary sentiment classification dataset. The performance of LR augmentation is compared to the proposed factor augmentations in the left-hand side of Figure 7. If the point in the plot is lower than the zero horizon line, it indicates that the factor augmentation outperforms the LR augmentation outperform. The left-most parts of the plot show that LR augmentation improves the performance of $\bm{X}$ in Lasso and Ridge regression, but barely in the other three studied algorithms. The comparison between LR augmentation and factor augmentation reveals that factor augmentations outperform LR augmentations on the DogCat dataset. Moreover, the right-hand side of Figure 7 displays the relative classification errors of different factor augmenting models with and without LR. If the point is below the zero horizon line, it indicates that further adding LR improves the performance in addition to the corresponding factor augmentation. It is shown that LR can further boost feature augmentation performance when using Lasso and Ridge, the two simple methods. Nevertheless, it should be noted that while LR augmentation adds the same number of features as the original dataset, the proposed factor augmentations only add a limited number of features and can be applied to various problems, not just binary sentiment classification. Therefore, based on the results, factor augmentations can achieve similar or even better performance in some cases compared to LR augmentations with way fewer additional features and a broader scope of application.

Cases Where Factor Augmentation Does Not Work Well

For most cases, feature augmentations do boost the estimation performance, as shown in Figure 5. Nevertheless, there exist cases where factor augmentation does not work well. For the classification problems, when the sample size is too small, such as the Cervical cancer example (see Figure 9), which only has 72 samples in total, some feature augmentation approaches may not be superior compared to estimating directly with $\bm{X}$ under the advanced learning methods RF, GBT, and FNN. Since these three learning methods are much more complicated than Lasso and Ridge, it makes sense that adding some features does not boost the performance under those methods, especially when the sample size is small. However, even for the Cervical cancer dataset, we notice that the proposed feature augmentation approaches beat $\bm{X}$ under Lasso and Ridge. Meanwhile, with RF and GBT, there are always some different factors that defeat $\bm{X}$. However, in the case of FNN, none of the factor augmentations outperformed $\bm{X}$. This is likely due to the fact that neural networks are capable of learning features themselves, and for the Cervical cancer dataset, the neural networks have already learned sufficient information even with only two hidden layers, rendering feature augmentation unnecessary.

In some regression problems, certain algorithms may also fail to benefit from feature augmentations. For bond risk premia prediction, while most factor models can significantly boost performance in Lasso, RF, and GBT, Ridge regression does not perform well here because the feature dimension is of similar order with sample size, while Ridge does not induce sparsity. Besides, in cases where factor augmentations do not significantly improve performance, such as the Covid-19 data with RF and GBT, the taxi data, and the Zillow data, it can be seen that the regression on $\bm{X}$ already achieves satisfactory accuracy. For example, using RF or GBT to predict new Covid-19 cases with $\bm{X}$ already achieves an out-of-sample $R^{2}$ ranging from $0.93$ to $0.99$. This is accurate enough, and therefore, additional augmentation techniques are not necessary. (A visualization of the predictions on Covid19 in Switzerland is shown in Figure 8.) What we have shown is that when the estimation is not that satisfactory, factor augmentations can bring about notable improvement and increase accuracy.

Selection of transformation methods

To make the proposed framework more accessible to practitioners, we provide practical guidance on selecting appropriate nonlinear transformations. Exhaustively trying all available options is neither computationally efficient nor practically feasible. Instead, the following aspects should be considered when choosing suitable transformations:

• •

Characteristics of the problem and data: The choice of transformation should align with the nature of the dataset and the domain-specific characteristics of the task. For instance, neural networks are known to be effective for textual data, while convolutional neural networks (CNNs) excel with image-based inputs. This intuition is consistent with our empirical findings, where $\bm{F}_{\mathrm{fnn}}$ performs well on the Chinese text data, and $\bm{F}_{\mathrm{cnn}}$ performs well on image datasets. Beyond such domain-specific preferences, more general considerations also apply: if interactions between samples are believed to be informative, kernel-based methods may be appropriate; if interactions among features are potentially meaningful, then constructing feature interaction matrices may be beneficial.

• •

Dimensionality of the original design matrix: Although the proposed framework is already computationally efficient, extremely high-dimensional data may still necessitate caution. When the sample size is very large, full kernel matrices may be infeasible to compute, in which case other transformations or sparse kernel representations, such as the one used for the Chinese text data, are recommended. Conversely, when the number of features is very large, computing all pairwise interactions may become intractable, and techniques such as sure screening (see Section 3.4) or alternative transformations should be considered.

• •

Complexity of the machine learning algorithm: As shown in our experiments, the effectiveness of a transformation may depend on the complexity of the base learning algorithm. For example, in the Chinese text data application, $\bm{F}_{\mathrm{fnn}}$ substantially outperforms other transformations when used with simpler models such as Lasso, Ridge, or RF. However, this advantage diminishes under more complex models like GBT and FNN. In general, when using a complex learning algorithm, simpler transformations may suffice, while more sophisticated transformations are more useful when the learning model is relatively simple, such as linear regression.

• •

Need for interpretability: While complex transformations such as those derived from neural networks (e.g., $\bm{F}_{\mathrm{fnn}}$) may offer strong predictive performance, they often lack interpretability due to their black-box nature. In domains where model transparency is critical — such as finance or econometrics — transformations that yield more interpretable factors may be preferable.

Although there is no universally optimal transformation method for a given dataset or algorithm, the considerations above provide a structured framework for narrowing the search space. Once a shortlist of plausible transformations is identified, the final selection can be made based on validation performance, akin to standard model selection procedures. Lastly, we note that it is also possible to combine multiple sets of factors or augmented features within a single model, as discussed at the beginning of this Section.

Highlights

Detailed as has been illustrated above, we highlight three takeaway messages as follows. First, as an additional and independent method, our proposed augmentation strategy is designed not to compete but to collaborate with existing algorithms and possibly other data augmentation methods. Second, it is evident that feature augmentation by various types of factors usually leads to improved performance, and the degree of improvement depends on the context of the problems. Not all feature augmentation methods will bring significant improvements, but there are always some factor augmentations that perform reasonably well. Finally, the proposed factor augmentation approaches usually tend to be powerful when the initial estimation does not have high accuracy and the sample size is not exceedingly small. This is understandable as they are fundamental limits on generation errors.

6 Conclusion And Futher Discussion

We have put forward a series of simple yet effective feature augmentation techniques to study the stock return prediction using Chinese financial news text data, with a generalization to all high-dimensional learning problems. These methods are based on extracting latent nonlinear factors through transformations of the original feature matrix. We illustrate three representative transformation families — interactions, kernel methods, and neural networks — due to their potential to uncover meaningful latent structures in text data in the stock return prediction problem. While these serve as core examples, the framework is flexible and can accommodate a wide range of transformation techniques, provided they are capable of capturing useful structure informed by the nature of the data.

Our empirical analysis demonstrates that augmenting the feature space with nonlinear factors significantly enhances the accuracy of stock return estimation, which in turn leads to consistent improvements in downstream financial applications, including event studies and portfolio construction. To further validate the generality of the approach, we conduct extensive experiments across diverse supervised learning problems, including both classification and regression tasks from domains such as image recognition, biology, finance, and natural language processing. These results affirm the versatility and practical utility of the proposed framework. We offer a broad set of factor types and augmentation strategies, and provide practical guidance for selecting suitable transformations based on problem-specific considerations. In the majority of studied cases, factor-based augmentation yields notable gains in generalization performance with minimal additional computational cost.

A key strength of the proposed framework lies in its modularity: it operates independently of the underlying learning algorithm and can be seamlessly integrated into any method that leverages covariate information. Moreover, the augmentation need not be confined to the initial input layer—it can also be applied at intermediate stages of learning pipelines, particularly in settings with highly correlated features or weak marginal signals. Also, while our implementation primarily relies on linear factor extraction techniques, such as principal component analysis and diversified projections, the framework is equally compatible with nonlinear alternatives, like autoencoder-decoders (Xiu and Shen, 2024; Cerqueti et al., 2024). For instance, one may construct an hourglass-shaped FNN, e.g., with hidden layer widths $[128,64,16,64,128]$, and train it to reconstruct the transformed design matrix. The bottleneck layer, which compresses the high-dimensional input into a low-dimensional representation, can then be treated as a set of learned nonlinear factors. Such approaches offer a promising direction for future research, particularly in capturing complex structures beyond the reach of linear models.

Finally, this augmentation strategy readily extends to matrix- and tensor-valued data, supported by recent developments in high-dimensional factor modeling (Chen, Fan and Zhu, 2024; Chen, Yang and Zhang, 2022). These extensions enhance the relevance and applicability of our framework in emerging areas such as macroeconomic forecasting, financial modeling, and biological data analysis.