Transformer-Based Deep Learning for Preoperative Prediction of Microvascular Invasion in Hepatocellular Carcinoma

TL;DR

A deep learning model using MRI and clinical data accurately predicts microvascular invasion in liver cancer, helping guide personalized treatment.

Contribution

A Transformer-based model for three-class MVI prediction in HCC using radiomics and clinical features.

Findings

The model achieved 0.733 accuracy and 0.880 AUC on the internal test set.

It maintained 0.758 accuracy and 0.886 AUC on an external validation set.

High sensitivity (0.73) and specificity (0.91) for high-risk M2 cases were observed.

Abstract

Microvascular invasion (MVI) is a key predictor of recurrence in hepatocellular carcinoma (HCC). We developed a Transformer-based deep learning model to noninvasively predict three MVI grades (M0, M1, and M2) using preoperative MRI radiomics and clinical biomarkers. The model achieved high accuracy across independent datasets. Notably, reliable identification of high-risk M2 patients before surgery could help optimize resection margins, guide adjuvant therapy, and support individualized treatment planning.

Background: Microvascular invasion (MVI) is a critical prognostic factor in hepatocellular carcinoma (HCC), but preoperative three-class prediction remains challenging. Radiomics and clinical biomarkers may enable more accurate and individualized assessment. Aim: The aim of this study was to develop and validate a Transformer-based deep learning framework that integrates radiomic and clinical features for direct three-class MVI classification in HCC patients. Methods: This retrospective study included 437 patients with pathologically confirmed hepatocellular carcinoma (HCC) and microvascular invasion (MVI) status from two campuses of a single institution. Patients from Hospital A (n = 305) were randomly divided into training and internal test cohorts, while patients from Hospital B (n = 132) were used as an independent external validation cohort. Radiomic features were extracted from preoperative Gd-BOPTA-enhanced MRI, and clinical laboratory data were collected. A two-stage feature selection strategy, combining univariate statistical testing and recursive feature elimination, was applied. A Transformer-based model was built to classify three MVI categories (M0, M1, M2), and its performance was evaluated in both the internal test cohort and the external validation cohort. Results were compared with those from traditional machine learning models, including Random Forest, Logistic Regression, XGBoost, and LightGBM. Results: On the internal test set (n = 76, Hospital A), the model achieved an accuracy of 0.733 (95% CI: 0.64–0.83), a weighted F1-score of 0.733, and a macro-average AUC of 0.880 (95% CI: 0.807–0.953). The sensitivity and specificity for M1 were 0.56 (95% CI: 0.31–0.78) and 0.86 (95% CI: 0.74–0.94), respectively; for high-risk M2 cases, the sensitivity was 0.73 (95% CI: 0.64–0.81) and the specificity was 0.91 (95% CI: 0.85–0.96). On the external validation set (n = 132, Hospital B), performance remained stable with an accuracy of 0.758, a weighted F1-score of 0.768, and a macro-average AUC of 0.886 (95% CI: 0.833–0.940). Conclusions: This Transformer-based model enables accurate and objective three-class MVI prediction using multi-modal features, supporting individualized surgical planning and improved clinical outcomes. In particular, the ability to preoperatively identify high-risk M2 patients may inform surgical margin design, guide adjuvant therapy strategies, and influence liver transplantation eligibility.

Figures

The overall workflow for radiomics-based three-class classification of microvascular invasion (MVI) in hepatocellular carcinoma (HCC). Radiomic features were extracted from Gd-enhanced MRI based on manually delineated regions of interest (VOIs). A two-step feature selection strategy was applied, combining univariate statistical tests (t-test or Mann–Whitney U test) and recursive feature elimination (RFE). A Transformer model was trained for MVI classification (M0, M1, M2), and performance was evaluated using confusion matrices and ROC curves.

Full text

1. Introduction

Hepatocellular carcinoma (HCC) is the fifth most common cancer globally and the third leading cause of cancer-related mortality [1,2]. In the Asia–Pacific region, particularly China, chronic hepatitis B virus (HBV) infection accounts for 70–90% of HCC cases [3]. Although surgical resection and liver transplantation offer potential curative options for early-stage HCC, recurrence rates remain high, affecting 40–70% of patients post-resection and 10–15% post-transplantation, with most recurrences occurring within two years [4,5].

Microvascular invasion (MVI), defined as the presence of tumor cells within microvessels lined by endothelial cells, is widely recognized as a major predictor of early recurrence and poor prognosis after surgery [6,7]. International guidelines such as those from the American Association for the Study of Liver Diseases (AASLD) and the European Association for the Study of the Liver (EASL) identify MVI as a critical prognostic factor [2], although detailed subclassification has not been standardized globally. According to histopathological consensus, particularly the Liver Cancer Pathology Group of China (LCPGC) proposal [8], MVI is commonly categorized into three subtypes: M0 (no invasion), M1 (≤5 invaded vessels within 1 cm from the tumor margin), and M2 (>5 invaded vessels or invasion beyond 1 cm). Previous studies have demonstrated that M2 status is strongly associated with increased recurrence risk and inferior long-term survival [6]. Therefore, accurate preoperative identification of high-risk M2 patients could inform surgical margin design, guide adjuvant therapies, and potentially influence eligibility for liver transplantation.

Currently, MVI diagnosis relies exclusively on postoperative pathological examination, limiting its utility for preoperative decision-making [9]. To address this limitation, radiomics- and MRI-based predictive models have emerged as promising noninvasive alternatives. However, most existing studies are restricted to binary classification (MVI presence vs. absence) and lack the capacity to differentiate high-risk M2 status [10,11,12]. Although these approaches often report high AUC values (~0.90), they fail to differentiate M1 from M2, thereby limiting their clinical applicability. Only a few studies have attempted direct three-class prediction. Nevertheless, these models remain underexplored and often rely on conventional machine learning algorithms with limited ability to capture complex feature dependencies.

Radiomics enables the extraction of high-throughput, quantitative features from medical images, offering insights into tumor heterogeneity that is imperceptible to the human eye [13]. When combined with clinical laboratory indicators, radiomics can offer a comprehensive characterization of tumor biology. Nevertheless, the high dimensionality, redundancy, and complex interactions among these features pose challenges for conventional machine learning approaches [14].

Transformer models, originally developed for natural language processing (NLP) tasks, have recently demonstrated superior performance in modeling high-dimensional structured data through their self-attention mechanisms [15,16]. Their ability to capture nonlinear and hierarchical feature dependencies makes them particularly suitable for integrating multi-modal data sources, such as radiomic features and clinical variables, which traditional models often struggle with. In addition to their established role in characterizing macroscopic tumor morphology, radiomic features have been shown to reflect microscopic pathological properties, such as angiogenesis, stromal composition, and local immune infiltration. Prior studies have demonstrated significant correlations between quantitative MRI-derived texture features and histopathological findings of microvascular invasion in HCC, supporting the biological plausibility of imaging-based prediction of microscopic phenomena. This cross-modality association underpins the rationale for leveraging radiomics and deep learning models to noninvasively approximate MVI status.

In this study, we propose a Transformer-based deep learning model that integrates radiomic features from preoperative MRI and clinical laboratory data for direct three-class MVI classification (M0, M1, M2) in HCC patients. Our approach has the potential to facilitate more detailed risk stratification in a relatively automated and reproducible manner.

2. Materials and Methods

2.1. Patient Enrollment and Clinical Data Collection

This study was approved by the Ethics Committee of Eastern Hepatobiliary Surgery Hospital (approval number: EHBHKY2022-H-P002; approval date: 26 October 2022). The requirement for informed consent was waived due to the retrospective nature of the study. Patient privacy was strictly protected through anonymization procedures.

A total of 437 patients with pathologically confirmed hepatocellular carcinoma (HCC) and available microvascular invasion (MVI) assessments were included between January 2017 and December 2023. Of these, 305 patients from Hospital A were used for model training and internal testing (229 for training and 76 for testing), while 132 patients from Hospital B served as an independent external validation cohort. According to standardized pathological criteria, the final cohort comprised 267 M0 cases, 103 M1 cases, and 67 M2 cases.

The inclusion criteria were as follows: patients who underwent liver resection and received Gd-BOPTA-enhanced MRI within two months prior to surgery; histopathological confirmation of HCC and MVI status; no prior treatments before surgery (e.g., radiofrequency ablation, transarterial chemoembolization (TACE)); absence of extrahepatic malignancies; availability of complete clinical, imaging, and pathological data.

Exclusion criteria included patients who underwent preoperative local therapies, had extrahepatic malignancies, or lacked essential imaging or clinical data.

Baseline demographic and clinical characteristics, including routine laboratory indicators (e.g., AFP, PIVKA-II, CA19-9, CEA, ALT, AST, ALB, TBIL, DBIL, GGT, PLT, PT), are summarized in Table 1. Liver function status was assessed based on the Child–Pugh classification, and all enrolled patients were classified as Child–Pugh class A or B before surgery. No statistically significant differences were observed among the training, internal test, and external validation cohorts. These clinical variables were selected based on their established relevance to tumor aggressiveness, liver function, and microvascular invasion, as supported by prior studies. They were used in conjunction with radiomic features as model inputs to provide complementary biological and imaging information. The patient selection process is illustrated in Figure 1.

2.2. MRI Image Acquisition

All participants were required to fast for at least 6 h and to abstain from water intake for at least 4 h before the MRI examination. The scanning range extended from the upper edge to the lower edge of the liver.

The contrast agent, 0.1 mmol/kg of Gd-BOPTA (MultiHance, Bracco Imaging S.p.A., Milan, Italy), was administered through injection into the median vein of the patient’s elbow using a high-pressure syringe at a rate of 2.0 mL/s, followed by a 20 mL saline flush lasting approximately 20 s.

After contrast agent administration, imaging was performed during specific phases: arterial phase (AP) at 25–45 s, portal venous phase (PVP) at 50–70 s, and delayed phase (DP) at 100–180 s. The MRI protocol also included fat-suppressed T2-weighted imaging (T2WI), T1-weighted imaging (T1WI), and diffusion-weighted imaging (DWI). To assess interobserver reproducibility, intraclass correlation coefficients (ICCs) were calculated for a subset of cases. Features with ICC > 0.75 were retained.

2.3. Imaging Analysis

Manual segmentation of HCC lesions was performed in a three-dimensional (3D) manner across all MRI sequences, including T2WI, T1WI, DWI, AP, PVP, and DP. Multiple contiguous slices covering the entire tumor volume were delineated to generate volumetric regions of interest (VOIs). Two radiologists (each with >5 years of experience in abdominal MRI interpretation) independently performed the segmentation, and a senior radiologist (>10 years of experience) reviewed the results. All radiologists were blinded to the pathological outcomes to avoid potential bias. Manual segmentation was conducted using ITK-SNAP software (version 3.6.0; http://www.itksnap.org, accessed on 1 October 2025). The consensus VOIs were subsequently used for radiomics feature extraction and statistical analysis. No qualitative radiological assessments were included; all model inputs consisted solely of objective data, including clinical laboratory parameters and quantitative radiomic features extracted from manually delineated VOIs. This strategy minimized potential interobserver variability and enhanced the objectivity, reproducibility, and generalizability of the predictive model.

2.4. Pathological Evaluation

All hematoxylin and eosin (H&E)-stained sections were retrospectively reviewed for MVI by two board-certified pathologists (each with >10 years of hepatopathology experience). Both pathologists were blinded to all clinical and imaging data, and discrepancies were resolved by consultation with a senior pathologist (>20 years of experience).

MVI status was determined according to the standardized histopathological criteria proposed by the Liver Cancer Pathology Group of China (LCPGC). Patients were classified into three subgroups: M0 (no microvascular invasion), M1 (≤5 invaded vessels within 1 cm of the tumor margin), and M2 (>5 invaded vessels or invasion beyond 1 cm).

This pathological classification (M0, M1, and M2) has been widely adopted and recognized in Chinese clinical practice. Although it has not yet been globally standardized, it demonstrates good consistency with histopathological assessments of microvascular invasion and is suitable for our study cohort based on Chinese patients.

Based on these criteria, the present cohort consisted of 267 M0 cases, 103 M1 cases, and 67 M2 cases. This pathology-based classification system has been widely adopted in clinical research for prognostic stratification of HCC.

2.5. Radiomic Feature Extraction

Radiomic feature extraction was conducted following manual segmentation of the regions of interest (VOIs) by an experienced radiologist using ITK-SNAP software (version 3.6.0) [17,18]. The delineated VOIs were used to extract quantitative features with the PyRadiomics (version 3.0.1, Computational Imaging Biomarker Group, Harvard Medical School, Boston, MA, USA) library. A total of 1500 radiomic features were obtained, including first-order statistical descriptors (such as mean, skewness, and kurtosis), shape-based metrics (such as sphericity and elongation), and texture features derived from gray-level matrices, including the Gray-Level Co-occurrence Matrix (GLCM) and the Gray-Level Run-Length Matrix (GLRLM) [18]. To better characterize tumor heterogeneity, features were extracted not only from the original images but also from filtered versions processed via wavelet and Laplacian of Gaussian (LoG) transformations [19,20]. These features collectively capture tumor intensity distributions, morphological characteristics, and intratumoral textural heterogeneity, which have been linked to histopathologic manifestations such as necrosis, angiogenesis, and stromal composition.

2.6. Data Preprocessing

Preprocessing involved two major steps:

• Missing Value Handling: Samples with more than 10% missing values were excluded. For the remaining samples, missing values were imputed separately for each feature using k-nearest neighbor (KNN) imputation (k = 3) implemented in Scikit-learn (version 1.3.1, Python Software Foundation, Wilmington, DE, USA)., based on the Euclidean distance in the feature space [21,22].
• Normalization: Radiomic features were normalized using Z-score transformation. The mean and standard deviation were calculated exclusively from the training set and then applied to the validation and test sets to ensure no information leakage during model evaluation [23]. Normalization parameters were derived from the training set and applied unchanged to the test and validation sets.

2.7. Feature Selection

To mitigate the risk of overfitting and enhance model generalizability, a two-stage feature selection strategy was employed. In the initial screening phase, features were first evaluated for reproducibility using the intraclass correlation coefficient (ICC) [19], and only those with ICC > 0.75 were retained. Subsequently, univariate statistical tests (t-test or Mann–Whitney U test, depending on the data distribution) were performed to exclude features with weak discriminative ability.

In the dimensionality reduction phase, recursive feature elimination (RFE) [24] based on Random Forest classifiers was conducted to further select the most informative subset of features.

To systematically evaluate the impact of different feature selection strategies, sixteen pipelines (4 initial screening methods × 4 dimensionality reduction methods) were constructed and compared using cross-validation performance within the training set. Ultimately, the pipeline combining “t-test or Mann–Whitney U test + RFE with Random Forest” was selected for final model training, as it achieved the best balance between feature compactness and model performance.The overall workflow of radiomics-based three-class MVI classification using the Transformer model is shown in Figure 2.

The overall workflow for radiomics-based three-class classification of microvascular invasion (MVI) in hepatocellular carcinoma (HCC). Radiomic features were extracted from Gd-enhanced MRI based on manually delineated regions of interest (VOIs). A two-step feature selection strategy was applied, combining univariate statistical tests (t-test or Mann–Whitney U test) and recursive feature elimination (RFE). A Transformer model was trained for MVI classification (M0, M1, M2), and performance was evaluated using confusion matrices and ROC curves.

2.8. Model Architecture and Training

We developed a Transformer-based deep learning model tailored for three-class MVI prediction. The model architecture was adapted from the Transformer framework, originally introduced for natural language processing tasks [15] and later extended to structured tabular data such as clinical and radiomic features [16]. The model was implemented in PyTorch (version 2.6.0, Meta AI, Menlo Park, CA, USA).As illustrated in Figure 3, the model architecture comprised three main components: an input embedding layer that projected the input features into an 8-dimensional latent space; two stacked Transformer encoder layers, each utilizing 2 attention heads, a feedforward dimension of 16, and a dropout rate of 0.2; and a final output classification layer that generated predictions across three categories.

Following the definition of the model architecture, we proceeded to model training. Cross-entropy loss with class-balanced weights was employed to mitigate class imbalance. Model parameters were optimized using the AdamW optimizer with a learning rate of 0.0005 and a weight decay of 0.02. The mini-batch size was set to 16. Training was conducted for 2000 epochs on an NVIDIA RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA).

2.9. Data Augmentation and Balancing

As illustrated in Figure 2 and Figure 3, the overall preprocessing and modeling pipeline included feature normalization, missing value imputation, two-step feature selection, and Transformer-based classification. To address the pronounced class imbalance in the training set, we employed a hybrid resampling strategy. Specifically, the Synthetic Minority Over-sampling Technique combined with Edited Nearest Neighbors (SMOTE-ENN) was applied to simultaneously oversample minority classes and remove borderline or noisy samples [22]. Importantly, resampling was performed exclusively on the training cohort, while the internal test and external validation cohorts were kept unchanged to preserve the integrity of model evaluation. After resampling, the training set exhibited a more balanced distribution across the three MVI categories (M0, M1, and M2), as demonstrated in Figure 4. Updated class distributions were then used to recompute class weights, which were incorporated into the cross-entropy loss function to further mitigate residual imbalance during training. This combined strategy has been widely reported to improve classifier robustness in highly imbalanced multi-class medical datasets.

3. Results

3.1. Baseline Characteristics

The baseline characteristics of the included patients are presented in Table 1. The absence of significant inter-cohort differences indicates that the three datasets were comparable for subsequent model development and validation.

3.2. Overall Model Performance

The Transformer-based model achieved consistently high performance across both internal and external cohorts (Table 2). In the internal test cohort, the model reached an accuracy of 0.733, a weighted F1-score of 0.733, and a macro-average AUC of 0.880 (95% CI: 0.807–0.953). In the external validation cohort, performance remained stable, with an accuracy of 0.758, a weighted F1-score of 0.768, and a macro-average AUC of 0.886 (95% CI: 0.833–0.940). These findings confirm the model’s robustness and generalizability across independent datasets.

3.3. Class-Wise Diagnostic Performance

Class-specific results are summarized in Table 3. The model achieved the highest accuracy for M0 cases, with sensitivity above 0.79 across the cohorts. For M1 cases, the sensitivity was moderate (0.56 in the test set and 0.62 in the validation set), but the specificity remained above 0.85. Importantly, for high-risk M2 cases, the sensitivity reached 0.74 in the external validation set, while the specificity remained above 0.89, highlighting the model’s potential clinical utility in identifying patients at greatest risk of recurrence.

3.4. Visualization of Classification Outcomes

The confusion matrices and ROC curves (Figure 5) illustrate classification outcomes across datasets. The model consistently showed strong discriminative ability for M0, whereas moderate overlap was observed between M1 and M2, reflecting the intrinsic difficulty of differentiating intermediate- and high-risk invasion.

3.5. Comparison with Traditional Machine Learning Models

Compared with traditional classifiers (Random Forest, Logistic Regression, XGBoost, and LightGBM), the Transformer-based model achieved superior performance across all metrics (Table 4). For example, it reached a validation AUC of 0.886 versus 0.834 for Random Forest and 0.837 for Logistic Regression. These results demonstrate the methodological advantage of the Transformer in handling high-dimensional, multi-modal data for MVI classification.

4. Discussion

Our adoption of a Transformer-based model in this context constitutes a methodological innovation in MVI classification. While Transformer architectures have been increasingly applied in medical imaging, their application to structured radiomic data for multi-class classification remains underrepresented. By tailoring the model to capture dependencies among quantitative features, our framework extends beyond prior binary MVI classification attempts, offering a more nuanced and clinically useful risk stratification strategy. Beyond methodological innovation, the proposed model also has practical clinical implications. Accurate preoperative identification of M2 cases may guide surgeons to consider wider resection margins, influence the decision to initiate adjuvant therapies, and potentially serve as an exclusion criterion for liver transplantation. Reliable recognition of M0 patients could prevent overtreatment and allow for more conservative surgical strategies. Importantly, distinguishing M1 cases provides additional value by identifying patients at intermediate risk, who may benefit from closer postoperative surveillance or tailored adjuvant treatment strategies. By providing this stratified information preoperatively, our framework has the potential to directly inform surgical planning and optimize patient management across all three subgroups.

This study demonstrated the feasibility and effectiveness of using a Transformer-based deep learning model to perform direct three-class classification of microvascular invasion (MVI) in hepatocellular carcinoma (HCC) patients by integrating radiomic features and clinical laboratory indicators. Compared with conventional machine learning methods such as Random Forest and Logistic Regression, the proposed model consistently achieved superior performance. Specifically, the Transformer obtained an AUC of 0.880 (95% CI: 0.807–0.953) and a weighted F1-score of 0.733 in the internal test cohort and maintained stable performance in the external validation cohort with an AUC of 0.886 (95% CI: 0.833–0.940) and a weighted F1-score of 0.768. These findings highlight not only the methodological advantages of the Transformer in modeling complex, high-dimensional, structured medical data but also its robustness and generalizability across independent datasets, attributable to its ability to capture nonlinear and hierarchical feature dependencies through its self-attention mechanism [15,16,25].

Previous radiomics-based studies have primarily focused on binary MVI classification (presence vs. absence), typically reporting AUCs ranging from 0.75 to 0.85 [11,26,27]. In contrast, our approach enables one-step, three-class prediction (M0, M1, M2), providing a more granular and clinically actionable preoperative risk stratification strategy. This advancement allows for refined surgical planning and supports individualized treatment decisions. Additionally, unlike earlier studies that often relied on subjective imaging features manually assessed by radiologists, our model is entirely based on objectively extracted features, which reduces interobserver variability and improves reproducibility [28].

The key contributions of this study lie in three aspects: proposing a Transformer-based framework for direct MVI three-class classification, demonstrating the benefit of multi-modal feature integration from radiomics and clinical data, and ensuring high model reproducibility by using only objective quantitative inputs. Nevertheless, several limitations must be acknowledged. First, although the model performed well on both internal and external validation sets, the dataset was collected from a limited geographic region, which may restrict its generalizability to broader populations and imaging protocols [29].

Future research should focus on external validation using multi-center and multi-ethnic datasets to improve robustness and generalizability [30]. In addition, incorporating multiomics data [e.g., genomics, transcriptomics, proteomics] and circulating tumor DNA (ctDNA) may further enhance the model’s predictive capacity and biological interpretability [31]. Recent evidence has shown that ctDNA can both predict and act independently of MVI [32], suggesting that its integration with imaging and clinical data represents a promising avenue for future research.

Nevertheless, several limitations should be acknowledged. First, this was a retrospective study conducted at two campuses of a single institution, which may limit generalizability to other populations and imaging protocols. Second, although the proposed Transformer model showed high performance, external validation using multi-center and multi-ethnic cohorts is required. Third, the radiomic features were extracted from manually segmented VOIs, which might have introduced observer bias despite quality control procedures. Future research should address these issues and explore integration with multiomics data to enhance model interpretability and clinical applicability.

The Transformer architecture inherently incorporates an attention mechanism, which offers potential for exploring feature importance and model interpretability. In future work, we plan to further investigate attention weight visualization to better understand the model’s decision process.

Although MVI serves as a crucial pathological biomarker associated with early recurrence and poor prognosis in HCC, it remains a surrogate rather than a direct clinical outcome.

Future work will focus on expanding the dataset through potential collaborations with other liver transplant centers, aiming to perform multi-center validation and improve the generalizability of the proposed Transformer-based model.

5. Conclusions

In this study, we proposed a Transformer-based deep learning framework that integrates radiomic features and clinical laboratory indicators for the preoperative three-class classification of MVI in patients with HCC.

The proposed model effectively captured complex high-dimensional feature relationships and achieved competitive performance across training, validation, and independent test sets.

By enabling direct three-class MVI prediction in a single step, our approach provides more granular prognostic information and holds promise for supporting individualized treatment planning and streamlining clinical decision-making.

Despite the encouraging results, further work is needed to enhance the model’s clinical applicability.

Future research should focus on expanding validation cohorts to larger, multi-center, and multi-ethnic datasets, improving model interpretability through explainable artificial intelligence (XAI) techniques, and incorporating multiomics data to enrich biological insights and further boost predictive accuracy.