A Multimodal and Multi-centric Head and Neck Cancer Dataset for Segmentation, Diagnosis and Outcome Prediction

TL;DR

This paper introduces a large, diverse multimodal head and neck cancer dataset with detailed annotations and clinical data, enabling advanced research in segmentation, diagnosis, and outcome prediction.

Contribution

It provides a publicly available, multi-institutional dataset with comprehensive annotations and benchmarks for key clinical tasks using state-of-the-art deep learning models.

Findings

Successful automated tumor segmentation using deep learning models

Effective recurrence-free survival prediction with multimodal data

Accurate HPV status classification leveraging clinical and imaging features

Abstract

We present a publicly available multimodal dataset for head and neck cancer research, comprising 1123 annotated Positron Emission Tomography/Computed Tomography (PET/CT) studies from patients with histologically confirmed disease, acquired from 10 international medical centers. All studies contain co-registered PET/CT scans with varying acquisition protocols, reflecting real-world clinical diversity from a long-term, multi-institution retrospective collection. Primary gross tumor volumes (GTVp) and involved lymph nodes (GTVn) were manually segmented by experienced radiation oncologists and radiologists following established guidelines. We provide anonymized NifTi files, expert-annotated segmentation masks, comprehensive clinical metadata, and radiotherapy dose distributions for a patient subset. The metadata include TNM staging, HPV status, demographics, long-term follow-up outcomes, survival times, censoring indicators, and treatment information. To demonstrate its utility, we benchmark three key clinical tasks: automated tumor segmentation, recurrence-free survival prediction, and HPV status classification, using state-of-the-art deep learning models like UNet, SegResNet, and multimodal prognostic frameworks.

Figures

Tables

Table 1. Table 1 : Overview of participating centers, their country of origin, acronyms, and the PET/CT scanners used in the study

Center	Country	Acronym	PET/CT scanner
Hôpital général juif	Canada	HGJ	Discovery ST, GE Healthcare
Centre hospitalier universitaire de Sherbrooke	Canada	CHUS	GeminiGXL 16, Philips
Hôpital Maisonneuve-Rosemont	Canada	HMR	Discovery STE, GE Healthcare
Centre hospitalier de l’Université de Montréal	Canada	CHUM	Discovery STE, GE Healthcare
Centre Hospitalier Universitaire de Poitiers	France	CHUP	Biograph mCT 40 ToF, Siemens
MD Anderson Cancer Center	USA	MDA	Discovery HR, Discovery RX, Discovery ST, Discovery STE (GE Healthcare)
University Hospital of Zürich	Switzerland	USZ	Discovery HR, Discovery RX, Discovery STE, Discovery LS, Discovery 690 (GE Healthcare)
Centre Henri Becquerel	France	CHB	GE710, GE Healthcare
Centre Hospitalier Universitaire de Brest	France	CHUB	Philips GEMINI, Siemens Biograph, Siemens Biograph Vision
Centre Hospitalier Universitaire de Nantes	France	CHUN	Siemens mCT 64 vision

Table 2. Table 2 : PET/CT acquisition parameters across centers. Values are median (range) unless stated otherwise. Var. indicates that the value is not fixed and may differ across cases.

Center	PET Dose (MBq)	Uptake (min)	PET System	PET Time/Pos (s)	PET Slice (mm)	PET In-plane ($m{m}^2$)	CT kVp	CT Slice (mm)
HGJ	584 (368–715)	90	Discovery ST (GE)	300 (180–420)	3.27	3.52×3.52 (3.52–4.69)	140	3.75
CHUS	325 (165–517)	90	Gemini GXL 16 (Philips)	150 (120–151)	4	4×4	140 (12–140)	3 (2–5)
HMR	475 (227–859)	90	Discovery STE (GE)	360 (120–360)	3.27	3.52×3.52 (3.52–5.47)	140 (120–140)	3.75
CHUM	315 (199–3182)	90	Discovery STE (GE)	300 (120–420)	4 (3.27–4)	4×4 (3.52–5.47)	120 (120–140)	1.5 (1.5–3.75)
CHUP	421$\pm$98 (220–695)	60$\pm$5	Biograph mCT 40 ToF (Siemens)	150	5	4×4×4 $m{m}^3$	120	5
MDA	401 (327–266)	90	Multiple hybrid PET/CT	180 (90–300)	3.27 (2.99–5)	5.46×5.46 (2.73–5.46)	120 (100–140)	3.75 (2.99–5)
USZ	178–513	$\sim$60	Multiple hybrid PET/CT	Var.	Var.	Var.	Var.	Var.
CHB	$\sim$3 MBq/kg	90$\pm$5	GE710 (GE)	Var.	3.27	2.73×2.73	120	2.5
CHUN	204 (129–354)	60	Biograph mCT 60/40 (Siemens)	Var.	2	4×4	120	2.0
CHUB	$\sim$6 or 3 MBq/kg	60	Philips Gemini G-XL 6 , Siemens Biograph mCT and Siemens Biograph Vision600	Var.	4, 2 and 1.6	4×4, 4×4 and 1.6×1.6	120	4.0, 3.0 and 2.0

Table 3. Table 3 : Segmentation Results on GTVp and GTVn for different models. Best values are in bold .

Model	GTVp_mean_dice	GTVn_DSC_agg	GTVn_F1_agg
UNet	0.62 $\pm$ 0.03	0.70 $\pm$ 0.01	0.32 $\pm$ 0.09
SegResNet	0.67 $\pm$ 0.02	0.73 $\pm$ 0.01	0.54 $\pm$ 0.04
SwinUNETR	0.64 $\pm$ 0.04	0.72 $\pm$ 0.01	0.15 $\pm$ 0.05

Table 4. Table 4 : C-index and Brier scores for different modalities and models. Best values per column within each modality group are in bold . Only 187 patients were available with RTDose compared to 680 for other modalities.

Modality	Model	C-Index ($\uparrow$)	Brier Score ($\downarrow$)
EHR	CoxPH	0.59 $\pm$ 0.06	0.16 $\pm$ 0.04
EHR	DeepHit	0.53 $\pm$ 0.06	0.19 $\pm$ 0.02
EHR	MTLR	0.55 $\pm$ 0.05	0.17 $\pm$ 0.02
EHR	ICARE	0.60 $\pm$ 0.06	0.26 $\pm$ 0.01
EHR + CT + PET	CoxPH	0.61 $\pm$ 0.06	0.26 $\pm$ 0.01
EHR + CT + PET	DeepHit	0.55 $\pm$ 0.07	0.21 $\pm$ 0.02
EHR + CT + PET	MTLR	0.59 $\pm$ 0.09	0.17 $\pm$ 0.01
EHR + CT + PET	ICARE	0.67 $\pm$ 0.06	0.23 $\pm$ 0.01
EHR + CT + PET + RTDose	CoxPH	0.59 $\pm$ 0.08	0.26 $\pm$ 0.02
EHR + CT + PET + RTDose	DeepHit	0.53 $\pm$ 0.08	0.22 $\pm$ 0.02
EHR + CT + PET + RTDose	MTLR	0.51 $\pm$ 0.08	0.17 $\pm$ 0.02
EHR + CT + PET + RTDose	ICARE	0.61 $\pm$ 0.08	0.24 $\pm$ 0.02

Table 5. Table 5 : Classification Results on 5 different model settings using CT, PET, and EHR data. Best values are in bold .

Model	F1-Macro	AUC	Sensitivity	Specificity	Balanced Accuracy	F1-weighted
DenseNet121 + MLP	0.64 $\pm$ 0.13	0.77 $\pm$ 0.22	0.99$\pm$ 0.04	0.22 $\pm$ 0.1918	0.61 $\pm$ 0.0968	0.89 $\pm$ 0.0360
ResNet18 + MLP	0.77 $\pm$ 0.09	0.94 $\pm$ 0.03	0.99 $\pm$ 0.00	0.43 $\pm$ 0.15	0.71 $\pm$ 0.07	0.93 $\pm$ 0.02
ViT + MLP	0.52 $\pm$ 0.11	0.852 $\pm$ 0.05	1.00 $\pm$ 0.00	0.07 $\pm$ 0.15	0.53 $\pm$ 0.07	0.86 $\pm$ 0.03
EfficientNet + MLP	0.55 $\pm$ 0.11	0.58 $\pm$ 0.25	0.99 $\pm$ 0.03	0.12 $\pm$ 0.16	0.55 $\pm$ 0.07	0.87 $\pm$ 0.03
SENet + MLP	0.58 $\pm$ 0.11	0.79 $\pm$ 0.04	0.99 $\pm$ 0.01	0.15 $\pm$ 0.15	0.57 $\pm$ 0.07	0.87 $\pm$ 0.03

Equations

$$\mathrm{DSC}$$

Full text

Background and Summary

Head and neck cancer (HNC) represents a heterogeneous group of malignancies arising from anatomically and functionally critical regions, including the oral cavity, pharynx, larynx, paranasal sinuses, and salivary glands [1]. The intricate anatomy of the head and neck, housing vital structures for speech, swallowing, and breathing, presents substantial challenges for diagnosis, treatment, and long-term disease control. Approximately 90% of HNC cases are histologically classified as squamous cell carcinomas (HNSCC). Established risk factors include tobacco use and alcohol consumption [2, 3, 4, 5, 6, 7, 8], which act synergistically to substantially increase disease risk (up to 35-fold in heavy users of both substances) [2, 6, 9, 10, 8, 11].

Despite therapeutic advances, the 5-year survival rate for HNC remains low, especially in advanced or recurrent cases. Current clinical workflows, which are based on TNM (tumor size, lymph nodes, metastasis) staging, imaging, and histopathology, do not consistently enable early detection or accurate prognostication. The use of specific biomarkers, such as Human PapillomaVirus (HPV) status and gene expression profiles, has improved risk stratification in specific subgroups. Regardless, most of these biomarkers remain investigational and are not routinely integrated into clinical decision-making. These limitations underscore the need for improved diagnostic and prognostic tools capable of capturing the complex, multimodal nature of the disease.

Artificial intelligence (AI) offers a promising avenue to address these gaps by integrating diverse biomedical data sources, such as imaging (Computed Tomography (CT), Positron Emission Tomography (PET)), radiotherapy dose distributions, and clinical records (EHR). Multimodal approaches can exploit complementary information to produce more accurate, robust, and generalizable models for diagnosis, prognosis, and treatment planning [12, 13, 14]. However, the development and rigorous validation of such models for HNC has been hindered by the scarcity of large, diverse, and publicly accessible datasets. Most existing studies rely on small, single-center cohorts, limiting reproducibility and clinical translation due to low generalizability of the trained models.

To address this issue, we present a large-scale, multi-institutional, multimodal HNC dataset collected from 10 international centers. The dataset integrates co-registered CT and PET imaging, radiotherapy dose distributions, and structured clinical data from electronic health records, complemented by expert-annotated segmentation masks, as well as diagnostic and prognostic labels. This breadth of information enables the exploration of AI methods for tumor detection and segmentation [15, 16], diagnostic tasks and outcome prediction [17], as well as treatment optimization. By spanning multiple institutions and a long retrospective period of time for data collection, the dataset reduces single-center bias and supports the development of models with improved generalizability across diverse populations, scanner models, acquisition protocols and image reconstruction settings. It thus provides an unprecedented resource for advancing methodological and clinical research in HNC.

Methods

Data Collection

The dataset is a large-scale, multi-institutional collection of multimodal PET/CT scans and rich clinical data from 1,123 patients with histologically confirmed HNC. Fig. 1 shows the variety of modalities available in the dataset. As illustrated in Fig. 2(a), the data is sourced from 10 international medical centers, and captures diversity in scanner models, acquisition and reconstruction protocols, and patient demographics, offering a realistic foundation for developing models that may generalize better across different institutions. The participating clinical centers, along with their acronyms, country, and PET/CT scanner models, are summarized in Table 1. It is one of the largest publicly available resources of its kind, featuring standardized three-dimensional imaging, expert-delineated segmentation, and curated metadata that include staging, treatment intent, and long-term follow-up outcomes. This breadth and depth make it a powerful benchmark for image-based prediction and for creating models with real clinical value.

The dataset has been designed to support three complementary tasks. The first is segmentation of the primary gross tumor volume (GTVp) and involved lymph nodes (GTVn). These segmentation masks were generated manually by experienced radiation oncologists or radiologists, following institutional pre-established guidelines and undergoing quality checks before release. They are co-registered with the PET/CT scans. The second task is recurrence-free survival (RFS) prediction, combining imaging and clinical variables with time-to-event data, enabling robust prognostic modeling. The third task is diagnosis of HPV status, allowing exploration of imaging and clinical correlates of this important biomarker.

The cohort composition reflects its multi-centric nature. MD Anderson center (MDA) is the largest contributor, with 444 patients (39.6%), followed by Centre Hospitalier Universitaire de Brest (CHUB) with 216 patients (19.2%) and the University Hospital of Zurich (USZ) with 101 patients (9.0%). The remaining seven centers contribute between 18 and 72 patients each, as shown in Fig. 2(a). This broad institutional representation provides the variation in imaging and patient populations that is critical for testing model generalization.

Clinical data were harmonized across all centers to include RFS times, censoring indicators, HPV status, demographics, and tumor staging. Fig. 3 shows that 843 patients (80.1%) were censored, while 209 (19.9%) experienced a recurrence or death during follow-up. Regarding the 873 patients for which the HPV status information is available, 587 patients (67.2%) were positive and 286 (32.8%) were negative( Fig. 3). The age distribution spans from 21 to 92 years, with a median of 60 years, and reveals clear demographic trends (Fig. 2(b)). The majority of patients are concentrated in the 50–69-year age range, accounting for more than 70% of the dataset, consistent with the peak incidence period for HNC. Males predominate in nearly every age bracket, particularly within this peak range, where the male-to-female ratio is most pronounced. This imbalance reflects established epidemiological patterns, influenced by historically higher exposure to risk factors such as tobacco use, alcohol consumption, and certain occupational hazards among men.

Acquisition Settings and Data properties

Table 2 summarizes the PET/CT acquisition parameters for all participating centers, including injected dose, uptake time, scanner model(s), acquisition duration, image resolution, and CT settings. Values are presented as medians with ranges where available to highlight inter-center variability in imaging protocols.

Annotation Protocol

CHUS, HGJ, and HMR: The GTVp and GTVn contours were delineated by an experienced radiation oncologist using a radiotherapy treatment planning system. In 40% of the training set (80 cases), the contours were created directly on the CT component of the PET/CT scan and then applied for treatment planning. The remaining 60% (121 cases) were contoured on a separate CT scan acquired specifically for treatment planning, and subsequently aligned to the FDG-PET/CT reference frame via intensity-based free-form deformable registration in MIM software (MIM Software Inc., Cleveland, OH). The number of annotators involved in the training set is not documented.

CHUP: The primary tumor’s metabolic volume was first obtained automatically using the Fuzzy Locally Adaptive Bayesian (FLAB) method dedicated to PET tumor volume delineation [18]. These results were then reviewed and manually refined by a single expert with reference to the CT images, making adjustments in cases where the PET-based segmentation encompassed air or non-tumor tissues visible on the CT.

MDA: Contours were applied on the CT image, while co-registered PET images were used to aid physicians in visualizing the tumor. Then, these contours were updated according to the guidelines mentioned in Annotation Guidelines.

USZ: The primary tumor was delineated independently on the CT and PET scans. CT-based segmentation was carried out manually, with two radiation oncologists, each with over 10 years of experience, participating in the process. The resulting contours were then refined to account for metal artifacts, ensuring exclusion of any non-tumor-related effects. If artifacts were present in a given tumor slice, the contour for that slice was completely removed. Tumors with more than 50% of their volume deemed unsuitable for analysis were excluded from the study. Furthermore, voxels falling outside the soft tissue Hounsfield unit (HU) range of 20 to 180 HU were eliminated. For the PET scans, tumor segmentation was performed automatically using a gradient-based approach available in MIMVISTA (MIM Software Inc., Cleveland, OH).

CHB: A senior nuclear medicine physician manually drew the contours for each patient for both the GTVp and GTVn using the software PET VCAR (GE Healthcare) on each FDG-PET/CT, using adaptive thresholding with visual control using merged PET and CT information.

CHUN: The CHUN contours were created using an nn-UNET automatic segmentation approach using the full training and testing sets of the HECKTOR 2022 data, with careful post-processing of results to ensure higher accuracy. Later, a consultant physician in a clinical imaging department reviewed the contours in a case-by-case manner, following the guidelines in the Annotation Guidelines section. This review process confirmed the contours of the GTVp and GTVn for correctly segmented cases and adjusted the cases missed by the model. After the initial CT/PET, a subset of confirmed cases had a follow-up CT with RTDose. The follow-up CT was registered to the initial CT, and the resulting transformation matrix was applied to the RTDose to bring it into the same space as the initial CT/PET.

Annotation Guidelines

Experts conducted quality control across both the training and test datasets to maintain consistency in the definition of ground-truth contours. When needed, they re-annotated the data to reflect the actual tumor volume, which was often smaller than the volumes originally outlined for radiotherapy. The contouring process was centralized through a shared cloud platform (MIM Cloud Software Inc.) to standardize the annotation environment. For cases lacking original radiotherapy GTVp or GTVn contours, experts generated annotations using PET/CT fusion in conjunction with N staging information. A set of guidelines for this quality control process was established by the expert panel, as detailed below. Cases showing PET–CT misregistration were excluded.

Guidelines for primary tumor annotation in PET/CT images: Oropharyngeal lesions are delineated on PET/CT by integrating information from both PET and unenhanced CT scans. The contour should encompass the complete boundaries of the morphological abnormality seen on unenhanced CT, typically visualized as a mass effect, along with the corresponding hypermetabolic volume identified on PET. This is achieved using PET, unenhanced CT, and PET/CT fusion views obtained through automatic co-registration. Areas of hypermetabolic uptake extending beyond the physical borders of the lesion, such as into the airway lumen or over bony structures without morphological evidence of invasion, are excluded. The standardized naming convention follows AAPM TG-263, with the primary tumor labeled as GTVp. The special considerations in this regard are to verify the clinical nodal category to ensure exclusion of adjacent FDG-avid and/or enlarged lymph nodes (e.g., submandibular, high level II, or retropharyngeal nodes). For cases showing fullness or enlargement of the tonsillar fossa or base of tongue without corresponding FDG uptake, consult the clinical datasheet to rule out pre-radiation tonsillectomy or extensive biopsy. Such cases should be excluded if confirmed.

Guidelines for nodal metastases tumor annotation in PET/CT images: Lymph node contours are defined on PET/CT by integrating data from both PET and unenhanced CT scans. Delineation should encompass the full boundaries of the morphologically abnormal lymph node observed on unenhanced CT, along with the corresponding hypermetabolic region identified on PET. This process uses PET, unenhanced CT, and PET/CT fusion views obtained through automatic co-registration, covering all cervical lymph node levels. Inclusion criteria for metastatic nodes are: standardized uptake value (SUV) greater than 2.5, or a short-axis diameter of at least 1 cm, or pathological confirmation. The standardized label for lymph node regions of interest is GTVn. Areas of hypermetabolic uptake extending beyond the anatomical limits of the node, such as into adjacent bony, muscular, or vascular structures, are excluded from the contour. If multiple nodes are touching or merging, they should be contoured as a single structure. GTVn volumes must always be kept separate from GTVp volumes.

Annotation Tool

To ensure accuracy and consistency of tumor delineations, a dedicated annotation tool was developed for use by board-certified radiation oncologists and radiologists. The tool integrates PET/CT viewing, segmentation, and quality control within a single interface, enabling precise identification and labeling of GTVp and GTVn according to standardized protocols.

The interface combines axial PET/CT fusion visualization with adjustable CT windowing and opacity controls, allowing annotators to optimize image contrast and highlight metabolic or structural features. Layer controls manage segmentation transparency, label selection, and editing modes, while patient navigation and progress tracking ensure a systematic annotation workflow. Segmentation follows embedded protocols defining inclusion criteria for GTVp (entire morphological anomaly on CT plus corresponding hyper-metabolic PET regions, excluding non-tumor uptake) and GTVn (morphologically abnormal lymph nodes with size thresholds). Workflow features include smart interpolation between slices, manual refinement tools, cleanup functions to remove stray voxels, and mandatory separation of contiguous GTVp/GTVn regions. This standardized platform reduces inter-observer variability and ensures high-quality, reproducible annotations for robust benchmarking in HNC AI research.

Data Records

Format

The dataset is organized by task, with separate subfolders for each of the three clinical tasks, as illustrated in Fig. 4. Within each task directory, patient-level folders contain co-registered PET and CT scans in NIfTI format. In addition, expert-generated segmentation masks for GTVp and GTVn are also included. For Task 2, a subset of patients includes additional modalities, namely dosimetry planning CT, and radiotherapy dose map, provided in the same format and aligned with the imaging volumes. Clinical metadata is provided in .csv files, with structured fields for patient ID, center, age, alcohol consumption, TNM staging, HPV status, performance status, recurrence status, and RFS in months. This task-based folder structure, combined with standardized file formats and naming conventions, facilitates modular loading and integration into automated deep learning pipelines. The details regarding the available data fields are shown in Fig. 1.

Dataset Limitations

A first limitation is that contours were generated using PET/(unenhanced) CT fusion, whereas modalities like contrast-enhanced CT are better suited for defining accurate contours in radiation oncology. Nevertheless, given that the intended clinical use is radiomics, this level of accuracy is less critical than it would be for radiotherapy planning. Another limitation stems from variability in the ground truth annotations. Even with established guidelines and quality control measures, differences in annotation practices persisted. For example, in the USZ test set only, lesions with metal artifacts were excluded, and in some cases, re-segmentation was performed within a specific HU range. Variations also arose from differences in the annotators’ expertise and profiles. Despite attempts to standardize the contours, these inconsistencies introduced a notable amount of noise into the labeled data used for training. Additionally, there were challenges faced related to the availability of annotators to work on the 216 cases coming from the CHUB center. This disables leveraging this considerable number of scans for any segmentation tasks where ground truth is needed. A third limitation concerns the HPV status, as the information was not available for a number of patients (n=250), for the most part because at the time of diagnosis and management for these patients, it was not yet standard clinical practice to analyze HPV status.

Technical Validation

To demonstrate the potential usefulness and validity of our dataset, we performed three representative downstream tasks relevant to HNC research: (1) automated tumor segmentation, (2) patient prognosis prediction, and (3) HPV status classification. For each task, we trained and evaluated state of the art machine learning models to establish performance benchmarks and showcase the dataset’s potential.

Deep Learning-based Tumor Segmentation

To showcase the dataset for segmentation tasks, we evaluated three deep learning architectures: UNet, SegResNet, and SwinUNETR. For model input, CT and PET images were preprocessed by 1) resampling to a $1\times 1\times 1mm^{3}$ isotropic resolution, 2) cropping to a uniform head and neck region of interest, and 3) normalizing intensity values. The processed images were then concatenated to form a 2-channel input. Using a 5-fold cross-validation scheme (80/20 split), we evaluated the models’ ability to segment primary tumors and nodal metastases.

We implemented three complementary metrics for segmentation performance evaluation. The Dice Similarity Coefficient (DSC) is used to quantify the volumetric overlap of the primary tumor. However, for lymph node segmentation, the DSC can be unreliable in cases without ground-truth volumes. In such cases, even a single false negative results in a DSC of zero. To mitigate this, the aggregated Dice Similarity Coefficient ($\text{DSC}_{\text{agg}}$) [19] is used to provide a more stable estimate by aggregating overlap across all cases, as shown in Equation (1), where the summation is taken over all cases. To also capture detection performance, we employ the F1-score at the lesion level [20]. The matching rules for these cases, including merges and splits, are illustrated in Figure 5. Together, these metrics provide a comprehensive evaluation that captures not only spatial overlap but also detection sensitivity.

[TABLE]

As detailed in Table 3, the SegResNet model achieved the best overall performance, with a mean DSC of $0.6660\pm 0.0199$ for primary tumors and an aggregated DSC of $0.7291\pm 0.0092$ for nodal metastases. Figure 6 provides qualitative visualization. These results confirm the dataset’s high quality and suitability for training and validating robust segmentation algorithms.

Deep Learning-based Prognosis

To demonstrate the dataset’s utility for clinical outcome modeling, we trained models to predict RFS. We benchmarked four prognostic models (CoxPH [21], DeepHit [22], MTLR [23], and ICARE [24]) using different data combinations: Electronic Health Records (EHR) alone, EHR combined with CT and PET imaging-based radiomics features, and a subset including radiotherapy dose (RTDose) maps. Model performance was evaluated using the Concordance Index (C-index) for discrimination (higher is better) and the Brier score for calibration (lower is better).

As detailed in Table 4, integrating imaging with EHR data consistently improved prognostic performance. The ICARE model achieved the highest discriminative power when trained on EHR, CT, and PET data, yielding a C-index of $0.6705\pm 0.0608$. The MTLR model showed the best calibration on the same data combination, with the lowest Brier score of $0.1653\pm 0.0130$. While the addition of RTDose maps was explored, it led to mixed results with some models benefiting from dosimetric information while others experienced slight performance decreases, likely attributable to the substantially reduced sample size (187 versus 680 patients) available for this modality, highlighting the trade-off between data richness and statistical power. Overall, these benchmarks validate the dataset’s value for developing and comparing sophisticated, multimodal survival models.

Deep Learning-based HPV Status Classification

Finally, we explored the task of classifying HPV status using imaging and EHR data. We evaluated five deep learning models where image features from a CNN (e.g., ResNet18) were combined with EHR data using a Multi-Layer Perceptron (MLP). Using a 5-fold cross-validation, we evaluated performance with multiple metrics, including the F1-Macro score and the Area Under the ROC Curve (AUC). The ResNet18 + MLP model yielded the best classification results, achieving an AUC of 0.9379 and a weighted F1-score of 0.9277 (Table 5). This best result was, however, associated with a limited balanced accuracy of 0.71 because of a low specificity (0.43) despite excellent sensitivity (0.99). This use case shows that the dataset is a valuable resource for developing machine learning models to predict molecular biomarkers from standard clinical and multimodal imaging data.

Code availability

The HECKTOR dataset generated and analyzed in this study is publicly available at https://hecktor25.grand-challenge.org/. We respectfully request that researchers cite this paper in any work that exploits this dataset. The code for data preprocessing, along with the training and evaluation scripts for all baseline models, is available on GitHub at https://github.com/BioMedIA-MBZUAI/HECKTOR2025. The final trained model weights are also provided within the same repository.