Automated Design of Deep Learning Methods for Biomedical Image Segmentation

TL;DR

The paper introduces nnU-Net, an automated deep learning framework for biomedical image segmentation that adapts to various datasets without manual tuning, outperforming specialized methods in numerous competitions.

Contribution

nnU-Net is a novel, fully automated framework that encodes domain knowledge and adapts deep learning architectures to different biomedical segmentation tasks.

Findings

Outperforms most specialized pipelines in 19 public competitions.

Sets new state-of-the-art in 49 segmentation tasks.

Available as open-source tool for broad accessibility.

Abstract

Biomedical imaging is a driver of scientific discovery and core component of medical care, currently stimulated by the field of deep learning. While semantic segmentation algorithms enable 3D image analysis and quantification in many applications, the design of respective specialised solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We propose nnU-Net, a deep learning framework that condenses the current domain knowledge and autonomously takes the key decisions required to transfer a basic architecture to different datasets and segmentation tasks. Without manual tuning, nnU-Net surpasses most specialised deep learning pipelines in 19 public international competitions and sets a new state of the art in the majority of the 49 tasks. The results demonstrate a vast hidden potential in the systematic adaptation of deep learning methods to different datasets. We make nnU-Net publicly available as an open-source tool that can effectively be used out-of-the-box, rendering state of the art segmentation accessible to non-experts and catalyzing scientific progress as a framework for automated method design.

Figures

Tables

Table 1. Table A.1: Overview over the challenge datasets used in this manuscript.

ID	Dataset Name	Associated Challenges	Modalities	Median Shape (Spacing [mm])	N Classes	Rarest Class Ratio	N Training Cases	Segmentation Tasks
D1	Brain Tumour	\citeSupplementSM_decathlonDataPaper, \citeSupplementSM_menze2014multimodal	MRI (T1, T1c, T2, FLAIR)	138x169x138 (1, 1, 1)	3	${7.310}^{-3}$	484	edema, active tumor, necrosis
D2	Heart	\citeSupplementSM_decathlonDataPaper	MRI	115x320x232 (1.37, 1.25, 1.25)	1	${4.010}^{-3}$	20	left ventricle
D3	Liver	\citeSupplementSM_decathlonDataPaper, \citeSupplementSM_bilic2019liver	CT	432x512x512 (1, 0.77, 0.77)	2	${2.610}^{-2}$	131	liver, liver tumors
D4	Hippocampus	\citeSupplementSM_decathlonDataPaper	MRI	36x50x35 (1, 1, 1)	2	${2.710}^{-2}$	260	anterior and posterior hippocampus
D5	Prostate	\citeSupplementSM_decathlonDataPaper	MRI (T2, ADC)	20x320x319 (3.6, 0.62, 0.62)	2	${5.410}^{-3}$	32	peripheral and transition zone
D6	Lung	\citeSupplementSM_decathlonDataPaper	CT	252x512x512 (1.24, 0.79, 0.79)	1	${3.910}^{-4}$	63	lung nodules
D7	Pancreas	\citeSupplementSM_decathlonDataPaper	CT	93x512x512 (2.5, 0.80, 0.80)	2	${2.010}^{-3}$	282	pancreas, pancreas cancer
D8	HepaticVessel	\citeSupplementSM_decathlonDataPaper	CT	49x512x512 (5, 0.80, 0.80)	2	${1.110}^{-3}$	303	hepatic vessels, tumors
D9	Spleen	\citeSupplementSM_decathlonDataPaper	CT	90x512x512 (5, 0.79, 0.79)	1	${4.710}^{-3}$	41	spleen
D10	Colon	\citeSupplementSM_decathlonDataPaper	CT	95x512x512 (5, 0.78, 0.78)	1	${5.610}^{-4}$	126	colon cancer
D11	AbdOrgSeg	\citeSupplementSM_BCVAbdomenChallenge	CT	128x512x512 (3, 0.76, 0.76)	13	${4.410}^{-3}$	30	13 abdominal organs
D12	Promise	\citeSupplementSM_promise12Challenge	MRI	24x320x320 (3.6, 0.61, 0.61)	1	${2.010}^{-2}$	50	prostate
D13	ACDC	\citeSupplementSM_ACDCTMI	cine MRI	9x256x216 (10, 1.56, 1.56)	3	${1.210}^{-2}$	200 (100x2) *	left ventricle, right ventricle, myocardium
D14	LiTS **	\citeSupplementSM_bilic2019liver	CT	432x512x512 (1, 0.77, 0.77)	2	${2.610}^{-2}$	131	liver, liver tumors
D15	MSLesion	\citeSupplementSM_mslesionchallenge	MRI (FLAIR, MPRAGE, PD, T2)	137x180x137 (1, 1, 1)	1	${1.710}^{-3}$	42 (21x2) *	multiple sclerosis lesions
D16	CHAOS	\citeSupplementSM_kavur2020chaos	MRI	30x204x256 (9, 1.66, 1.66)	4	${3.310}^{-2}$	60 (20 + 20x2) *	liver, spleen, left and right kidney
D17	KiTS	\citeSupplementSM_heller2019state	CT	107x512x512 (3, 0.78, 0.78)	2	${7.510}^{-3}$	206	kidney, kidney tumor
D18	SegTHOR	\citeSupplementSM_initialsegthorpaper	CT	178x512x512 (2.5, 0.98, 0.98)	4	${4.610}^{-4}$	40	heart, aorta, esophagus, trachea
D19	CREMI	\citeSupplementSM_heinrich2018synaptic	Electron Microscopy	125x1250x1250 (40, 4, 4)	1	${5.210}^{-3}$	3	synaptic clefts
* multiple annotated examples per training case
** almost identical to Decathlon Liver; Decathlon changed the training cases and test set slightly

Table 2. Table F.1: Network configurations generated by nnU-Net for the BrainTumour dataset from the Medical Segmentation Decathlon (D1). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1 x 1	1 x 1 x 1	-
Median image shape at target spacing:	NA x 169 x 138	138 169 138	-
Patch size:	192 x 160	128 x 128 x 128	-
Batch size:	107	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 3. Table F.2: Decathlon BrainTumour (D1) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set") Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	edema	non-enhancing tumor	enhancing tumour	mean
2D	0.7957	0.5985	0.7825	0.7256
3D_fullres *	0.8101	0.6199	0.7934	0.7411
Best Ensemble	0.8106	0.6179	0.7926	0.7404
Postprocessed	0.8101	0.6199	0.7934	0.7411
Test set	0.68	0.47	0.68	0.61

Table 4. Table F.3: Network configurations generated by nnU-Net for the Heart dataset from the Medical Segmentation Decathlon (D2). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1.25 x 1.25	1.37 x 1.25 x 1.25	-
Median image shape at target spacing:	NA x 320 x 232	115 x 320 x 232	-
Patch size:	320 x 256	80 x 192 x 160	-
Batch size:	40	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 1]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], ]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 5. Table F.4: Decathlon Heart (D2) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	left atrium	mean
2D	0.9090	0.9090
3D_fullres *	0.9328	0.9328
Best Ensemble	0.9268	0.9268
Postprocessed	0.9329	0.9329
Test set	0.93	0.93

Table 6. Table F.5: Network configurations generated by nnU-Net for the Liver dataset from the Medical Segmentation Decathlon (D3). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.7676 x 0.7676	1 x 0.7676 x 0.7676	2.47 x 1.90 x 1.90
Median image shape at target spacing:	NA x 512 x 512	482 x 512 x 512	195 x 207 x 207
Patch size:	512 x 512	128 x 128 x 128	128 x 128 x 128
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 7. Table F.6: Decathlon Liver (D3) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D low resolution U-Net and the 3D full resolution U-Net.

	liver	cancer	mean
2D	0.9547	0.5637	0.7592
3D_fullres	0.9571	0.6372	0.7971
3D_lowres	0.9563	0.6028	0.7796
3D cascade	0.9600	0.6386	0.7993
Best Ensemble*	0.9613	0.6564	0.8088
Postprocessed	0.9621	0.6600	0.8111
Test set	0.96	0.76	0.86

Table 8. Table F.7: Network configurations generated by nnU-Net for the Hippocampus dataset from the Medical Segmentation Decathlon (D4). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1 x 1	1 x 1 x 1	-
Median image shape at target spacing:	NA x 50 x 35	36 x 50 x 35	-
Patch size:	56 x 40	40 x 56 x 40	-
Batch size:	366	9	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 9. Table F.8: Decathlon Hippocampus (D4) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	Anterior	Posterior	mean
2D	0.8787	0.8595	0.8691
3D_fullres *	0.8975	0.8807	0.8891
Best Ensemble	0.8962	0.8790	0.8876
Postprocessed	0.8975	0.8807	0.8891
Test set	0.90	0.89	0.895

Table 10. Table F.9: Network configurations generated by nnU-Net for the Prostate dataset from the Medical Segmentation Decathlon (D5). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.62 x 0.62	3.6 x 0.62 x 0.62	-
Median image shape at target spacing:	NA x 320 x 319	20 x 320 x 319	-
Patch size:	320 x 320	20 x 320 x 256	-
Batch size:	32	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [1, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2], [1, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 11. Table F.10: Decathlon Prostate (D5) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	PZ	TZ	mean
2D	0.6285	0.8380	0.7333
3D_fullres	0.6663	0.8410	0.7537
Best Ensemble *	0.6611	0.8575	0.7593
Postprocessed	0.6611	0.8577	0.7594
Test set	0.77	0.90	0.835

Table 12. Table F.11: Network configurations generated by nnU-Net for the Lung dataset from the Medical Segmentation Decathlon (D6). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.79 x 0.79	1.24 x 0.79 x 0.79	2.35 x 1.48 x 1.48
Median image shape at target spacing:	NA x 512 x 512	252 x 512 x 512	133 x 271 x 271
Patch size:	512 x 512	80 x 192 x 160	80 x 192 x 160
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 13. Table F.12: Decathlon Lung (D6) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D low resolution U-Net and the 3D full resolution U-Net.

	cancer	mean
2D	0.4989	0.4989
3D_fullres	0.7211	0.7211
3D_lowres	0.7109	0.7109
3D cascade	0.6980	0.6980
Best Ensemble*	0.7241	0.7241
Postprocessed	0.7241	0.7241
Test set	0.74	0.74

Table 14. Table F.13: Network configurations generated by nnU-Net for the Pancreas dataset from the Medical Segmentation Decathlon (D7). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.8 x 0.8	2.5 x 0.8 x 0.8	2.58 x 1.29 x 1.29
Median image shape at target spacing:	NA x 512 x 512	96 x 512 x 512	93 x 318 x 318
Patch size:	512 x 512	40 x 224 x 224	64 x 192 x 192
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 15. Table F.14: Decathlon Pancreas (D7) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D full resolution U-Net and the 3D U-Net cascade.

	pancreas	cancer	mean
2D	0.7738	0.3501	0.5619
3D_fullres	0.8217	0.5274	0.6745
3D_lowres	0.8118	0.5286	0.6702
3D cascade	0.8101	0.5380	0.6741
Best Ensemble *	0.8214	0.5428	0.6821
Postprocessed	0.8214	0.5428	0.6821
Test set	0.82	0.53	0.675

Table 16. Table F.15: Network configurations generated by nnU-Net for the HepaticVessel dataset from the Medical Segmentation Decathlon (D8). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.8 x 0.8	1.5 x 0.8 x 0.8	2.42 x 1.29 x 1.29
Median image shape at target spacing:	NA x 512 x 512	150 x 512 x 512	93 x 318 x 318
Patch size:	512 x 512	64 x 192 x 192	64 x 192 x 192
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 17. Table F.16: Decathlon HepaticVessel (D8) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D full resolution U-Net and the 3D low resolution U-Net.

	Vessel	Tumour	mean
2D	0.6180	0.6359	0.6269
3D_fullres	0.6456	0.7217	0.6837
3D_lowres	0.6294	0.7079	0.6687
3D cascade	0.6424	0.7138	0.6781
Best Ensemble *	0.6485	0.7250	0.6867
Postprocessed	0.6485	0.7250	0.6867
Test set	0.66	0.72	0.69

Table 18. Table F.17: Network configurations generated by nnU-Net for the Spleen dataset from the Medical Segmentation Decathlon (D9). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.79 x 0.79	1.6 x 0.79 x 0.79	2.77 x 1.38 x 1.38
Median image shape at target spacing:	NA x 512 x 512	187 x 512 x 512	108 x 293 x 293
Patch size:	512 x 512	64 x 192 x 160	64 x 192 x 192
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 19. Table F.18: Decathlon Spleen (D9) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D U-Net cascade and the 3D full resolution U-Net.

	spleen	mean
2D	0.9492	0.9492
3D_fullres	0.9638	0.9638
3D_lowres	0.9683	0.9683
3D cascade	0.9714	0.9714
Best Ensemble *	0.9723	0.9723
Postprocessed	0.9724	0.9724
Test set	0.97	0.97

Table 20. Table F.19: Network configurations generated by nnU-Net for the Colon dataset from the Medical Segmentation Decathlon (D10). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.78 x 0.78	3 x 0.78 x 0.78	3.09 x 1.55 x 1.55
Median image shape at target spacing:	NA x 512 x 512	150 x 512 x 512	146 x 258 x 258
Patch size:	512 x 512	56 x 192 x 160	96 x 160 x 160
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 21. Table F.20: Decathlon Colon (D10) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform and only two significant digits are reported. Best ensemble on this dataset was the combination of the 3D U-Net cascade and the 3D full resolution U-Net.

	colon cancer primaries	mean
2D	0.2852	0.2852
3D_fullres	0.4553	0.4553
3D_lowres	0.4538	0.4538
3D cascade *	0.4937	0.4937
Best Ensemble	0.4853	0.4853
Postprocessed	0.4937	0.4937
Test set	0.58	0.58

Table 22. Table F.21: Network configurations generated by nnU-Net for the BCV challenge (D131. For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.76 x 0.76	3 x 0.76 x 0.76	3.18 x 1.60 x 1.60
Median image shape at target spacing:	NA x 512 x 512	148 x 512 x 512	140 x 243 x 243
Patch size:	512 x 512	48 x 192 x 192	80 x 160 x 160
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 23. Table F.22: Multi Atlas Labeling Beyond the Cranial Vault Abdomen (D11) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. Postprocessing was applied to the model marked with *. This model (incl postprocessing) was used for test set predictions. Note that the Dice scores for the test set are computed with the online platform. Best ensemble on this dataset was the combination of the 3D U-Net cascade and the 3D full resolution U-Net.

	1	2	3	4	5	6	7	8
2D	0.8860	0.8131	0.8357	0.6406	0.7724	0.9453	0.8405	0.9128
3D_fullres	0.9083	0.8939	0.8675	0.6632	0.7840	0.9557	0.8816	0.9229
3D_lowres	0.9132	0.9045	0.9132	0.6525	0.7810	0.9554	0.8903	0.9209
3D cascade	0.9166	0.9069	0.9137	0.7036	0.7885	0.9587	0.9037	0.9215
Best Ensemble *	0.9135	0.9065	0.8971	0.6955	0.7897	0.9589	0.9026	0.9248
Postprocessed	0.9135	0.9065	0.8971	0.6959	0.7897	0.9590	0.9026	0.9248
Test set	0.9721	0.9182	0.9578	0.7528	0.8411	0.9769	0.9220	0.9290

Table 24. Table F.23: Network configurations generated by nnU-Net for the PROMISE12 challenge (D12). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.61 x 0.61	2.2 x 0.61 x 0.61	-
Median image shape at target spacing:	NA x 327 x 327	39 x 327 x 327	-
Patch size:	384 x 384	28 x 256 x 256	-
Batch size:	22	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2], [1, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 25. Table F.24: PROMISE12 (D12) results. Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the scores for the test set are computed with the online platform. The evaluation score of our test set submission is 89.6507. The test set Dice score reported in the table was computed from the detailed submission results (Detailed results available here https://promise12.grand-challenge.org/evaluation/results/89044a85-6c13-49f4-9742-dea65013e971/ ). Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	prostate	mean
2D	0.8932	0.8932
3D_fullres	0.8891	0.8891
Best Ensemble *	0.9029	0.9029
Postprocessed	0.9030	0.9030
Test set	0.9194	0.9194

Table 26. Table F.25: Network configurations generated by nnU-Net for the ACDC challenge (D13). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1.56 x 1.56	5 x 1.56 x 1.56	-
Median image shape at target spacing:	NA x 237 x 208	18 x 237 x 208	-
Patch size:	256 x 224	20 x 256 x 224	-
Batch size:	58	3	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2], [1, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 27. Table F.26: ACDC results (D13). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform. The online platform reports the Dice scores for enddiastolic and endsystolic time points separately. We averaged these values for a more condensed presentation. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	RV	MLV	LVC	mean
2D	0.9053	0.8991	0.9433	0.9159
3D_fullres	0.9059	0.9022	0.9458	0.9179
Best Ensemble *	0.9145	0.9059	0.9479	0.9227
Postprocessed	0.9145	0.9059	0.9479	0.9228
Test set	0.9295	0.9183	0.9407	0.9295

Table 28. Table F.27: Network configurations generated by nnU-Net for the LiTS challenge (D14). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.77 x 0.77	1 x 0.77 x 0.77	2.47 x 1.90 x 1.90
Median image shape at target spacing:	NA x 512 x 512	482 x 512 x 512	195 x 207 x 207
Patch size:	512 x 512	128 x 128 x 128	128 x 128 x 128
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 29. Table F.28: LiTS results (D14). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform. Best ensemble on this dataset was the combination of the 3D low resolution U-Net and the 3D full resolution U-Net.

	liver	cancer	mean
2D	0.9547	0.5603	0.7575
3D_fullres	0.9576	0.6253	0.7914
3D_lowres	0.9585	0.6161	0.7873
3D cascade	0.9609	0.6294	0.7951
Best Ensemble*	0.9618	0.6539	0.8078
Postprocessed	0.9631	0.6543	0.8087
Test set	0.9670	0.7630	0.8650

Table 30. Table F.29: Network configurations generated by nnU-Net for the MSLesion challenge (D15). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1 x 1	1 x 1 x 1	-
Median image shape at target spacing:	NA x 180 x 137	137 x 180 x 137	-
Patch size:	192 x 160	112 x 128 x 96	-
Batch size:	107	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 31. Table F.30: MSLesion results (D15). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. * marks the best performing model selected for subsequent postprocessing (see "Postprocessed") and test set submission (see "Test set"). Note that the Dice scores for the test set are computed with the online platform based on the detailed results (which are available here https://smart-stats-tools.org/sites/lesion_challenge/temp/top25/nnUNetV2_12032019_0903.csv ). The ranking is based on a score, which includes other metrics as well (see \citeSupplement SM_mslesionchallenge for details). The score of our submission is 92.874. Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	lesion	mean
2D	0.7339	0.7339
3D_fullres *	0.7531	0.7531
Best Ensemble	0.7494	0.7494
Postprocessed	0.7531	0.7531
Test set	0.6785	0.6785

Table 32. Table F.31: Network configurations generated by nnU-Net for the CHAOS challenge (D16). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2 .

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 1.66 x 1.66	5.95 x 1.66 x 1.66	-
Median image shape at target spacing:	NA x 195 x 262	45 x 195 x 262	-
Patch size:	224 x 320	40 x 192 x 256	-
Batch size:	45	2	-
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [1, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2], [1, 1, 2]]	-
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Table 33. Table F.32: CHAOS results (D16). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. Postprocessing was applied to the model marked with *. This model (incl postprocessing) was used for test set predictions. Note that the evaluation of the test set was performed with the online platform of the challenge which does not report Dice scores for the individual organs. The score of our submission was 72.44 for Task 5 and 75.10 for Task3 (see \citeSupplement SM_kavur2020chaos for details). Best ensemble on this dataset was the combination of the 2D U-Net and the 3D full resolution U-Net.

	liver	right kidney	left kidney	spleen	mean
2D	0.9132	0.8991	0.8897	0.8720	0.8935
3D_fullres	0.9202	0.9274	0.9209	0.8938	0.9156
Best Ensemble *	0.9184	0.9283	0.9255	0.8911	0.9158
Postprocessed	0.9345	0.9289	0.9212	0.894	0.9197
Test set	-	-	-	-	-

Table 34. Table F.33: Network configurations generated by nnU-Net for the KiTS challenge (D17). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.78 x 0.78	0.78 x 0.78 x 0.78	1.99 x 1.99 x 1.99
Median image shape at target spacing:	NA x 512 x 512	525 x 512 x 512	206 x 201 x 201
Patch size:	512 x 512	128 x 128 x 128	128 x 128 x 128
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 35. Table F.34: KiTS results (D17). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. Postprocessing was applied to the model marked with *. This model (incl postprocessing) was used for test set predictions. Note that the Dice scores for the test set are computed with the online platform which computes the kidney Dice score based of the union of the kidney and tumor labels whereas nnU-Net always evaluates labels independently, resulting in a missing value for kindey in the table. The reported kindey Dice by the platform (which is not comparable with the value computed by nnU-Net) is 0.9793. Best ensemble on this dataset was the combination of the 3D U-Net cascade and the 3D full resolution U-Net.

	Kidney	Tumor	mean
2D	0.9613	0.7563	0.8588
3D_fullres	0.9702	0.8367	0.9035
3D_lowres	0.9629	0.8420	0.9025
3D cascade	0.9702	0.8546	0.9124
Best Ensemble*	0.9707	0.8620	0.9163
Postprocessed	0.9707	0.8620	0.9163
Test set	-	0.8542	-

Table 36. Table F.35: Network configurations generated by nnU-Net for the SegTHOR challenge (D18). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	NA x 0.89 x 0.89	2.50 x 0.89 x 0.89	3.51 x 1.76 x 1.76
Median image shape at target spacing:	NA x 512 x 512	171 x 512 x 512	122 x 285 x 285
Patch size:	512 x 512	64 x 192 x 160	80 x 192 x 160
Batch size:	12	2	2
Downsampling strides:	[[2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]]	[[1, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]]	[[2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]
Convolution kernel sizes:	[[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]]	[[1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	[[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]

Table 37. Table F.36: SegTHOR results (D18). Note that all reported Dice scores (except the test set) were computed using five fold cross-validation on the training cases. Postprocessing was applied to the model marked with *. This model (incl postprocessing) was used for test set predictions. Note that the Dice scores for the test set are computed with the online platform. Best ensemble on this dataset was the combination of the 3D U-Net cascade and the 3D full resolution U-Net.

	esophagus	heart	trachea	aorta	mean
2D	0.8181	0.9407	0.9077	0.9277	0.8986
3D_fullres	0.8495	0.9527	0.9055	0.9426	0.9126
3D_lowres	0.8110	0.9464	0.8930	0.9284	0.8947
3D cascade	0.8553	0.9520	0.9045	0.9403	0.9130
Best Ensemble*	0.8545	0.9532	0.9066	0.9427	0.9143
Postprocessed	0.8545	0.9532	0.9083	0.9438	0.9150
Test set	0.8890	0.9570	0.9228	0.9510	0.9300

Table 38. Table F.37: Network configurations generated by nnU-Net for the CREMI challenge (D19). For more information on how to decode downsampling strides and kernel sizes into an architecture, see F.2

	2D U-Net	3D full resolution U-Net	3D low resolution U-Net
Target spacing (mm):	-	40 x 4 x 4	-
Median image shape at target spacing:	-	125 x 1250 x 1250	-
Patch size:	-	24 x 256 x256	-
Batch size:	-	2	-
Downsampling strides:	-	[[1, 2, 2], [1, 2, 2], [1, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]]	-
Convolution kernel sizes:	-	[[1, 3, 3], [1, 3, 3], [1, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]	-

Full text

††$\dagger$ Equal contribution. $\ddagger$ Work started while doing a PhD at the German Cancer Research Center.

1 Introduction

Semantic segmentation transforms raw biomedical image data into meaningful, spatially structured information and thus plays an essential role for scientific discovery in the field \citeManuscriptfalk2019u,hollon2020near. At the same time, semantic segmentation is an essential ingredient to numerous clinical applications \citeManuscriptaerts2014decoding,nestle2005comparison, including applications of artificial intelligence in diagnostic support systems \citeManuscriptde2018clinically,ACDCTMI, therapy planning support \citeManuscriptnikolov2018deep, intra-operative assistance \citeManuscripthollon2020near or tumor growth monitoring \citeManuscriptkickingereder2019automated. The high interest in automatic segmentation methods manifests in a thriving research landscape, accounting for $70\%$ of international image analysis competitions in the biomedical sector \citeManuscriptmaier2018rankings.

Despite the recent success of deep learning-based segmentation methods, their applicability to specific image analysis problems of end-users is often limited. The task-specific design and configuration of a method requires high levels of expertise and experience, with small errors leading to strong performance drops \citeManuscriptlitjens2017survey. Especially in 3D biomedical imaging, where dataset properties like imaging modality, image size, (anisotropic) voxel spacing or class ratio vary drastically, the pipeline design can be cumbersome, because experience on what constitutes a successful configuration may not translate to the dataset at hand. The numerous expert decisions involved in designing and training a neural network range from the exact network architecture to the training schedule and methods for data augmentation or post-processing. Each sub-component is controlled by essential hyperparameters like learning rate, batch size, or class sampling \citeManuscriptlitjens2017survey. An additional layer of complexity on the overall setup is posed by the hardware available for training and inference \citeManuscriptlecun20191. Algorithmic optimization of the codependent design choices in this high dimensional space of hyperparameters is technically demanding and amplifies both the number of required training cases as well as compute resources by orders of magnitude \citeManuscriptelsken2019neural. As a consequence, the end-user is commonly left with an iterative trial and error process during method design that is mostly driven by their individual experience, only scarcely documented and hard to replicate, inevitably evoking suboptimal segmentation pipelines and methodological findings that do not generalize to other datasets \citeManuscriptlitjens2017survey,bergstra2012random.

To further complicate things, there is an unmanageable number of research papers that propose architecture variations and extensions for performance improvement. This bulk of studies is incomprehensible to the non-expert and difficult to evaluate even for experts \citeManuscriptlitjens2017survey. Approximately 12000 studies cite the 2015 U-Net architecture on biomedical image segmentation \citeManuscriptronneberger2015u, many of which propose extensions and advances. We put forward the hypothesis that a basic U-Net is still hard to beat if the corresponding pipeline is designed adequately.

To this end, we propose nnU-Net (“no new net”), which makes successful 3D biomedical image segmentation accessible for biomedical research applications. nnU-Net automatically adapts to arbitrary datasets and enables out-of-the-box segmentation on account of two key contributions:

We formulate the pipeline optimization problem in terms of a data fingerprint (representing the key properties of a dataset) and a pipeline fingerprint (representing the key design choices of a segmentation algorithm). 2. 2.

We make their relation explicit by condensing domain knowledge into a set of heuristic rules that robustly generate a high quality pipeline fingerprint from a corresponding data fingerprint while considering associated hardware constraints.

In contrast to algorithmic approaches for method configuration that are formulated as a task-specific optimization problem, nnU-Net readily executes systematic rules to generate deep learning methods for previously unseen datasets without need for further optimization.

In the following, we demonstrate the superiority of this concept by presenting a new state of the art in numerous international challenges through application of our algorithm without manual intervention. The strong results underline the significance of nnU-Net for users who require algorithms for semantic segmentation on their custom datasets: as an open source tool, nnU-Net can simply be downloaded and trained out-of-the box to generate state of the art segmentations without requiring manual adaptation or expert knowledge. We further demonstrate shortcomings in the design process of current biomedical segmentation methods. Specifically, we take an in-depth look at the 2019 Kidney and Kidney Tumor Segmentation (KiTS) semantic image segmentation challenge and demonstrate how important task-specific design and configuration of a method are in comparison to choosing one of the many architectural extensions and advances previously proposed on top of the U-Net. By automating this design and configuration process, nnU-Net fosters the ambition and the ability of researchers to validate novel ideas on larger numbers of datasets, while at the same time serving as an ideal reference method when demonstrating methodological improvements.

2 Results

nnU-Net is a deep learning framework that enables 3D semantic segmentation in many biomedical imaging applications, without requiring the design of respective specialised solutions. Exemplary segmentation results generated by nnU-Net for a variety of datasets are shown in Figure 1.

nnU-Net automatically adapts to any new dataset

Figure 2a shows the current practice of adapting segmentation pipelines to a new dataset. This process is expert-driven and involves manual trial-and-error experiments that are typically specific to the task at hand \citeManuscriptlitjens2017survey. As shown in Figure 2b, nnU-Net addresses the adaptation process systematically. Therefore, we define a dataset fingerprint as a standardized dataset representation comprising key properties such as image sizes, voxel spacing information or class ratios, and a pipeline fingerprint as the entirety of choices being made during method design. nnU-Net is designed to generate a successful pipeline fingerprint for a given dataset fingerprint. In nnU-Net, the pipeline fingerprint is divided into three groups: blueprint, inferred and empirical parameters. The blueprint parameters represent fundamental design choices (such as using a plain U-Net-like architecture template) as well as hyperparameters for which a robust default value can simply be picked (for example loss function, training schedule and data augmentation). The inferred parameters encode the necessary adaptations to a new dataset and include modifications to the exact network topology, patch size, batch size and image preprocessing. The link between a data fingerprint and the inferred parameters is established via execution of a set of heuristic rules, without the need for expensive re-optimization when applied to unseen datasets. Note that many of these design choices are co-dependent: The target image spacing, for instance, affects image size, which in return determines the size of patches the model should see during training, which affects the network topology and has to be counterbalanced by the size of training mini-batches in order to not exceed GPU memory limitations. nnU-Net strips the user of the burden to manually account for these co-dependencies. The empirical parameters are autonomously identified via cross-validation on the training cases. Per default, nnU-Net generates three different U-Net configurations: a 2D U-Net, a 3D U-Net that operates at full image resolution and a 3D U-Net cascade where the first U-Net operates on downsampled images and the second is trained to refine the segmentation maps created by the former at full resolution. After cross-validation nnU-Net empirically chooses the best performing configuration or ensemble. Finally, nnU-Net empirically opts for “non-largest component suppression” as a postprocessing step if performance gains are measured. The output of nnU-Net’s automated adaptation and training process are fully trained U-Net models that can be deployed to make predictions on unseen images. We provide an in-depth description of the methodology behind nnU-Net in the online methods. The overarching design principles, i.e. our best-practice recommendations for method adaptation to new datasets, are summarized in Supplementary Information B. All segmentation pipelines generated by nnU-Net in the context of this manuscript are provided in Supplementary Information F.

nnU-Net handles a wide variety of target structures and image properties

We demonstrate the value of nnU-Net as an out-of-the-box segmentation tool by applying it to 10 international biomedical image segmentation challenges comprising 19 different datasets and 49 segmentation tasks across a variety of organs, organ substructures, tumors, lesions and cellular structures in magnetic resonance imaging (MRI), computed tomography scans (CT) as well as electron microscopy (EM) images. Challenges are international competitions that can be seen as the equivalent to clinical trials for algorithm benchmarking. Typically, they are hosted by individual researchers, institutes, or societies, aiming to assess the performance of multiple algorithms in a standardized environment \citeManuscriptmaier2018rankings. In all segmentation tasks, nnU-Net was trained from scratch using only the provided challenge data. While the methodology behind nnU-Net was developed on the 10 training sets provided by the Medical Segmentation Decathlon \citeManuscriptdecathlonDataPaper, the remaining datasets and tasks were used for independent testing, i.e. nnU-Net was simply applied without further optimization. Qualitatively, we observe that nnU-Net is able to handle a large disparity in dataset properties and diversity in target structures, i.e. generated pipeline configurations are in line with what human experts consider a reasonable or sensible setting (see Supplementary Information C.1and C.2). Examples for segmentation results generated by nnU-Net are presented in Figure 1.

nnU-Net outperforms specialized pipelines in a range of diverse tasks

Most international challenges use the Soerensen-Dice coefficient as a measure of overlap to quantify segmentation quality \citeManuscriptheller2019state,bilic2019liver, menze2014multimodal, ACDCTMI. Here, perfect agreement results in a Dice coefficient of 1, whereas no agreement results in a score of 0. Other metrics used by some of the challenges include the Normalized Surface Dice (higher is better) \citeManuscriptde2018clinically and the Hausdorff Distance (lower is better), both quantifying the distance between the borders of two segmentations. Figure 3 provides an overview of the quantitative results achieved by nnU-Net and the competing challenge teams across all 49 segmentation tasks. Despite its generic nature, nnU-Net outperforms most existing semantic segmentation solutions, even though the latter were specifically optimized towards the respective task. Overall, nnU-Net sets a new state of the art in 29 out of 49 target structures and otherwise shows performances on par with or close to the top leaderboard entries.

Details in pipeline configuration have more impact on performance than architectural variations

To highlight how important the task-specific design and configuration of a method are in comparison to choosing one of the many architectural extensions and advances previously proposed on top of the U-Net, we put our results into context of current research by analyzing the participating algorithms in the recent Kidney and Kidney Tumor Segmentation (KiTS) 2019 challenge hosted by the Medical Image Computing and Computer Assisted Intervention (MICCAI) society \citeManuscriptheller2019state. The MICCAI society has consistently been hosting at least $50\%$ of all annual biomedical image analysis challenges \citeManuscriptmaier2018rankings. With more than 100 competitors, the KiTS challenge was the largest competition at MICCAI 2019. Our analysis of the KiTS leaderboard111http://results.kits-challenge.org/miccai2019/ (see Figure 4a) reveals several insights on the current landscape of deep learning based segmentation method design: First, the top-15 methods were offspring of the (3D) U-Net architecture from 2016, confirming its impact on the field of biomedical image segmentation. Second, the figure demonstrates that contributions using the same type of network result in performances spread across the entire leaderboard. Third, when looking closer into the top-15, none of the commonly used architectural modifications (e.g. residual connections \citeManuscriptmilletari2016v,he2016deep, dense connections \citeManuscriptjegou2017one,huang2017densely, attention mechanisms \citeManuscriptoktay2018attention or dilated convolutions \citeManuscriptchen2017deeplab,mckinley2018ensembles) represent a necessary condition for good performance on the KiTS task. By example this shows that many of the previously introduced algorithm modifications may not generally be superior to a properly tuned baseline method.

Figure 4b underlines the importance of hyperparameter tuning by analyzing algorithms using the same architecture variant as the challenge-winning contribution, a 3D U-Net with residual connections. While one of these methods won the challenge, other contributions based on the same principle cover the entire range of evaluation scores and rankings. Key configuration parameters were selected from respective pipeline fingerprints and are shown for all non-cascaded residual U-Nets, illustrating the co-dependent design choices that each team made during pipeline design. The drastically varying configurations submitted by contestants indicate the underlying complexity of the high-dimensional optimization problem that is implicitly posed by designing a deep learning method for biomedical 3D image segmentation.

nnU-Net experimentally confirms the importance of good hyperparameters over architectural variations on the KiTS dataset by setting a new state of the art on the open leaderboard (which also includes the original challenge submissions analysed here) with a plain 3D U-Net architecture (see Figure 3). Our results from further international challenge participations confirm this observation across a variety of datasets.

Different datasets require different pipeline configurations

We extract the data fingerprints of 19 biomedical segmentation datasets. As displayed in Figure 5, this documents an exceptional dataset diversity in biomedical imaging, and reveals the fundamental reason behind the lack of out-of-the-box segmentation algorithms: The complexity of method design is amplified by the fact that suitable pipeline settings either directly or indirectly depend on the data fingerprint under potentially complex relations. As a consequence, pipeline settings that are identified as optimal for one dataset (such as KiTS, see above) may not generalize to others, resulting in a need for (currently manual) re-optimization on each individual dataset. An example for configuration parameters depending on dataset properties is the image size which affects the size of patches that the model sees during training, which in turn affects the required network topology (i.e. number of downsampling steps, size of convolution filters, etc.). The network topology itself again influences several other hyperparameters in the pipeline.

Multiple tasks enable robust design decisions

nnU-Net is a framework that enables benchmarking of new modifications or extensions of methods across multiple datasets without having to manually reconfigure the entire pipeline for each dataset. To demonstrate this, and also to support some of the core design choices made in nnU-Net, we systematically tested the performance of common pipeline variations in the nnU-Net blueprint parameters on 10 different datasets (Figure 6): the application of two alternative loss functions (Cross-entropy and TopK10 \citeManuscriptwu2016bridging), the introduction of residual connections in the encoder \citeManuscripthe2016identity, using three convolutions per resolution instead of two (resulting in a deeper network architecture), two modifications of the optimizer (a reduced momentum term and an alternative optimizer (Adam \citeManuscriptadam)), batch norm \citeManuscriptioffe2015batch instead of instance norm \citeManuscriptulyanov2016instance and the omission of data augmentation. Ranking stability was estimated by bootstrapping as suggested by the challengeR tool \citeManuscriptwiesenfarth2019methods.

The volatility of the ranking between datasets demonstrates how single hyperparameter choices can affect segmentation performance depending on the dataset. The results clearly show that caution is required when drawing methodological conclusions from evaluations that are based on an insufficient number of datasets. While five out of the nine variants achieved rank 1 in at least one of the datasets, neither of them exhibits consistent improvements across the ten tasks. The original nnU-Net configuration shows the best generalization and ranks first when aggregating results of all datasets.

In current research practice, evaluation is rarely performed on more than two datasets and even then the datasets come with largely overlapping properties (such as both being abdominal CT scans). As we showed here, such evaluation is unsuitable for drawing general methodological conclusions. We relate the lack of sufficiently broad evaluations to the manual tuning effort required when adapting existing pipelines to individual datasets. nnU-Net alleviates this shortcoming in two ways: As a framework that can be extended to enable effective evaluation of new concepts across multiple tasks, and as a plug-and-play, standardized and state-of-the-art baseline to compare against.

nnU-Net is freely available and can be used out-of-the-box

nnU-Net is freely available as an open-source tool. It can be installed via Python Package Index (PyPI). The source code is publicly available on Github (https://github.com/MIC-DKFZ/nnUNet). A comprehensive documentation is available together with the source code. Pretrained models for all presented datasets are available for download at https://zenodo.org/record/3734294.

3 Discussion

We presented nnU-Net, a deep learning framework for biomedical image analysis that automates model design for 3D semantic segmentation tasks. The method sets a new state of the art in the majority of tasks it was evaluated on, outperforming all respective specialized processing pipelines. The strong performance of nnU-Net is not achieved by a new network architecture, loss function or training scheme (hence the name nnU-Net - “no new net”), but by replacing the complex process of manual pipeline optimization with a systematic approach based on explicit and interpretable heuristic rules. Requiring zero user-intervention, nnU-Net is the first segmentation tool that can be applied out-of-the-box to a very large range of biomedical imaging datasets and is thus the ideal tool for users who require access to semantic segmentation methods and do not have the expertise, time, or compute resources required to manually adapt existing solutions to their problem.

Our analysis on the KITS leaderboard as well as nnU-Net’s performance across 19 datasets confirms our initial hypothesis that common architectural modifications proposed by the field during the last 5 years may not necessarily be required to achieve state-of-the-art segmentation performance. Instead, we observed that contributions using the same type of network result in performances spread across the entire leaderboard. This observation is in line with Litjens et al., who, in their review from 2017, found that "many researchers use the exact same architectures […] but have widely varying results" \citeManuscriptlitjens2017survey. There are several possible reasons for why performance improvements based on architectural extensions proposed by the literature may not hold beyond the dataset they were proposed on: many of them are evaluated on a limited amount of datasets, often as low as a single one. In practice this largely limits their success on unseen datasets with varying properties, because the quality of the hyperparameter configuration often overshadows the effect of the evaluated architectural modification. This finding is in line with an observation by Litjens et al., who concluded that "the exact architecture is not the most important determinant in getting a good solution" \citeManuscriptlitjens2017survey. Moreover, as shown above, it can be difficult to tune existing baselines to a given dataset. This obstacle can unknowingly, but nonetheless unduly, make a new approach look better than the baseline, resulting in biased literature.

In this work, we demonstrated that nnU-Net is able to alleviate this bottleneck of current research in biomedical image segmentation in two ways: On the one hand, nnU-Net serves as a framework for methodological modifications enabling simple evaluation on an arbitrary number of datasets. On the other hand, nnU-Net represents the first standardized method that does not require manual task-specific adaptation and as such can readily serve as a strong baseline on any new 3D segmentation task.

The research performed in “AutoML” \citeManuscripthutter2011sequential,cubuk2019autoaugment or “Neural architecture search” \citeManuscriptelsken2019neural has similarities to our approach in that this line of research seeks to strip the ML user or researcher of the burden to manually find good hyperparameters. In contrast to nnU-Net however, AutoML aims to learn hyperparameters directly from the data. This comes with practical difficulties such as enormous requirements with respect to compute and data resources. Additionally, AutoML methods need to optimize the hyperparameters for each new task. The same disadvantages apply to “Grid Search” \citeManuscriptbergstra2012random, where extensive trial and error sweeps in the hyperparameter landscape are performed to empirically find good configurations for a specific task. In contrast, nnU-Net transforms domain knowledge into inductive biases, thus shortcuts the high dimensional optimization of hyperparameters and minimizes required computational and data resources. As elaborated above, these heuristics are developed on the basis of 10 different datasets of the Medical Segmentation Decathlon. The diversity within these 10 datasets has proven sufficient to achieve robustness to the variability encountered in all the remaining challenge participations. This is quite remarkable given the underlying complexity of method design and strongly confirms the suitability of condensing the process in a few generally applicable rules that are simply executed when given a new dataset fingerprint and do not require any further task-specific actions. The formal definition and also publishing of these explicit rules is a step towards systematicity and interpretability in the task of hyperparameter selection, which has previously been considered a “highly empirical exercise”, for which “no clear recipe can be given.” \citeManuscriptlitjens2017survey.

Despite its strong performance across 49 diverse tasks, there might be segmentation tasks for which nnU-Net’s automatic adaptation is suboptimal. For example, nnU-Net was developed with a focus on the Dice coefficient as performance metric. Some tasks, however, might require highly domain specific target metrics for performance evaluation, which could influence method design. Also, yet unconsidered dataset properties could exist which may cause suboptimal segmentation performance. One example is the synaptic cleft segmentation task of the CREMI challenge (https://cremi.org). While nnU-Net’s performance is highly competitive (rank 6/39), manual adaptation of the loss function as well as electron microscopy-specific preprocessing may be necessary to surpass state-of-the-art performance \citeManuscriptheinrich2018synaptic. In principle, there are two ways of handling cases that are not yet optimally covered by nnU-Net: For potentially re-occurring cases, nnU-Net’s heuristics could be extended accordingly; for highly domain specific cases, nnU-Net should be seen as a good starting point for necessary modifications.

In summary, nnU-Net sets a new state of the art in various semantic segmentation challenges and displays strong generalization characteristics without need for any manual intervention, such as the tuning of hyper-parameters. As pointed out by Litjens et al. and quantitatively confirmed here, hyper-parameter optimization constitutes a major difficulty for past and current research in biomedical image segmentation. nnU-Net automates the otherwise often unsystematic and cumbersome procedure and may thus help alleviate this burden. We propose to leverage nnU-Net as an out-of-the box tool for state-of-the-art segmentation, a framework for large-scale evaluation of novel ideas without manual effort, and as a standardized baseline method to compare ideas against without the need for task-specific optimization.

4 Acknowledgements

This work was co-funded by the National Center for Tumor Diseases (NCT) in Heidelberg and the Helmholtz Imaging Platform (HIP) of the German Cancer Consortium (DKTK). We thank our colleagues at DKFZ who were involved in the various challenge contributions, especially Andre Klein, David Zimmerer, Jakob Wasserthal, Gregor Koehler, Tobias Norajitra and Sebastian Wirkert who contributed to the Decathlon submission. We also thank the MITK team who supported us in producing all medical dataset visualizations. We are also thankful to all the challenge organizers, who provided an important basis for our work. We want to especially mention Nicholas Heller, who enabled the collection of all the details from the KiTS challenge through excellent challenge design, and Emre Kavur from the CHAOS team, who generated comprehensive leaderboard information for us. We thank Manuel Wiesenfarth for his helpful advice concerning the ranking of methods and the visualization of rankings. Last but not least, we thank Olaf Ronneberger and Lena Maier-Hein for their important feedback on this manuscript.

Methods

A quick overview of the nnU-Net design principles can be found in the Supplemental Material B. This section provides detailed information on how these principles are implemented.

Dataset fingerprints

As a first processing step, nnU-Net crops the provided training cases to their nonzero region. While this had no effect on most datasets in our experiments, it reduced the image size of brain datasets such as D1 (Brain Tumor) and D15 (MSLes) substantially and thus improved computational efficiency. Based on the cropped training data, nnU-Net creates a dataset fingerprint that captures all relevant parameters and properties: image sizes (i.e. number of voxels per spatial dimension) before and after cropping, image spacings (i.e. the physical size of the voxels), modalities (read from metadata) and number of classes for all images as well as the total number of training cases. Furthermore, the fingerprint includes the mean, standard deviation as well as the 0.5 and 99.5 percentiles of the intensity values in the foreground regions, i.e. the voxels belonging to any of the class labels, computed over all training cases.

Pipeline fingerprints

nnU-Net automizes the design of deep learning methods for biomedical image segmentation by generating a so-called pipeline fingerprint that contains all relevant information. Importantly, nnU-Net reduces the design choices to the really essential ones and automatically infers these choices using a set of heuristic rules. These rules condense the domain knowledge and operate on the above-described data fingerprint and the project-specific hardware constraints. These inferred parameters are complemented by blueprint parameters, which are data-independent, and empirical parameters, which are optimized during training.

Blueprint parameters

Architecture template: All U-Net architectures configured by nnU-Net originate from the same template. This template closely follows the original U-Net [16] and its 3D counterpart [3]. According to our hypothesis that a well-configured plain U-Net is still hard to beat, none of our U-Net configurations make use of recently proposed architectural variations such as residual connections [6, 7], dense connections [10, 12], attention mechanisms [14], squeeze and excitation [9] or dilated convolutions [2]. Minor changes with respect to the original architecture were made: To enable large patch sizes, the batch size of the networks in nnU-Net is small. In fact, most 3D U-Net configurations were trained with a batch size of only 2 (see Supplementary Material Figure E.1a). Batch normalization [11], which is often used to speed up or stabilize the training, does not perform well with small batch sizes [20, 17]. We therefore use instance normalization [19] for all U-Net models. Furthermore, we replace ReLU with leaky ReLUs [13] (negative slope 0.01). Networks are trained with deep supervision: additional auxiliary losses are added in the decoder to all but the two lowest resolutions, allowing gradients to be injected deeper into the network and facilitating the training of all layers in the network. All U-Nets employ the very common configuration of two blocks per resolution step in both encoder and decoder, with each block consisting of a convolution, followed by instance normalization and a leaky ReLU nonlinearity. Downsampling is implemented as strided convolution (motivated by representational bottleneck, see [18]) and upsampling as convolution transposed. As a tradeoff between performance and memory consumption, the initial number of feature maps is set to 32 and doubled (halved) with each downsampling (upsampling) operation. To limit the final model size, the number of feature maps is additionally capped at 320 and 512 for 3D and 2D U-Nets, respectively.

Training schedule: Based on experience and as a trade-off between runtime and reward, all networks are trained for 1000 epochs with one epoch being defined as iteration over 250 minibatches. Stochastic gradient descent with nesterov momentum ($\mu=0.99$) and an initial learning rate of 0.01 is used for learning network weights. The learning rate is decayed throughout the training following the ‘poly’ learning rate policy [2]: $(1-\mathrm{epoch}/\mathrm{epoch}_{\mathrm{max}})^{0.9}$. The loss function is the sum of cross-entropy and Dice loss [4]. For each deep supervision output, a corresponding downsampled ground truth segmentation mask is used for loss computation. The training objective is the sum of the losses at all resolutions: $L=w_{1}\cdot L_{1}+w_{2}\cdot L_{2}+…$ . Hereby, the weights halve with each decrease in resolution, resulting in $w_{2}=1/2\cdot w_{1};w_{3}=1/4\cdot w_{1}$, etc. and are normalized to sum to 1. Samples for the mini batches are chosen from random training cases. Oversampling is implemented to ensure robust handling of class imbalances: $66.7\%$ of samples are from random locations within the selected training case while $33.3\%$ of patches are guaranteed to contain one of the foreground classes that are present in the selected training sample (randomly selected). The number of foreground patches is rounded with a forced minimum of 1 (resulting in 1 random and 1 foreground patch with batch size 2). A variety of data augmentation techniques are applied on the fly during training: rotations, scaling, Gaussian noise, Gaussian blur, brightness, contrast, simulation of low resolution, gamma and mirroring. Details are provided in Supplementary Information D.

Inference: Images are predicted with a sliding window approach, where the window size equals the patch size used during training. Adjacent predictions overlap by half the size of a patch. The accuracy of segmentation decreases towards the borders of the window. To suppress stitching artifacts and reduce the influence of positions close to the borders, a Gaussian importance weighting is applied, increasing the weight of the center voxels in the softmax aggregation. Test time augmentation by mirroring along all axes is applied.

Inferred Parameters

Intensity normalization: There are two different image intensity normalization schemes supported by nnU-Net. The default setting for all modalities except CT images is z-scoring. For this option, during training and inference, each image is normalized independently by subtracting its mean, followed by division with its standard deviation. If cropping resulted in an average size decrease of $25\%$ or more, a mask for central non-zero voxels is created and the normalization is applied within that mask only, ignoring the surrounding zero voxels. For computed tomography (CT) images, nnU-Net employs a different scheme, as intensity values are quantitative and reflect physical properties of the tissue. It can therefore be beneficial to retain this information by using a global normalization scheme that is applied to all images. To this end, nnU-Net uses the 0.5 and 99.5 percentiles of the foreground voxels for clipping as well as the global foreground mean a standard deviation for normalization on all images.

Resampling: In some datasets, particularly in the medical domain, the voxel spacing (the physical space the voxels represent) is heterogeneous. Convolutional neural networks operate on voxel grids and ignore this information. To cope with this heterogeneity, nnU-Net resamples all images to the same target spacing (see paragraph below) using either third order spline, linear or nearest neighbor interpolation. The default setting for image data is third order spline interpolation. For anisotropic images (maximum axis spacing / minimum axis spacing > 3), in-plane resampling is done with third order spline whereas out of plane interpolation is done with nearest neighbor. Treating the out of plane axis differently in anisotropic cases suppresses resampling artifacts, as large contour changes between slices are much more common. Segmentation maps are resampled by converting them to one hot encodings. Each channel is then interpolated with linear interpolation and the segmentation mask is retrieved by an argmax operation. Again, anisotropic cases are interpolated using “nearest neighbor” on the low resolution axis.

Target spacing: The selected target spacing is a crucial parameter. Larger spacings result in smaller images and thus a loss of details whereas smaller spacings result in larger images preventing the network from accumulating sufficient contextual information since the patch size is limited by the given GPU memory budget. Although this tradeoff is in part addressed by the 3D U-Net cascade (see below), a sensible target spacing for low and full resolution is still required. For the 3D full resolution U-Net, nnU-Net uses the median value of the spacings found in the training cases computed independently for each axis as default target spacing. For anisotropic datasets, this default can result in severe interpolation artifacts or in a substantial loss of information due to large variances in resolution across the training data. Therefore, the target spacing of the lowest resolution axis is selected to be the 10th percentile of the spacings found in the training cases if both voxel and spacing anisotropy (i.e. the ratio of lowest spacing axis to highest spacing axis) are larger than 3. For the 2D U-Net, nnU-Net generally operates on the two axes with the highest resolution. If all three axes are isotropic, the two trailing axes are utilized for slice extraction. The target spacing is the median spacing of the training cases (computed independently for each axis). For slice-based processing, no resampling along the out-of-plane axis is required.

Adaptation of network topology, patch size and batch size: Finding an appropriate U-Net architecture configuration is crucial for good segmentation performance. nnU-Net prioritizes large patch sizes while remaining within a predefined GPU memory budget. Larger patch sizes allow for more contextual information to be aggregated and thus typically increase segmentation performance. They come, however, at the cost of a decreased batch size which results in noisier gradients during backpropagation. To improve the stability of the training, we require a minimum batch size of 2 and choose a large momentum term for network training (see blueprint parameters). Image spacing is also considered in the adaptation process: Downsampling operations may operate only on specific axes and convolutional kernels in the 3D U-Nets can operate on certain image planes only (pseudo-2D). The network topology for all U-Net configurations is chosen on basis of the median image size after resampling as well as the target spacing the images were resampled to. A flow chart for the adaptation process is presented in the Supplements in Figure E.1. The adaptation of the architecture template, which is described in more detail in the following, is computationally inexpensive. Due to the GPU memory consumption estimate being based on feature map sizes, no GPU is required to run the adaptation process.

Initialization: The patch size is initialized as the median image shape after resampling. If the patch size is not divisible by $2^{n_{d}}$ for each axis, where $n_{d}$ is the number of downsampling operations, it is padded accordingly.

Architecture topology: The architecture is configured by determining the number of downsampling operations along each axis depending on the patch size and voxel spacing. Downsampling is performed until further downsampling would reduce the feature map size to smaller than 4 voxels or the feature map spacings become anisotropic. The downsampling strategy is determined by the voxel spacing: high resolution axes are downsampled separately until their resolution is within factor 2 of the lower resolution axis. Subsequently, all axes are downsampled simultaneously. Downsampling is terminated for each axis individually, once the respective feature map constraint is triggered. The default kernel size for convolutions is $3\times 3\times 3$ and $3\times 3$ for 3D U-Net and 2D U-Net, respectively. If there is an initial resolution discrepancy between axes (defined as a spacing ratio larger than 2), the kernel size for the out-of-plane axis is set to 1 until the resolutions are within a factor of 2. Note that the convolutional kernel size then remains at 3 for all axes.

Adaptation to GPU memory budget: The largest possible patch size during configuration is limited by the amount of GPU memory. Since the patch size is initialized to the median image shape after resampling, it is initially too large to fit into the GPU for most datasets. nnU-Net estimates the memory consumption of a given architecture based on the size of the feature maps in the network, comparing it to reference values of known memory consumption. The patch size is then reduced in an iterative process while updating the architecture configuration accordingly in each step until the required budget is reached (see Figure E.1 in the Supplements). The reduction of the patch size is always applied to the largest axis relative to the median image shape of the data. The reduction in one step amounts to $2^{n_{d}}$ voxels of that axis, where $n_{d}$ is the number of downsampling operations.

Batch size: As a final step, the batch size is configured. If a reduction of patch size was performed the batch size is set to 2. Otherwise, the remaining GPU memory headroom is utilized to increase the batch size until the GPU is fully utilized. To prevent overfitting, the batch size is capped such that the total number of voxels in the minibatch do not exceed $5\%$ of the total number of voxels of all training cases. Examples for generated U-Net architectures are presented in Supplementary Information C.1 and C.2.

Configuration of 3D U-Net cascade: Running a segmentation model on downsampled data increases the size of patches in relation to the image and thus enables the network to accumulate more contextual information. This comes at the cost of a reduction in details in the generated segmentations and may also cause errors if the segmentation target is very small or characterized by its texture. In a hypothetical scenario with unlimited GPU memory, it is thus generally favored to train models at full resolution with a patch size that covers the entire image. The 3D U-Net cascade approximates this approach by first running a 3D U-Net on downsampled images and then training a second, full resolution 3D U-Net to refine the segmentation maps of the former. This way, the “global”, low resolution network utilizes maximal contextual information to generate its segmentation output, which then serves as an additional input channel that guides the second, “local” U-Net. The cascade is triggered only for datasets where the patch size of the 3d full resolution U-Net covers less than $12.5\%$ of the median image shape. If this is the case, the target spacing for the downsampled data and the architecture of the associated 3D low resolution U-Net are configured jointly in an iterative process. The target spacing is initialized as the target spacing of the full resolution data. In order for the patch size to cover a large proportion of the input image, the target spacing is then increased stepwise by $1\%$ while updating the architecture configuration accordingly in each step until the patch size of the resulting network topology surpasses $25\%$ of the current median image shape. If the current spacing is anisotropic (factor 2 difference between lowest and highest resolution axis), only the spacing of the higher resolution axes is increased. The configuration of the second 3D U-Net of the cascade is identical to the standalone 3D U-Net for which the configuration process is described above (except that the upsampled segmentation maps of the first U-Net are concatenated to its input). Figure E.1b in the Supplements provides an overview of this optimization process.

Empirical parameters

Ensembling and selection of U-Net configuration(s): nnU-Net automatically determines which (ensemble of) configuration(s) to use for inference based on the average foreground Dice coefficient computed via cross-validation on the training data. The selected model(s) can be either a single U-Net (2D, 3D full resolution, 3D low resolution or the full resolution U-Net of the cascade) or an ensemble of any two of these configurations. Models are ensembled by averaging softmax probabilities.

Postprocessing: Connected component-based postprocessing is commonly used in medical image segmentation [1, 8]. Especially in organ segmentation it often helps to remove spurious false positive detections by removing all but the largest connected component. nnU-Net follows this assumption and automatically benchmarks the effect of suppressing smaller components on the cross-validation results. First, all foreground classes are treated as one component. If suppression of all but the largest region improves the average foreground Dice coefficient and does not reduce the Dice coefficient for any of the classes, this procedure is selected as the first postprocessing step. Finally, nnU-Net builds on the outcome of this step and decides whether the same procedure should be performed for individual classes.

Implementation details

nnU-Net is implemented in Python utilizing the PyTorch [15] framework. The Batchgenerators library [5] is used for data augmentation. For reduction of computational burden and GPU memory footprint, mixed precision training is implemented with Nvidia Apex/Amp (https://github.com/NVIDIA/apex). For use as a framework, the source code is available on GitHub (https://github.com/MIC-DKFZ/nnUNet). Users who seek to use nnU-Net as a standardized benchmark or to run inference with our pretrained models can install nnU-Net via PyPI. For a full description of how to use nnU-Net, please refer to the online documentation available on the GitHub page.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Code availability

The nnU-Net repository is available at: https://github.com/mic-dkfz/nnunet. Pre-traiend models for all datasets utilized in this study are available for download at https://zenodo.org/record/3734294.

Data availability

All 19 datasets used in this study are publicly available. References for web access as well as key data properties can be found in the Supplementary Material A and F.