The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery

TL;DR

This study investigates how super-resolution techniques improve object detection accuracy in satellite imagery across various resolutions, demonstrating significant gains at finer resolutions and modest benefits at coarser levels.

Contribution

It introduces a comprehensive evaluation of super-resolution effects on satellite object detection, combining VDSR and RFSR methods with a unified detection framework across multiple resolutions.

Findings

Super-resolution improves detection accuracy at 30 cm resolution by up to 36%.

Detection performance drops significantly as native resolution coarsens.

Super-resolution to 15 cm yields the most substantial accuracy gains.

Abstract

We explore the application of super-resolution techniques to satellite imagery, and the effects of these techniques on object detection algorithm performance. Specifically, we enhance satellite imagery beyond its native resolution, and test if we can identify various types of vehicles, planes, and boats with greater accuracy than native resolution. Using the Very Deep Super-Resolution (VDSR) framework and a custom Random Forest Super-Resolution (RFSR) framework we generate enhancement levels of 2x, 4x, and 8x over five distinct resolutions ranging from 30 cm to 4.8 meters. Using both native and super-resolved data, we then train several custom detection models using the SIMRDWN object detection framework. SIMRDWN combines a number of popular object detection algorithms (e.g. SSD, YOLO) into a unified framework that is designed to rapidly detect objects in large satellite images. This approach allows us to quantify the effects of super-resolution techniques on object detection performance across multiple classes and resolutions. We also quantify the performance of object detection as a function of native resolution and object pixel size. For our test set we note that performance degrades from mean average precision (mAP) = 0.53 at 30 cm resolution, down to mAP = 0.11 at 4.8 m resolution. Super-resolving native 30 cm imagery to 15 cm yields the greatest benefit; a 13-36% improvement in mAP. Super-resolution is less beneficial at coarser resolutions, though still provides a small improvement in performance.

Tables

Table 1. Table 1: Object Counts

Category	Mean Size	Counts
	(meters)	Train	Test	Total
Boat	16.5	2379	2347	4726
Large Aircraft	36.9	424	294	718
Small Aircraft	13.2	264	178	442
Bus/Truck	8.8	19337	13269	32606
Small Vehicle	4.7	129438	89923	219361

Table 2. Table 2: Average inference time per 544 × 544\times 544 pixel image and training time for a set of 1,500 images at native 30 cm GSD resolution. RFSR used a 64GB RAM CPU and VDSR used a NVIDIA Titan Xp GPU for inference and training.

	VDSR	RFSR
Inference Time
(per image)	0.16 seconds	0.7 seconds
Training Time
(for 2, 4, $8\times$)	55.9 hours	10.8 hours

Table 3. Table 3 : Average PSNR / SSIM scores for scale 2 × 2\times , 4 × 4\times , and 8 × 8\times across five super-resolution output GSDs. All test imagery is the xView validation dataset (281 images). Bicubic indicates the scores if LR images are just upscaled using bicubic interpolation to match the HR image size.

GSDout	Scale	Bicubic	VDSR	RFSR
30cm	$2\times$	38.68 / 0.8108	42.39 / 0.8925	39.79 / 0.8582
30cm	$4\times$	35.86 / 0.6610	38.79 / 0.7795	35.85 / 0.7064
30cm	$8\times$	33.82 / 0.5394	35.69 / 0.6117	34.32 / 0.5874
60cm	$2\times$	41.26 / 0.9275	45.08 / 0.9635	43.03 / 0.9408
60cm	$4\times$	36.98 / 0.8082	40.50 / 0.8904	37.41 / 0.8330
60cm	$8\times$	33.99 / 0.6771	35.44 / 0.7293	33.78 / 0.6799
1.2m	$2\times$	36.73 / 0.9151	39.33 / 0.9497	38.17 / 0.9448
1.2m	$4\times$	32.49 / 0.7738	35.25 / 0.8633	33.47 / 0.8332
1.2m	$8\times$	29.41 / 0.6097	30.58 / 0.6709	29.84 / 0.6700
2.4m	$2\times$	35.26 / 0.8848	41.50 / 0.9624	36.67 / 0.9250
2.4m	$4\times$	31.09 / 0.6898	33.75 / 0.8117	32.00 / 0.7659
2.4m	$8\times$	28.46 / 0.5004	30.78 / 0.6089	28.87 / 0.5572
4.8m	$2\times$	34.14 / 0.8404	37.01 / 0.9097	35.45 / 0.8953
4.8m	$4\times$	30.42 / 0.6079	33.13 / 0.7527	31.24 / 0.6934
4.8m	$8\times$	27.98 / 0.4013	30.22 / 0.5110	28.39 / 0.4488

Table 4. Table 4 : Performance for each data type in mAP. For both RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ). The native sensor resolution of our original imagery and the input into the super-resolution models is shown on the X-axis. We then compare the super-resolved outputs vs. the original native imagery to test the change in object detection performance.

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.53\pm 0.03$	$0.49\pm 0.03$	$0.41\pm 0.03$	$0.21\pm 0.02$	$0.11\pm 0.01$
YOLT	RFSR $2\times$	$0.60\pm 0.03$ (+1.7$\sigma$)	$0.52\pm 0.03$ (+0.7$\sigma$)	$0.39\pm 0.03$ (-0.5$\sigma$)	$0.24\pm 0.02$ (+1.1$\sigma$)	$0.12\pm 0.01$ (+0.7$\sigma$)
YOLT	VDSR $2\times$	$0.60\pm 0.03$ (+1.7$\sigma$)	$0.52\pm 0.03$ (+0.7$\sigma$)	$0.41\pm 0.03$ (+0.0$\sigma$)	$0.22\pm 0.01$ (+0.4$\sigma$)	$0.13\pm 0.01$ (+1.4$\sigma$)
YOLT	RFSR $4\times$		$0.56\pm 0.03$ (+1.6$\sigma$)	$0.40\pm 0.03$ (-0.2$\sigma$)	$0.23\pm 0.01$ (+0.9$\sigma$)	$0.12\pm 0.01$ (+0.7$\sigma$)
YOLT	VDSR $4\times$		$0.59\pm 0.02$ (+2.8$\sigma$)	$0.39\pm 0.03$ (-0.5$\sigma$)	$0.25\pm 0.02$ (+1.4$\sigma$)	$0.10\pm 0.01$ (-0.7$\sigma$)
SSD	Native	$0.30\pm 0.01$	$0.32\pm 0.01$	$0.22\pm 0.01$	$0.08\pm 0.01$	$0.00\pm 0.00$
SSD	RFSR $2\times$	$0.36\pm 0.01$ (+4.2$\sigma$)	$0.33\pm 0.02$ (+0.4$\sigma$)	$0.24\pm 0.01$ (+1.4$\sigma$)	$0.13\pm 0.01$ (+3.5$\sigma$)	$0.07\pm 0.01$ (+7.0$\sigma$)
SSD	VDSR $2\times$	$0.41\pm 0.03$ (+3.5$\sigma$)	$0.32\pm 0.02$ (+0.0$\sigma$)	$0.26\pm 0.01$ (+2.8$\sigma$)	$0.14\pm 0.01$ (+4.2$\sigma$)	$0.08\pm 0.01$ (+8.0$\sigma$)

Table 5. Table 1 : Average PSNR / SSIM scores for scale 2 × 2\times , 4 × 4\times , and 8 × 8\times across five super-resolution output GSDs. All test imagery is not blurred, decimated bicubically, and then bicubically upsampled. (Ideal or Traditional super-resolution settings). As in the main text the xView validation dataset is used (281 images). Bicubic indicates the scores if LR images are just upscaled using bicubic interpolation to match the HR image size.

GSDout	Scale	Bicubic	VDSR	RFSR
30cm	×2	40.30 / 0.8734	42.95 / 0.9104	40.90 / 0.8885
30cm	×4	36.83 / 0.7265	39.07 / 0.7939	36.81 / 0.7419
30cm	×8	34.69 / 0.5930	36.86 / 0.6605	35.06 / 0.6092
60cm	×2	43.69 / 0.9594	45.66 / 0.9717	44.25 / 0.9496
60cm	×4	38.61 / 0.8656	41.07 / 0.9055	38.89 / 0.8729
60cm	×8	35.20 / 0.7324	37.61 / 0.7861	35.18 / 0.7439
1.2m	×2	40.73 / 0.9640	43.14 / 0.9777	41.70 / 0.9716
1.2m	×4	34.73 / 0.8562	37.39 / 0.9006	35.23 / 0.8773
1.2m	×8	30.92 / 0.6862	33.16 / 0.7500	31.19 / 0.7197
2.4m	×2	39.33 / 0.9529	42.15 / 0.9720	40.36 / 0.9640
2.4m	×4	33.24 / 0.7998	36.00 / 0.8638	33.74 / 0.8326
2.4m	×8	29.68 / 0.5802	31.90 / 0.6566	29.92 / 0.6177
4.8m	×2	37.99 / 0.9320	40.94 / 0.9592	38.98 / 0.9491
4.8m	×4	32.23 / 0.7294	34.87 / 0.8070	32.66 / 0.7703
4.8m	×8	29.01 / 0.4786	31.15 / 0.5552	29.19 / 0.5131

Table 6. Table 2 : Performance for each data type in mAP. For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ). The native sensor resolution of our original imagery and the input into the super-resolution models is shown on the X-axis. We then compare the super-resolved outputs vs. the original native imagery to test the change in object detection performance.

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.53\pm 0.03$	$0.49\pm 0.03$	$0.41\pm 0.03$	$0.21\pm 0.02$	$0.11\pm 0.01$
YOLT	RFSR $2\times$	$0.60\pm 0.03$ (+1.6$\sigma$)	$0.52\pm 0.03$ (+0.8$\sigma$)	$0.39\pm 0.03$ (-0.6$\sigma$)	$0.24\pm 0.02$ (+1.0$\sigma$)	$0.12\pm 0.01$ (+1.0$\sigma$)
YOLT	VDSR $2\times$	$0.60\pm 0.03$ (+1.7$\sigma$)	$0.52\pm 0.03$ (+0.8$\sigma$)	$0.41\pm 0.03$ (-0.1$\sigma$)	$0.22\pm 0.01$ (+0.3$\sigma$)	$0.13\pm 0.01$ (+1.4$\sigma$)
YOLT	RFSR $4\times$		$0.56\pm 0.03$ (+1.9$\sigma$)	$0.40\pm 0.03$ (-0.2$\sigma$)	$0.23\pm 0.01$ (+0.8$\sigma$)	$0.12\pm 0.01$ (+1.4$\sigma$)
YOLT	VDSR $4\times$		$0.59\pm 0.02$ (+3.0$\sigma$)	$0.39\pm 0.03$ (-0.6$\sigma$)	$0.25\pm 0.02$ (+1.7$\sigma$)	$0.10\pm 0.01$ (-0.3$\sigma$)
SSD	Native	$0.30\pm 0.01$	$0.32\pm 0.01$	$0.22\pm 0.01$	$0.08\pm 0.01$	$0.00\pm 0.00$
SSD	RFSR $2\times$	$0.36\pm 0.01$ (+2.6$\sigma$)	$0.33\pm 0.02$ (+0.7$\sigma$)	$0.24\pm 0.01$ (+1.1$\sigma$)	$0.13\pm 0.01$ (+3.2$\sigma$)	$0.07\pm 0.01$ (+7.0$\sigma$)
SSD	VDSR $2\times$	$0.41\pm 0.03$ (+3.5$\sigma$)	$0.32\pm 0.02$ (-0.0$\sigma$)	$0.26\pm 0.01$ (+2.3$\sigma$)	$0.14\pm 0.01$ (+3.9$\sigma$)	$0.08\pm 0.01$ (+8.7$\sigma$)

Table 7. Table 3 : Performance for the boat class. For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ).

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.36\pm 0.03$	$0.28\pm 0.03$	$0.17\pm 0.04$	$0.06\pm 0.02$	$0.02\pm 0.01$
YOLT	RFSR $2\times$	$0.43\pm 0.05$ (+1.3$\sigma$)	$0.28\pm 0.03$ (+0.0$\sigma$)	$0.17\pm 0.03$ (+0.0$\sigma$)	$0.07\pm 0.02$ (+0.5$\sigma$)	$0.02\pm 0.01$ (+0.4$\sigma$)
YOLT	VDSR $2\times$	$0.49\pm 0.03$ (+2.0$\sigma$)	$0.32\pm 0.03$ (+1.0$\sigma$)	$0.20\pm 0.03$ (+0.6$\sigma$)	$0.06\pm 0.01$ (+0.1$\sigma$)	$0.03\pm 0.01$ (+1.6$\sigma$)
YOLT	RFSR $4\times$		$0.31\pm 0.04$ (+0.7$\sigma$)	$0.16\pm 0.03$ (-0.1$\sigma$)	$0.08\pm 0.02$ (+0.9$\sigma$)	$0.03\pm 0.01$ (+1.0$\sigma$)
YOLT	VDSR $4\times$		$0.44\pm 0.06$ (+2.5$\sigma$)	$0.20\pm 0.03$ (+0.7$\sigma$)	$0.09\pm 0.03$ (+0.9$\sigma$)	$0.03\pm 0.01$ (+1.4$\sigma$)
SSD	Native	0.12 $\pm$ 0.02	0.14 $\pm$ 0.03	0.03 $\pm$ 0.01	0.01 $\pm$ 0	0 $\pm$ 0
SSD	RFSR $2\times$	0.14 $\pm$ 0.03 (+0.6$\sigma$)	0.09 $\pm$ 0.02 (-1.3$\sigma$)	0.05 $\pm$ 0.01 (+0.9$\sigma$)	0.02 $\pm$ 0.01 (+1.1$\sigma$)	0.01 $\pm$ 0 (+1.8$\sigma$)
SSD	VDSR $2\times$	0.11 $\pm$ 0.02 (-0.2$\sigma$)	0.1 $\pm$ 0.02 (-0.9$\sigma$)	0.04 $\pm$ 0.01 (-2.9$\sigma$)	0.03 $\pm$ 0.01 (+2.8$\sigma$)	0.01 $\pm$ 0 (+2.1$\sigma$)

Table 8. Table 4 : Performance for the small aircraft class For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ).

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.59\pm 0.13$	$0.55\pm 0.11$	$0.55\pm 0.11$	$0.11\pm 0.05$	$0.0\pm 0.00$
YOLT	RFSR $2\times$	$0.70\pm 0.10$ (+0.7$\sigma$)	$0.64\pm 0.12$ (+0.5$\sigma$)	$0.46\pm 0.12$ (-0.6$\sigma$)	$0.18\pm 0.06$ (+0.8$\sigma$)	$0.01\pm 0.01$ (+1.1$\sigma$)
YOLT	VDSR $2\times$	$0.66\pm 0.14$ (+0.4$\sigma$)	$0.52\pm 0.10$ (-0.2$\sigma$)	$0.54\pm 0.13$ (-0.1$\sigma$)	$0.11\pm 0.04$ (+0.0$\sigma$)	$0.0\pm 0.00$ (+0.3$\sigma$)
YOLT	RFSR $4\times$		$0.72\pm 0.13$ (+1.0$\sigma$)	$0.53\pm 0.12$ (-0.2$\sigma$)	$0.13\pm 0.04$ (+0.2$\sigma$)	$0.01\pm 0.01$ (+1.0$\sigma$)
YOLT	VDSR $4\times$		$0.75\pm 0.09$ (+1.3$\sigma$)	$0.47\pm 0.12$ (-0.5$\sigma$)	$0.20\pm 0.07$ (+0.9$\sigma$)	$0.01\pm 0.01$ (+1.0$\sigma$)
SSD	Native	0.14 $\pm$ 0.04	0.14 $\pm$ 0.04	0.05 $\pm$ 0.03	0 $\pm$ 0	0 $\pm$ 0
SSD	RFSR $2\times$	0.05 $\pm$ 0.03 (-1.8$\sigma$)	0.24 $\pm$ 0.07 (+1.2$\sigma$)	0.04 $\pm$ 0.02 (-0.1$\sigma$)	0 $\pm$ 0 (0$\sigma$)	0 $\pm$ 0 (0$\sigma$)
SSD	VDSR $2\times$	0.37 $\pm$ 0.12 (+1.8$\sigma$)	0.15 $\pm$ 0.07 (+0.1$\sigma$)	0.08 $\pm$ 0.03 (-1.2$\sigma$)	0.01 $\pm$ 0.01 (+0.9$\sigma$)	0 $\pm$ 0 (0$\sigma$)

Table 9. Table 5 : Performance for the large aircraft class. For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ).

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.70\pm 0.04$	$0.69\pm 0.04$	$0.68\pm 0.04$	$0.63\pm 0.04$	$0.48\pm 0.05$
YOLT	RFSR $2\times$	$0.77\pm 0.04$ (+1.2$\sigma$)	$0.74\pm 0.03$ (+0.9$\sigma$)	$0.68\pm 0.04$ (-0.1$\sigma$)	$0.62\pm 0.04$ (-0.2$\sigma$)	$0.53\pm 0.05$ (+0.7$\sigma$)
YOLT	VDSR $2\times$	$0.80\pm 0.03$ (+1.8$\sigma$)	$0.78\pm 0.04$ (+1.6$\sigma$)	$0.61\pm 0.04$ (-1.3$\sigma$)	$0.65\pm 0.04$ (+0.2$\sigma$)	$0.56\pm 0.05$ (+1.3$\sigma$)
YOLT	RFSR $4\times$		$0.78\pm 0.03$ (+1.7$\sigma$)	$0.75\pm 0.04$ (+1.1$\sigma$)	$0.68\pm 0.05$ (+0.7$\sigma$)	$0.56\pm 0.05$ (+1.1$\sigma$)
YOLT	VDSR $4\times$		$0.77\pm 0.03$ (+1.5$\sigma$)	$0.70\pm 0.04$ (+0.3$\sigma$)	$0.71\pm 0.04$ (+1.3$\sigma$)	$0.43\pm 0.05$ (-0.7$\sigma$)
SSD	Native	0.7 $\pm$ 0.04	0.68 $\pm$ 0.04	0.69 $\pm$ 0.04	0.36 $\pm$ 0.05	0 $\pm$ 0
SSD	RFSR $2\times$	0.58 $\pm$ 0.03 (-2.4$\sigma$)	0.67 $\pm$ 0.04 (-0.2$\sigma$)	0.65 $\pm$ 0.04 (-0.7$\sigma$)	0.56 $\pm$ 0.05 (+2.7$\sigma$)	0.34 $\pm$ 0.05 (+6.9$\sigma$)
SSD	VDSR $2\times$	0.53 $\pm$ 0.05 (-2.7$\sigma$)	0.63 $\pm$ 0.04 (-1$\sigma$)	0.65 $\pm$ 0.04 (-0.5$\sigma$)	0.55 $\pm$ 0.04 (+2.8$\sigma$)	0.41 $\pm$ 0.05 (+8.6$\sigma$)

Table 10. Table 6 : Performance for the small vehicle class. For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ).

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.63\pm 0.02$	$0.58\pm 0.02$	$0.43\pm 0.02$	$0.19\pm 0.01$	$0.03\pm 0.00$
YOLT	RFSR $2\times$	$0.66\pm 0.01$ (+1.3$\sigma$)	$0.59\pm 0.02$ (+0.4$\sigma$)	$0.42\pm 0.02$ (-0.1$\sigma$)	$0.23\pm 0.02$ (+1.8$\sigma$)	$0.03\pm 0.00$ (+0.4$\sigma$)
YOLT	VDSR $2\times$	$0.66\pm 0.01$ (+1.7$\sigma$)	$0.60\pm 0.02$ (+0.8$\sigma$)	$0.47\pm 0.02$ (+1.6$\sigma$)	$0.20\pm 0.02$ (+0.4$\sigma$)	$0.02\pm 0.00$ (-1.8$\sigma$)
YOLT	RFSR $4\times$		$0.56\pm 0.03$ (+1.9$\sigma$)	$0.40\pm 0.03$ (-0.2$\sigma$)	$0.23\pm 0.01$ (+0.8$\sigma$)	$0.12\pm 0.01$ (+1.4$\sigma$)
YOLT	VDSR $4\times$		$0.59\pm 0.02$ (+3.0$\sigma$)	$0.39\pm 0.03$ (-0.6$\sigma$)	$0.25\pm 0.02$ (+1.7$\sigma$)	$0.10\pm 0.01$ (-0.3$\sigma$)
SSD	Native	0.48 $\pm$ 0.01	0.46 $\pm$ 0.02	0.27 $\pm$ 0.02	0.04 $\pm$ 0.01	0 $\pm$ 0
SSD	RFSR $2\times$	0.75 $\pm$ 0.01 (+14.3$\sigma$)	0.49 $\pm$ 0.02 (+1.2$\sigma$)	0.31 $\pm$ 0.02 (+1.7$\sigma$)	0.07 $\pm$ 0.01 (+2.4$\sigma$)	0 $\pm$ 0 (-2.5$\sigma$)
SSD	VDSR $2\times$	0.72 $\pm$ 0.02 (+11.1$\sigma$)	0.49 $\pm$ 0.02 (+1.2$\sigma$)	0.38 $\pm$ 0.02 (-3.2$\sigma$)	0.08 $\pm$ 0.01 (+3.4$\sigma$)	0 $\pm$ 0 (-2.5$\sigma$)

Table 11. Table 7 : Performance for the truck and bus class. For RFSR and VDSR at each resolution we note the error and statistical difference from the baseline model (e.g. +0.5 σ 𝜎 \sigma ).

Model	Data	30 cm	60 cm	120 cm	240 cm	480 cm
YOLT	Native	$0.38\pm 0.02$	$0.32\pm 0.01$	$0.22\pm 0.01$	$0.07\pm 0.01$	$0.01\pm 0.00$
YOLT	RFSR $2\times$	$0.41\pm 0.02$ (+1.7$\sigma$)	$0.33\pm 0.01$ (+0.4$\sigma$)	$0.22\pm 0.03$ (+0.2$\sigma$)	$0.08\pm 0.01$ (+0.9$\sigma$)	$0.01\pm 0.00$ (+1.5$\sigma$)
YOLT	VDSR $2\times$	$0.41\pm 0.02$ (+1.4$\sigma$)	$0.35\pm 0.01$ (+1.3$\sigma$)	$0.22\pm 0.01$ (+0.5$\sigma$)	$0.08\pm 0.01$ (+0.5$\sigma$)	$0.01\pm 0.00$ (+1.8$\sigma$)
YOLT	RFSR $4\times$		$0.36\pm 0.02$ (+1.8$\sigma$)	$0.20\pm 0.01$ (-0.6$\sigma$)	$0.08\pm 0.01$ (+0.4$\sigma$)	$0.01\pm 0.00$ (+2.0$\sigma$)
YOLT	VDSR $4\times$		$0.38\pm 0.02$ (+2.6$\sigma$)	$0.21\pm 0.01$ (-0.2$\sigma$)	$0.07\pm 0.01$ (+0.2$\sigma$)	$0.01\pm 0.00$ (+2.6$\sigma$)
SSD	Native	0.28 $\pm$ 0.01	0.16 $\pm$ 0.01	0.06 $\pm$ 0.01	0 $\pm$ 0	0 $\pm$ 0
SSD	RFSR $2\times$	0.26 $\pm$ 0.02 (-0.8$\sigma$)	0.17 $\pm$ 0.01 (+0.9$\sigma$)	0.14 $\pm$ 0.01 (+6.1$\sigma$)	0 $\pm$ 0 (+3.2$\sigma$)	0 $\pm$ 0 (+0$\sigma$)
SSD	VDSR $2\times$	0.33 $\pm$ 0.02 (+2$\sigma$)	0.2 $\pm$ 0.01 (+2.7$\sigma$)	0.15 $\pm$ 0.01 (-0.5$\sigma$)	0.01 $\pm$ 0 (+4.4$\sigma$)	0 $\pm$ 0 (+0$\sigma$)

Equations

$$kernel=0.5\times GS{D}_{out}/GS{D}_{native}$$

$${\sigma }_{tot}^2={\sigma }_b^2+{\sigma }_{sr}^2$$

$${\sigma }_{diff}=\frac{Y_{sr}-Y_b}{{\sigma }_{tot}}$$

Full text

1 Introduction

The interplay between super-resolution techniques and object detection frameworks remains largely unexplored, particularly in the context of satellite or overhead imagery. Intuitively, one might assume that super-resolution methods should increase object detection performance, as an increase in resolution should add more distinguishable features that an object detection algorithm can use for discrimination. Detecting small objects such as vehicles in satellite imagery remains an exceedingly difficult task for multiple reasons [37] and an artificial increase in resolution may help to alleviate some of these issues. Some of the issues present include:

Objects such as cars in satellite imagery have a small spatial extent (as low as 10 pixels) and are often densely clustered. 2. 2.

All objects exhibit complete rotation invariance and can have any orientation. 3. 3.

Training example frequency is low versus other disciplines. Few datasets exist that have appropriate labels for objects within satellite imagery. The most notable are: SpaceNet [38], A Large-scale Dataset for Object DeTection in Aerial Images (DOTA) [40], Cars Overhead With Context (COWC) [27], and xView [18]. 4. 4.

Most satellite imagine sensors cover a broad area and contain hundreds of megapixels, thereby producing the equivalent of an ultra-high resolution image. For example, the native imagery used in this study was on average $\approx 57$ times larger than benchmark super-resolution datasets Set5, Set14, BSD100, and Urban100. When working with modern neural network architectures these images must be tiled into smaller chunks for both training and inference.

Although several studies have been conducted using SR as a pre-processing step [1, 11, 12, 33, 42, 3, 10, 5], none have quantified its affect on object detection performance in satellite imagery across multiple resolutions. This study aims to accomplish that task by training multiple custom object detection models to identify vehicles, boats, and planes in both native and super-resolved data. We then test the models performance on the native (ground-truth) imagery and super-resolved imagery of the same Ground Sample Distance (GSD: the distance between pixels measured on the ground). Additionally, this is the first study to demonstrate the output of super-resolved 15 cm GSD satellite imagery. Although no native 15 cm satellite imagery exists for comparison, this data can be compared against coarser resolutions to test the benefits provided by super-resolution.

The cost-benefit analysis of such a study is enormous. Satellite manufacturers spend the majority of their budget on the design and launch of satellites. For example, the DigitalGlobe WorldView-4 satellite cost an estimated \835$ million dollars when one includes spacecraft, insurance, and launch [8]. Ideally, one could couple an effective SR enhancement algorithm with a smaller, cheaper satellite that captures images in coarser resolution. The process of capturing and subsequently enhancing coarser data could drastically reduce launch cost, expand satellite field of view, reduce the number of satellites in orbit, and improve downlink speeds between satellites and ground control stations.

2 Related Work

2.1 Super-Resolution Techniques and Application to Overhead Imagery

Single-image Super-Resolution (SR) is the process for deriving high-resolution (HR) images from a single low-resolution (LR) image. Although super-resolution remains an ill-posed and difficult problem, recent advances in neural networks and machine learning have enabled more robust SR algorithms that exhibit effective performance. These techniques use high-resolution image pairs to learn the most likely HR features to map to the LR image features and create an output SR product.

Over the past five years, convolutional neural network approaches have been used to produce state of the art super-resolution results. Dong et al. [7] was the first to establish a deep learning approach with SRCNN. This has been followed up by several successive approaches, major alterations, and improvements. Very Deep Super Resolution (VDSR) [15] exhibited state of the art performance and was one of the first to modify the SRCNN approach by creating a deeper network with 20 layers to learn a residual image and transform LR images into HR images. Developed concurrently, the Deeply-Recursive CNN (DRCN) [16] introduced a recursive neural network approach to super-resolve imagery. The Deeply Recursive Residual Network (DRRN) [34] builds upon the VDSR and DRCN advancements using a combination of the residual layers approach and recursive learning in a compact network.

More complex methods followed, such as the Laplacian Pyramid Super-Resolution Network (LapSRN) [17]. Adversarial training has also been employed and the SR Generative Adversarial Network (SRGAN) [19] produces photo-realistic $4\times$ enhanced images. The use of wider and deeper networks has also been proposed. The most notable being Lim et al. [21], which proposed Enhanced Deep Residual Networks (EDSR). Most recently, the Deep Back Projection Network (DBPN) [9] showed state of the art performance for an $8\times$ enhancement by connecting a series of iterative up- and down-sampling stages. Newer block based methods such as the Information Distillation Network (IDN) [14] was developed as a compact network that could gradually extract common features for fast the reconstruction of HR images. In another example, the Residual Dense Network (RDN) [43] uses residual dense blocks to produce strong performance.

Although new and powerful single image SR techniques continue to be developed, these techniques have been infrequently applied to overhead imagery. One of the most notable applications of super resolution to satellite and overhead imagery remains the recent paper by Bosch et al. [2]. The authors analyze several sources of satellite imagery for this research and quantify their success in terms of PSNR for an $8\times$ enhancement using a GAN. In another example, [22] use deep neural networks for simultaneous $4\times$ super-resolution and colorization of satellite imagery. Several papers [41, 25, 35, 20, 28] modify or leverage SRCNN [7] and/or VDSR [15] to successfully super-resolve Jilin-1, SPOT, Pleiades, Sentinel-2, and Landsat imagery.

Ultimately, a few specific papers are direct precursors for this work: In the first, [3] use fine resolution aerial imagery and coarser satellite imagery with a coupled dictionary learning approach to super enhance vehicles and detect them with a simple linear Support Vector Machine model. Their results showed that object detection performance improves when using SR as a pre-processing step versus the native coarser imagery. Xu et al. [42] use sparse dictionary learning to generate synthetic $8\times$ and $16\times$ super-resolved imagery from Landsat and MODIS image pairs. Their results show an increase performance for land-cover change mapping when using the super-resolved imagery. Although these approaches are similar to ours, they fail to use newer neural network based approaches, and are narrower in scope. Finally, [10] super-resolve imagery using DBPN [9] and detect various objects in traditional photography using SSD [23]. They quantify their success in terms of mAP and also add a novel element to this work of designing a loss function to optimize SR for object detection performance. Their results show that end-to-end training of these algorithms gave a performance boost for object detection tasks, and is a promising avenue to explore for future research.

Overall, we hypothesized that SR techniques could improve object detection performance, particularly when using satellite imagery, however no such study has been conducted. To address this question, our study investigates the relationship between object detection performance and resolution, spanning five unique GSD resolutions, with six SR outputs per resolution. Ultimately, we investigate 35 separate resolution profiles for object detection performance.

2.2 Object Detection Techniques

A number of recent papers have applied advanced machine learning techniques to aerial or satellite imagery, yet have focused on a slightly different problem than the one we attempt to address. For example, [24] demonstrated the ability to localize objects in overhead imagery; yet application to larger areas would be problematic, with an inference speed of 10 - 40 seconds per $1280\times 1280$ pixel image chip. Efforts to localize surface to-air-missile sites [26] with satellite imagery and sliding window classifiers work if one only is interested in a single object size of hundreds of meters. Running a sliding window classifier across a large satellite image to search for small objects of interest quickly becomes computationally intractable, however, since multiple window sizes will be required for each object size. For perspective, one must evaluate over one million sliding window cutouts if the target is a 10 meter boat in a DigitalGlobe image. Application of rapid object detection algorithms to the remote sensing sphere is still relatively nascent, as evidenced by the lack of reference to SSD [23], Faster-RCNN [30], R-FCN [6], or YOLO [29] in a recent survey of object detection in remote sensing [4]. While tiling a large image is still necessary, the larger field of view of these frameworks (a few hundred pixels) compared to simple classifiers (as low as 10 pixels) results in a reduction in the number of tiles required by a factor of over 1000. This reduced number of tiles yields a corresponding marked increase in inference speed. In addition, object detection frameworks often have much improved background differentiation (compared to sliding window classifiers) since the network encodes contextual information for each object.

As we seek to study the effect of super-resolution on object detection performance in real-world satellite imagery, and for all of the reasons listed above - rapid object detection frameworks are the logical choice for this study. The premier rapid object detection algorithms (SSD, Faster-RCNN, R-FCN, and a modified version of YOLO called YOLT [36]) were recently incorporated into the unified framework of SIMRDWN [37] that is optimized for ingesting satellite imagery, typically several hundred megapixels in size. The SIMRDWN paper reported the highest performance stemmed from the YOLT algorithm, followed by SSD, with Faster R-CNN and RFCN significantly behind.

3 Dataset

The xView Dataset [18] was chosen for the application of super-resolution techniques and the quantification of object detection performance. Imagery consists of 1,415 km2 of DigitalGlobe WorldView-3 pan-sharpened RGB imagery at 30 cm native GSD resolution spread across 56 distinct global locations and 6 continents (sans Antarctica). The labeled dataset for object detection contains 1 million object instances across 60 classes annotated with bounding boxes, including various types of buildings, vehicles, planes, trains, and boats. For our purposes, we ultimately discarded classes such as “Building,” “Hangar,” and “Vehicle Lot” because we found that such objects are better represented by polygonal labels rather than bounding boxes for foundational mapping [38] purposes.

We chose an aggregation schema due to inconsistent labeling within the dataset. Unfortunately, many objects are mislabeled or simply missed by labelers (see Figure 1). This leads to an increase in false positive detection rates and objects being inaccurately tagged as mis-classifications after inference. In addition, many xView classes have a very low number of training examples (e.g. Truck w/Liquid has only 149 examples) that are poorly differentiated from similar classes (e.g. Truck w/Box has 3653 examples and looks very similar to Truck w/Liquid). The question of how many training examples are necessary to disentangle similar classes is beyond the scope of this paper.

Our classes ultimately consist of the following (original xView classes listed in parentheses): Small Aircraft (Fixed-wing Aircraft, Small Aircraft), Large Aircraft (Cargo Plane), Small Vehicle (Passenger Vehicle, Small Car, Pickup Truck, Utility Truck), Bus/Truck (Bus, Truck, Cargo Truck, Truck w/Box, Truck w/Flatbed, Truck w/Liquid, Dump Truck, Haul Truck, Cement Mixer, Truck Tractor), and Boat (Motorboat, Sailboat, Yacht, Maritime Vessel, Tugboat, Barge, Fishing Vessel, Ferry). See Table 1 for dataset details and Figure 2 for object size histograms.

3.1 Simulation of Optics and Sensors

All data were preprocessed consistently to simulate coarser resolution imagery and test the affects of our SR techniques on a range of resolutions. We intend our results to showcase what can be reasonably accomplished given coarser satellite imagery, rather than simply what is possible given the ideal settings (no blurring, bicubic decimation) under which most SR algorithms are introduced. We attempt to simulate coarser resolution satellite imagery as accurately as possible by simulating the optical point-spread function (PSF) and using a more robust decimation algorithm. This is important because the optics of the telescope greatly impact the appearance of very small objects. The common practice of simply resizing an image by reducing its dimensions by a factor of two will simulate a different sensor containing $1/4$ the number of pixels; yet this approach ignores the different optics present in a properly designed telescope that would be coupled to such a sensor. A properly designed sensor should have pixel size determined by the Nyquist sampling rate: half the size of the mirror resolution determined by the diffraction limit. Given the cost and complexity of launching satellite imaging constellations to orbit, we assume that all imaging satellites will have properly designed sensors. We can use the assumption of Nyquist sampling to determine the PSF of the telescope optics, which can be approximated by a Gaussian of appropriate kernel size:

[TABLE]

For our study, data were degraded from the native 30 cm GSD using a variable Gaussian blur kernel to simulate the point-spread function of the satellite depending upon our desired output resolution (Equation 1). We then used inter-area decimation to reduce the dimensions of the blurred imagery to the appropriate output size (e.g. 60 cm imagery will have $1/4$ the number of pixels as 30 cm imagery over the same field of view). We repeat the above procedure to simulate resolutions of 60, 120, 240, and 480 cm. The ground truth data and the outputs from the super-resolution algorithms were randomly split into training (60$\%$) and validation (40$\%$) categories for object detection. The same images are contained in both the training and test sets regardless of resolution to maintain consistency when comparing validation scores.

4 Super-Resolution Techniques

For this study, super-resolution is conducted with two techniques for enhancement levels of $2\times$, $4\times$, $8\times$ over five distinct resolutions ranging from 30 cm to 4.8 meters. We also create 15 cm GSD output imagery using the models trained to super-resolve imagery from 60 cm to 30 cm and 120 cm to 30 cm.

Our first method is a convolutional neural network derived technique called Very Deep Super-Resolution (VDSR) [15]. VDSR has been featured as a baseline for the majority of recent super-resolution research and was one of the first to modify the initially proposed convolutional neural network method SRCNN [7]. This architecture was chosen due its ease of implementation, ability to train for multiple levels of enhancement, use as a standard baseline when introducing new techniques, and favorable performance in the past. We use the standard network parameters as set in the original paper [15] and train for 60 epochs. We chose a patch size of 41 $\times$ 41 pixels and augment by rotations (4) and flipping (2) for eight unique combinations per patch. This process is repeated for each enhancement level (2, 4, and 8$\times$), and each is fed into the same network for concurrent training. Average training time for a 2, 4, and 8$\times$ enhancement on  200 million pixel example is 55.9 hours on a single Titan Xp GPU. Inference speed on a 544 $\times$ 544 pixel image is very fast $\approx 0.2$ seconds on the same hardware, allowing for this method to easily scale to accommodate large satellite images.

The second method is an approach that we have called Random-Forest Super-Resolution (RFSR) and was designed for this work; it requires minimal training time and exhibits high inference speeds. RFSR is an adaptation of other random forest super-resolution techniques such as SRF [32] or SRRF [13] and can process both georeferenced satellite imagery or traditional photography. We chose to include this simpler, less computationally intensive algorithm that does not require GPUs to test its effectiveness against a near state of the art SR solution. The hypothesis is that even a simple technique may improve object detection performance.

Our method uses a random forest regressor with a few standard parameters. The number of estimators is set to 100, the maximum depth to 12, and the minimum samples to split an internal node equal to 200. Finally, we use bootstrapping and out-of-bag samples to estimate the error and R2 scores on randomly selected unseen data during training. These parameters were finely tuned using empirical testing to maximize PSNR scores (see Section 6 for details on metrics) while maintaining minimal training time (4 hours or less per level of enhancement on a 64GB RAM CPU). It should be noted that PSNR scores could be mildly improved using a deeper tree with more estimators, at the cost of training time.

Like several other SR techniques, RFSR is trained only using the luminance component from a YCbCr converted image. HR images are degraded to create LR and HR image pairs. The degraded LR image is then shifted by one and then two pixels in each direction versus the HR image and then compressed into a 3-dimensional array. The original up-sampled LR image is then subtracted from the 3-D LR array, and from the HR image for a residual training schema. This normalizes the LR stack and HR image pair and also removes homogeneous areas, emphasizing important edge effects. After training and inference the interpolated LR image is then added back to the models’ output image to create the super-resolved output. RFSR can only produce one level of enhancement (2, 4, or $8\times$) at a time. Average training time for all three enhancements on $\sim 200$ million pixel examples is 10.8 hours on a 64GB RAM CPU. Average inference speed on a $544\times$544 pixel image is 0.7 seconds for this same hardware (Table 2).

5 Object Detection Techniques

As discussed in Section 1, advanced object detection frameworks have only recently been applied to large satellite imagery via the SIMRDWN framework. In the SIMRDWN paper, the authors reported the highest performance stemmed from the YOLT algorithm, followed by SSD, with Faster R-CNN and RFCN significantly behind. Therefore, we opt to utilize the YOLT and SSD models within SIMRDWN for this study. For the YOLT model we adopt the dense 22-layer network of [36] with a momentum of 0.9, and a decay rate of 0.0005. We use a $544\times 544$ pixel training input size (corresponding to $164\times 164$ meters). Training occurs for 150 epochs. For the SSD model we follow the TensorFlow Object Detection API implementation with the Inception V2 architecture. We adopt a base learning rate of 0.004 and a decay rate of 0.95. We train for 30,000 iterations with a batch size of 16, and use the same $544\times 544$ pixel input size as YOLT. For both YOLT and SSD we train models on the “native” imagery (original 30 cm data, the convolved and resized imagery described in Section 3.1), as well as on the outputs of RFSR and VDSR applied to the object detection training set. This approach yields a multitude of models across the myriad architectures, super-resolution techniques, and resolutions (see Figure LABEL:figs/sr_fig2_1.png, thus enabling a detailed study of performance.

6 Metrics

Overall, super-resolution remains an active field of research with rather limited direct focus on end application. Typical performance metrics include Peak Signal-to-Noise Ratio (PSNR) or the Structural SIMilarity (SSIM) Index (which we report in Section 7.1), however these measures do not quantify the enhancement to object detection performance [39]. Although these images may be more visually appealing as a result of super-resolution, such techniques may have little impact on object detection performance.

For object detection metrics, we compare the ground truth bounding boxes to the predicted bounding boxes for each test image. For comparison of predictions to ground truth we define a true positive as having an intersection over union (IOU) of greater than a given threshold. An IoU of 0.5 is often used as the threshold for a correct detection, though we adopt a lower threshold of 0.25 since most of our objects are very small (e.g.: cars are only  10 pixels in extent). This mimics Equation 5 of ImageNet [31], which sets an IoU threshold of 0.25 for objects 10 pixels in extent. Precision-recall curves are computed by evaluating test images over a range of probability thresholds. At each of 30 evenly spaced thresholds between 0.05 and 0.95, we discard all detections below the given threshold. Non-max suppression for each object class is subsequently applied to the remaining bounding boxes; the precision and recall at that threshold is tabulated from the summed true positive, false positive, and false negatives of all test images. Finally, we compute the average precision (AP) for each object class and each model, along with the mean average precision (mAP) for each model. One-sigma error bars are computed via bootstrap resampling, using 500 samples for each scenario.

7 Experimental Results

7.1 Super-Resolution Performance

As expected, super-resolution performance was strongest for the VDSR method, although RFSR produces comparable results in some circumstances (Table 1). As in other studies, the metrics degrade as the amount of enhancement increases. Both techniques performed the strongest on the 60 cm imagery, likely because initial bicubic interpolation scores are high and the fact that the image resolution is situated between a coarse and fine scale where the image features are easier to detect and enhance.

A few specific examples of super resolution performance are visible in Figure 3, where we test the effects of our algorithm on a large object like a plane. Visually, VDSR and RFSR both perform strongly at 30 cm for both a $2\times$ (60 cm input $\rightarrow$ 30 cm SR output) and $4\times$ (120 cm input $\rightarrow$ 30 cm SR output) enhancement, where both the fine details of the plane, and small neighboring objects can be accurately recovered. Recovering the plane at coarser resolutions is extremely difficult, particularly at 4.8 m with an $8\times$ enhancement. In this case the input for the SR algorithm is 38.4 m GSD; at this resolution the satellite is simply insufficiently sensitive to resolve finer objects. Overall, we observe that when the imagery possesses fewer fine features to identify in coarser resolutions, algorithms are unable to hallucinate and recover all object types. A different algorithm such as a GAN may be able to hallucinate visually finer features, however previous studies [2] have shown that these algorithms are unable to exactly recover specific features of various object types.

Finally, in Figure 4 we demonstrate the visual enhancement provided by simulated 15 cm super-resolved output from both VDSR and RFSR. Both methods improve the visual quality by reducing pixelization and enhancing the clarity of features and characters. RFSR appears to produce slightly brighter edge effects than VDSR.

7.2 Object Detection Performance

For each model we compute mean average precision (mAP) performance on a 338-image test set spanning 6 continents ( 632 sq. km) at each resolution. Example precision recall curves are shown in Figure 6. The YOLT model is clearly superior to SSD, particularly for small objects.

Repeating the computation shown in Figure 6 for all models allows us to determine the degradation of performance as a function of resolution, as shown in Figure 7. In this plot we display $1\sigma$ bootstrap error bars for each model group. Results for SSD models are significantly worse than YOLT models, with a mAP of 0.30 at native 30 cm resolution. The YOLT model (mAP = 0.53) at this resolution is 77% better than SSD, which aligns fairly well with the findings of [37]. Ultimately, object detection performance decreases by $22-27\%$ when resolution degrades from 30 cm to 120 cm, and another $73-100\%$ from 120 cm to 480 cm when looking across broad object classes.

We also plot the results of the effects of $2\times$ super-resolution models when using both YOLT and SSD (Figures S11 and 9). When using YOLT, performance improvements are statistically significant only in the finest resolutions (Table 7) with comparable results for both VDSR and RFSR. In Figure 11 we show the change in mAP versus the original 30 cm and 60 cm imagery. The largest performance boosts can be seen when enhancing imagery from 30 cm to 15 cm ($+13\%$ vs 30 cm) and 60 cm to 15 cm ($14-20\%$ improvement vs 60 cm). Interestingly, enhancing imagery from 60 cm to 30 cm was much less effective than enhancing imagery from 60 to 15 cm. These findings showcase the value of super-resolution as a pre-processing step in these GSDs. Combined with a state of the art object detection framework, super-resolution has the ability to improve detection rates beyond what is possible with the best commercially available satellite imagery.

Furthermore, although performance is much worse with SSD, super-resolution techniques are much more effective. With SSD, for both RFSR and VDSR performance boosts are evident for all resolutions, except for 60 cm to 30 cm. VDSR is generally shown to be slightly superior to RFSR when detecting objects with SSD. For SSD the improvement at 480 cm is statistically quite significant, though this is primarily due to the mAP of 0.0 for native imagery. Performance increases significantly once objects are greater than $\approx 20$ pixels in extent. This trend extends across object classes, as shown in the performance curves for individual object classes (See Supplemental Material).

8 Conclusions

In this paper we undertook a rigorous study of the utility provided by super-resolution techniques towards the detection of objects in satellite imagery. We paired two super-resolution techniques (VDSR and RFSR) with advanced object detection methods and searched for objects in a satellite imagery dataset with over 250,000 labeled objects in a diverse set of environments. In order to establish super-resolution effects at multiple sensor resolutions, we degrade this imagery from 30 cm to 60, 120, 240, and 480 cm resolutions. Our baseline tests with both the YOLT and SSD models of the SIMRDWN object detection framework indicate that object detection performance decreases by $22-27\%$ when resolution degrades from 30 cm to 120 cm.

The application of SR techniques as a pre-processing step provides an improvement in object detection performance at most resolutions (Table 7). For both object detection frameworks, the greatest benefit is achieved at the highest resolutions, as super-resolving native 30 cm imagery to 15 cm yields a $13-36\%$ improvement in mAP. Furthermore, when using YOLT, we find that enhancing imagery from 60 cm to 15 cm provides a significant boost in performance over both the native 30 cm imagery ($+13\%$) and native 60 cm imagery ($+20\%$). The performance boost applies to all classes, but is most significant for boats, large aircraft, and buses/trucks. Again with YOLT, in coarser resolutions (120 cm to 480 cm) SR provides little to no boost in performance (-0.02 to +0.04 change in mAP). When using SSD, super-resolving imagery from 30 to 15 cm provides a substantial boost for the identification of small vehicles ($+56\%$), but provides mixed results for other classes (See supplemental material). In coarser resolutions, with SSD, SR techniques provide a greater boost in performance however the performance for most classes is still worse compared to YOLT with native imagery. Overall, given the relative ease of applying SR techniques, the general improvement observed in this study is noteworthy and suggests SR could be a valuable pre-processing step for future object detection applications with satellite imagery.

1 Super-Resolution Outputs

2 Super-Resolution Scores- Bicubic Decimation with No Blurring

3 Object Detection Performance

The figures below illustrate bounding boxes output by YOLT models at various resolutions. Cars are green, buses/trucks are blue, small aircraft are red, and large aircraft are yellow.

4 Object Detection Performance Curves and Tables

To compute the statistical difference ($\sigma_{diff}$) between the super-resolved and baseline models, we follow the procedure used in Table 4 of the main text. The joint error estimate between the baseline ($Y_{b}$) and super-resolved ($Y_{sr}$) data point can be estimated as:

[TABLE]

The statistical difference between two models is then simply:

[TABLE]

The tables and plots below show the performance of various models for each object class.