Fine-Tuning Without Forgetting: Adaptation of YOLOv8 Preserves COCO Performance

TL;DR

This paper systematically studies how fine-tuning depth affects YOLOv8's performance on specialized tasks, showing deep fine-tuning improves target accuracy with negligible forgetting of original capabilities.

Contribution

It provides empirical evidence that unfreezing deeper backbone layers in YOLOv8 enhances fine-grained task performance without significant loss on the original dataset.

Findings

Deeper fine-tuning yields +10% mAP50 on fruit detection.

Negligible (<0.1%) performance loss on COCO after deep fine-tuning.

Unfreezing down to layer 10 is highly effective for specialization.

Abstract

The success of large pre-trained object detectors hinges on their adaptability to diverse downstream tasks. While fine-tuning is the standard adaptation method, specializing these models for challenging fine-grained domains necessitates careful consideration of feature granularity. The critical question remains: how deeply should the pre-trained backbone be fine-tuned to optimize for the specialized task without incurring catastrophic forgetting of the original general capabilities? Addressing this, we present a systematic empirical study evaluating the impact of fine-tuning depth. We adapt a standard YOLOv8n model to a custom, fine-grained fruit detection dataset by progressively unfreezing backbone layers (freeze points at layers 22, 15, and 10) and training. Performance was rigorously evaluated on both the target fruit dataset and, using a dual-head evaluation architecture, on the original COCO validation set. Our results demonstrate unequivocally that deeper fine-tuning (unfreezing down to layer 10) yields substantial performance gains (e.g., +10\% absolute mAP50) on the fine-grained fruit task compared to only training the head. Strikingly, this significant adaptation and specialization resulted in negligible performance degradation (<0.1\% absolute mAP difference) on the COCO benchmark across all tested freeze levels. We conclude that adapting mid-to-late backbone features is highly effective for fine-grained specialization. Critically, our results demonstrate this adaptation can be achieved without the commonly expected penalty of catastrophic forgetting, presenting a compelling case for exploring deeper fine-tuning strategies, particularly when targeting complex domains or when maximizing specialized performance is paramount.