Sim-to-Real Fruit Detection Using Synthetic Data: Quantitative Evaluation and Embedded Deployment with Isaac Sim

TL;DR

This paper evaluates the use of synthetic data generated in NVIDIA Isaac Sim for training fruit detection models, demonstrating that hybrid training improves performance and robustness, and that models can be efficiently deployed on embedded systems.

Contribution

It provides a quantitative analysis of synthetic data's effectiveness for sim-to-real transfer in object detection and demonstrates embedded deployment on Jetson Orin NX.

Findings

Hybrid training improves detection accuracy over synthetic-only models.

Synthetic data combined with real data approaches real-only performance.

Models can be deployed in real-time on embedded hardware using TensorRT.

Abstract

This study investigates the effectiveness of synthetic data for sim-to-real transfer in object detection under constrained data conditions and embedded deployment requirements. Synthetic datasets were generated in NVIDIA Isaac Sim and combined with limited real-world fruit images to train YOLO-based detection models under real-only, synthetic-only, and hybrid regimes. Performance was evaluated on two test datasets: an in-domain dataset with conditions matching the training data and a domain shift dataset containing real fruit and different background conditions. Results show that models trained exclusively on real data achieve the highest accuracy, while synthetic-only models exhibit reduced performance due to a domain gap. Hybrid training strategies significantly improve performance compared to synthetic-only approaches and achieve results close to real-only training while reducing the need for manual annotation. Under domain shift conditions, all models show performance degradation, with hybrid models providing improved robustness. The trained models were successfully deployed on a Jetson Orin NX using TensorRT optimization, achieving real-time inference performance. The findings highlight that synthetic data is most effective when used in combination with real data and that deployment constraints must be considered alongside detection accuracy.