Mini Autonomous Car Driving based on 3D Convolutional Neural Networks

TL;DR

This paper introduces a 3D CNN-based methodology for mini autonomous car driving using RGB-D data, evaluated in simulation, showing promising results over RNNs in control and generalization.

Contribution

The work presents a novel application of 3D CNNs for autonomous driving in mini cars, enabling efficient evaluation and comparison in simulated environments.

Findings

3D CNNs outperform RNNs in certain driving tasks

Architectural modifications impact model generalization

Track complexity affects control performance

Abstract

Autonomous driving applications have become increasingly relevant in the automotive industry due to their potential to enhance vehicle safety, efficiency, and user experience, thereby meeting the growing demand for sophisticated driving assistance features. However, the development of reliable and trustworthy autonomous systems poses challenges such as high complexity, prolonged training periods, and intrinsic levels of uncertainty. Mini Autonomous Cars (MACs) are used as a practical testbed, enabling validation of autonomous control methodologies on small-scale setups. This simplified and cost-effective environment facilitates rapid evaluation and comparison of machine learning models, which is particularly useful for algorithms requiring online training. To address these challenges, this work presents a methodology based on RGB-D information and three-dimensional convolutional neural networks (3D CNNs) for MAC autonomous driving in simulated environments. We evaluate the proposed approach against recurrent neural networks (RNNs), with architectures trained and tested on two simulated tracks with distinct environmental features. Performance was assessed using task completion success, lap-time metrics, and driving consistency. Results highlight how architectural modifications and track complexity influence the models' generalization capability and vehicle control performance. The proposed 3D CNN demonstrated promising results when compared with RNNs.

Tables

Table 1. Table I : Average times lap with standard deviation for the models evaluated on Mini Monaco and Generated Track.

Model	Mini Monaco (sec)	Generated Track (sec)
RNN Default	36.24 (±0.56)	14.75 (±0.22)
RNN Modified	36.56 (±0.48)	14.45 (±0.18)
3D CNN Modified	35.10 (±0.63)	14.58 (±0.25)

Table 2. Table II : Modified 3DCNN on Mini Monaco.

Model	Mini Monaco
3D CNN Modified (1 layers less)	34.10 (±0.63)
3D CNN Modified	35.56 (±0.68)
3D CNN Modified (1 layers more)	Overfit

Equations

$$z_t$$

$$r_t$$

$${\tilde{h}}_t$$

$$h_t$$

$$y=F(x,\{W_i\})+x$$

$$\hat{x}=\frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}},y=\gamma \hat{x}+\beta$$

Full text

Video link: https://youtu.be/6S2mfHWn3E8

I Introduction

Autonomous vehicle technology has become a prominent research area within the automotive industry, driven by increasing market demand for integrating autonomous driving features into commercial vehicles [1]. This application field inherently requires solutions characterized by high levels of trustworthiness and reliability. Typically, available AI-based solutions for autonomous driving are complex, require extensive training periods and naturally involve certain levels of randomness or imprecision. Simulation environments have emerged as central tools in the evaluation of machine learning (ML)-based models. By employing a scaled-down version of the proposed problem, methodologies can be effectively presented and validated, facilitating the identification and assessment of relevant phenomena associated with the study. [2]. Therefore, Mini Autonomous Cars (MAC)[3, 4] may provide an experimental environment in which solutions validated in small-scale setups may be easily scaled up to meet more realistic conditions. MAC provides a flexible and cost-effective setup for easily adapting experimental scenarios. By simplifying the creation and management of diverse conditions (e.g. lighting settings and obstacle configurations), it better supports extended evaluation of proposed autonomous driving methodologies and reduces the risks of damage.

At the same time, the development of autonomous driving systems has increased consistently in recent decades thanks to advances in ML and Computer Vision [5, 6]. Artificial Neural Networks (ANNs) have proven helpful for directly mapping images captured by front-facing cameras to vehicle control commands, enabling behavior cloning in controlled environments [3, 7]. However, these systems inherent challenges, such as high complexity, prolonged training periods, and intrinsic levels of uncertainty, which demands improved sensing for increase performance. Also, as training and validating models with minor changes in real-world settings may be complex, simulation has become a practical approach to developing and evaluating algorithms [8, 1]. Therefore, complex sensing information, such as from RGB-D cameras, can increase the model’s robustness, demanding more complex ANNs

Built upon this context, the present study provides a methodology for MAC autonomous driving based on 3D CNNs, performing a comparative assessment with RNNs within two simulated scenarios with distinct characteristics. Figure 1 presents the simplification of our proposed methodology.

Specific datasets were generated for each track, and both standard and modified versions of each architecture were trained and tested. The performance was evaluated using objective metrics focused on time and consistency within the simulator. The results provide insights into how architectural changes and environmental complexity impact the networks’ ability to generalize and control autonomous vehicles, thereby facilitating the selection of effective models suited for future implementation on embedded hardware platforms. Overall, this paper presents the following contributions:

• •

We show that our proposed method 3D CNN provides better performance in terms of average time to complete a proposed task when comparing with other RNN models.

• •

We provide an Ablation Study showing the impact of adding and removing layers in our proposed model, concluding that the version with the smallest amount of layers provided the best performance in terms of average time to complete a task.

The remainder of this paper is organized as follows: Section II reviews related works addressing the proposed problem. Section III describes the proposed methodology. Experimental results and discussions are reported in Section IV. Finally, Section V summarizes the concluding remarks and future directions.

II Related Works

Recent advances in autonomous driving research have increasingly focused on leveraging deep learning techniques, such as behavior cloning and reinforcement learning, to train control models in both simulated and real-world environments. Within this domain, a range of studies has investigated different model architectures, control strategies, and evaluation setups for improving the learning and generalization capabilities of autonomous systems.

Elmquist et al. (2022) [5] describes a software framework, along with a hardware platform, used to design and analyze robotics autonomy algorithms in both simulation and real-world environments. Their physical platform is a 1/6-scale vehicle with reconfigurable components, allowing for testing of algorithms and sensor setups in both domains. Yildirim et al. (2024) [9] validate perception systems for autonomous vehicles using behavior cloning for lateral control in real-world conditions, showing accurate steering angle predictions in real-time.

Codevilla et al. (2019) [10] analyze the limitations of behavior cloning, especially in generalizing to dynamic objects and lacking a causal model, highlighting areas that need further research. Karnala and Campbell (2020) [11] investigate how model architecture affects vulnerability to adversarial attacks in machine learning-based autonomous driving systems.

Zhang and Du (2019) [12] propose a deep reinforcement learning method to train a scaled autonomous car in simulation, with transfer to the real world. Samak et al. (2023) [6] introduce AutoDRIVE, a modular ecosystem for research and education in autonomous driving that supports simulation and deployment of solutions including behavior cloning via deep imitation learning. Jung (2018) [7] uses behavior cloning to train a supervised CNN classifier on the DonkeyCar platform, optimizing autonomous driving in a realistic environment

Although previous studies have explored autonomous vehicle control using behavior cloning and deep learning models, such as RNNs and CNNs, there is a lack of detailed comparative evaluations between architectures, particularly between RNNs and 3D CNNs, in small-scale environments like Mini Autonomous Cars (MACs). Additionally, while the general performance and accuracy of models have been investigated, few studies analyze the impact of architectural adjustments (such as adding or removing layers) on model performance in visually complex environments.

The present work addresses this gap by directly comparing RNN and 3D CNN models in the context of small autonomous vehicles (MACs) in a simulated environment. We also introduce an ablation study to examine how model complexity, modified by adding or removing layers, can improve performance in tracks with high visual complexity. The proposed methodology combines behavior cloning and simulation to develop autonomous driving systems based on prior works [3, 5, 6, 4].

III Methodology

The proposed methodology based on 3D CNNs to investigate how architectural characteristics and environmental complexity impact the learning and control processes of autonomous vehicles, looking to apply these results to physical mini-autonomous vehicles in the future. It leverages open-source tools, such as MAC and DonkeyCar, as simulation testbeds for evaluating the performance of deep learning models under two distinct scenarios.

The block diagram in Figure 1 illustrates the initial setup in which the simulated vehicle and environments were created and configured. Data from each simulation scenario were collected through manual driving, employing a joystick to control the virtual vehicle, thereby synchronizing front-camera images with corresponding control commands. This dataset provided the foundation for training our 3D CNNs models, including a comparison with standard and modified versions of Recurrent Neural Networks (RNNs). After the training phase, the models were deployed as autopilots to evaluate their performance in autonomous driving tasks and their generalization capabilities. The following subsections present detailed information regarding each stage of development and evaluation.

III-A Neural Network Architectures

For this study, a main architecture based on 3D CNN were selected and compared with a RNN model. The Three-Dimensional Convolutional Neural Network (3D CNN), which extends convolutions into the temporal dimension, allowing joint spatial and temporal analysis of video sequences, shown in Figure 5. This architecture facilitates detection of dynamic patterns across entire video sequences and has been successfully applied in autonomous driving simulations [13].

The Recurrent Neural Network (RNN): designed to process temporal sequences of images and capture the dynamics of the environment, using Long Short-Term Memory (LSTM) layers in the standard version (see Figure 3). These networks apply 2D convolutions to extract spatial features from each image, followed by LSTM layers that model long-term temporal dependencies through gating mechanisms, enabling the prediction of control commands based on the evolving visual environment [14].

For each architecture, a standard and modified versions were trained, incorporating adjustments in architecture or parameters with the goal of improving performance. In particular, the modified RNN version replaces LSTM layers with Gated Recurrent Unit (GRU) layers, which are known for their simpler structure and computational efficiency while maintaining comparable capacity for modeling temporal dependencies [15, 16, 17] (see Figure 3).

Mathematically, the GRU layers implement gating mechanisms that regulate the flow of information, defined by the following equations:

[TABLE]

where $\sigma$ is the sigmoid function, $\odot$ denotes element-wise multiplication, $x_{t}$ is the input, and $h_{t}$ the hidden state at time $t$.

{justify}

Regarding the 3D CNN networks, the modified version introduces residual blocks with LeakyReLU activation functions and Batch Normalization, following the principles of residual networks (ResNet) that ease training of deep models and improve generalization [18, 19] (see Figure 5).

Residual blocks implement a direct shortcut connection that sums the original input $x$ to the output of the residual function $F(x)$, expressed as:

[TABLE]

This structure helps mitigate degradation problems and facilitates gradient flow during backpropagation.

Additionally, Batch Normalization standardizes intermediate activations to stabilize and accelerate training, defined for an activation $x$ in a mini-batch as:

[TABLE]

where $\mu_{B}$ and $\sigma_{B}^{2}$ are the mini-batch mean and variance, and $\gamma,\beta$ are learnable parameters.

These modifications aim to improve the learning capacity, robustness, and computational efficiency of the models to address the complexity of the simulated scenarios and facilitate future deployment on embedded hardware.

III-B Simulation Scenarios

Two simulated scenarios with distinct characteristics were selected to evaluate the performance of the autonomous driving models:

Mini Monaco: a closed circuit with walls on both sides, featuring sharp turns and layout variations that challenge the models’ maneuvering and adaptation capabilities.

Generated Track: a shorter and simpler open track with cones randomly distributed outside the main route, designed to evaluate stability and speed under less demanding conditions.

Both scenarios were implemented in the DonkeyCar, a Unity-based simulated environment, leveraging open-source tools available on GitHub [8, 20]. It offers visual and physical features similar to those of a real environment, facilitating the potential transfer of trained models to physical hardware. Figures 7 and 7 shows both scenarios.

III-C Dataset Creation

The datasets for each scenario were generated by manual driving of the simulated standard vehicle, using a joystick connected to the DonkeyCar framework. The vehicle’s front RGB-D camera images were captured synchronized with the control commands (steering angle and throttle).8 shows the simulation environment configuration.

The camera system was configured to operate as a webcam capturing video at 20 frames per second (FPS), with a resolution of 160×120 pixels and three color channels (RGB) and a depth channel. It was synchronized with the control cycle A total of 10,062 images were collected for Mini Monaco and 10,054 for Generated Track, forming representative datasets for each environment. For data splitting, 80% was used for training and 20% for validation, ensuring temporal and spatial consistency in each partition. The Figure 9 shows how the vehicle perceives each scenario, while Figures 10 and 11 present the control commands data (steering angle and throttle) collected by the user in the respective datasets.

III-D Evaluation Metrics

After training, the trained 3d CNN model was evaluated on two simulated tracks (Mini Monaco and Generated Track) and compared to the trained RNN model. Each model was tested over 30 laps per track, and the following metrics were used for evaluation:

• •

Average Lap Time (sec): Time taken to complete each lap, averaged over 30 laps.

• •

Standard Deviation of Lap Times: Measure of the consistency in lap times.

• •

Track Deviations: Number of times the model went off-track or required major corrections.

• •

Response Speed (sec): Time is taken to adapt to track changes.

IV Results

{justify}

This section presents the evaluation of our trained models trained on the simulated tracks. The models were assessed in terms of stability (lap times), precision in track following, and response speed. Figures 13 and 12 depict the training curves for the 3D CNN and for the RNN models, respectively, in their standard and modified versions, trained on both simulated tracks.

The following subsections present the performance of individual models for each track, along with a summary of the results and the outcomes of the ablation study.

IV-A 3D CNN Model (Figure 12 (b))

The 3D CNN Modified model struggled with unstable performance on Generated Track. Despite no major failures, the model exhibited clear signs of overfitting, with less ability to follow the track. Its lap times varied more than those of the RNN models. The 3D CNN Modified model also underperformed on Mini Monaco. Although it maintained high speed, it struggled to stay on track during several laps, showing signs of overfitting to the training data. However, it adapted quickly during the initial laps.

IV-B RNN model (Figure 13 (a))

The RNN Default model followed the track smoothly with consistent lap times. It performed well in the curves and adapted effectively to layout variations. Track deviations were minimal, with no significant failures during the 30 laps. On this track, the RNN Default model also displayed stable performance, maintaining consistent lap times throughout all 30 laps. The model followed the track with few errors and stayed on the path during every lap. The Modified RNN, utilizing GRU layers, learned to navigate with slightly greater speed than the standard version. It showed improved consistency by staying close to the right side of the track, reducing the frequency of track departures compared to the RNN Default. Similar to its performance on Mini Monaco, the Modified RNN demonstrated good speed and stability on Generated Track. The model showed consistency throughout all 30 laps, adapting well to the simpler layout, with no failures or significant variations.

Overall, Table I presents the average lap times for each model and track evaluated in this study. These values were obtained after 30 attempts per model on each simulated track.

IV-C Ablation Study

To further analyze our proposed method and the RNN version used to compare, we provide an Ablation Study. The first 3D CNN model version that was implemented did not achieve effective learning, as evidenced by its training curves, which indicated signs of overfitting. During testing on the Mini Monaco track—a scenario characterized by high visual complexity—the model performed poorly. It was unable to maintain a consistent trajectory and failed to respond adequately to visual cues in the environment. The vehicle ultimately collided with a wall, highlighting the model’s inability to anticipate and adapt to dynamic visual obstacles present on the track.

To improve robustness, some modifications to the model were made, adjusting its architecture and removing some layers 5, (12, c). The modified model showed significant improvement in Mini Monaco, learning to follow the track with greater precision and stability 5, (12, d). However, despite this improvement, we decided to perform an Ablation Study to evaluate if adding or removing layers could further enhance its performance.

The Ablation Study was conducted on the modified 3D CNN model. Two versions were tested: one with an additional 3D convolutional layer and another with one less 3D convolutional layer. The goal was to observe how these modifications affected the model’s ability to generalize and adapt to more complex scenarios, in this case, the Mini Monaco track. The training data can be seen in 15, 15.

Comparison of the performance of the three modified 3D CNN models can be seen in Table II, showing the average time lap on the Mini Monaco track for each of the trained models. Based on the results, we selected 3D CNN Modified version with one 3D convolutional layer less which was able to outperform the other structures.

V Conclusions

Based on the results obtained from the various models and scenarios, we can conclude that the 3D CNN models, particularly the Modified version, showed better performance on tracks with more complex visual features (Mini Monaco). These models, which integrate spatial and temporal feature extraction, learned the track’s dynamics effectively. However, overfitting was observed in simpler environments, such as the Generated Track, highlighting the need for fine-tuning the model’s complexity.

For the RNN Models, both the RNN Default and the Modified RNN (with GRU layers) demonstrated consistent performance across the simulated tracks. The RNN models showed good adaptability to changes in track layout and environmental factors, achieving stable lap times with minimal errors. However, their performance was limited by the lack of spatial-temporal feature extraction, which the 3D CNN models excel at.

For the last, our Ablation Study revealed that the model with one less layer (Modified 3D CNN) outperformed the others. It achieved faster lap times and better track-following precision, making it the most efficient model. The model with an additional layer exhibited overfitting, resulting in slower performance. This suggests that for the Mini Monaco track, a less complex model strikes the optimal balance between speed and accuracy.

This contribution provides a deeper understanding of how to design efficient machine-learning models for autonomous vehicles in complex environments, which may serve as a foundation for future research and applications in MACs.

Overall, the results indicate that reducing model complexity enhances performance in environments with numerous visual details. The best-performing model was the one with fewer layers, which showed faster speed and more consistent track-following, making it the most suitable choice for complex tracks like Mini Monaco. The RNN models, while more stable, did not outperform the 3D CNN models in more challenging tracks. Future research includes evaluating our real MAC and performing more evaluation with other ANNs and methods.

Acknowledgements

The authors would like to thank the Technological University of Uruguay team and the UruBots robotics competitions team in Rivera.