A Tutorial on Sim-ATAV: Simulation-based Adversarial Testing Framework for Autonomous Vehicles

TL;DR

This paper introduces Sim-ATAV, an open-source simulation framework designed for testing autonomous vehicles through various adversarial test generation techniques, facilitating research and development in safe autonomous driving.

Contribution

It provides a comprehensive tutorial and example usage of Sim-ATAV, enabling researchers to effectively utilize the framework for autonomous vehicle testing.

Findings

Facilitates testing of autonomous vehicles in simulation

Supports various adversarial test generation techniques

Open-source and accessible for research use

Abstract

Testing autonomous vehicles in simulation environments is crucial. Sim-ATAV is an open-source framework developed for experimenting with different test generation techniques in simulation environments for research purposes. This document provides a tutorial on Sim-ATAV with a running example.

Figures

Tables

Table 1. Table 1: Simulation Environment Class.

Class Name: SimEnvironment
Property	Default	Description
fog	None	Keeps fog object.
heart_beat_config	None	The heartbeat configuration.
view_follow_config	None	The viewpoint configuration.
ego_vehicles_list	[]	The list of Ego vehicles.
agent_vehicles_list	[]	The list of agent vehicles.
pedestrians_list	[]	The list of pedestrians.
road_list	[]	The list of roads.
road_disturbances_list	[]	The list of road disturbances.
generic_sim_objects_list	[]	The list of generic simulation objects.
control_params_list	[]	The list of controller parameters that will be set in the run time.
initial_state_config_list	[]	The list of initial state configurations.
data_log_description_list	[]	The list of data log descriptions.
data_log_period_ms	None	Data log period (ms).

Table 2. Table 2: Simulation Entity: Road.

Class Name: WebotsRoad
Property	Default	Description
def_name	“STRROAD”	Name as it appears in the simulation tree.
road_type	“StraightRoadSegment”	Road type name as used by Webots.
rotation	[0, 1, 0, math.pi/2]	Rotation of the road
position	[0, 0.02, 0]	Starting position.
number_of_lanes	2	Number of lanes.
width	number_of_lanes * 3.5	Road width (m).
length	1000	Road length (m).

Table 3. Table 3: Simulation Entity: Vehicle.

Class Name: WebotsVehicle
Property	Default	Description
def_name	“”	Name as it appears in Webots simulation tree.
vhc_id	0	Integer ID for referencing to the vehicle.
vehicle_model	“AckermannVehicle”	Vehicle model can be any model name available in Webots.
rotation	[0, 1, 0, 0]	Rotation of the object.
current_position	[0, 0.3, 0]	x,y,z values of the position.
color	[1, 1, 1]	R,G,B values of the color in the range [0,1].
controller	“void”	Name of the vehicle controller.
is_controller_name_absolute	False	Indicates where to find the vehicle controller. Find the details below.
vehicle_parameters	[]	Additional parameters for the vehicle object.
controller_parameters	[]	Parameters that will be sent to the vehicle controller.
controller_arguments	True	Arguments passed to the vehicle controller executable.
sensor_array	[]	List of sensors on the vehicle (WebotsSensor objects).

Table 4. Table 4: Simulation Entity: Sensor.

Class Name: WebotsSensor
Property	Default	Description
sensor_type	“”	Type of the sensor as defined in Webots.
sensor_location	FRONT	Sensor slot enumeration.
		FRONT, CENTER, LEFT, RIGHT, TOP$>$
sensor_fields	[]	List of WebotsSensorField objects to customize the sensor.
Class Name: WebotsSensorField
field_name	“”	Name of the field that will be set.
field_val	“”	Value of the field.

Table 5. Table 5: Simulation Entity: Pedestrian.

Class Name: WebotsPedestrian
Property	Default	Description
def_name	“PEDESTRIAN”	Name as it appears in Webots simulation tree.
ped_id	0	Integer ID for referencing to the pedestrian.
rotation	[0, 1, 0, math.pi/2.0]	Rotation of the object.
current_position	[0, 0, 0]	x,y,z values of the position.
shirt_color	[0.25, 0.55, 0.2]	R,G,B values of the shirt color in the range [0, 1].
pants_color	[0.24, 0.25, 0.5]	R,G,B values of the pants color in the range [0, 1].
shoes_color	[0.28, 0.15, 0.06]	R,G,B values of the shoes color in the range [0, 1].
controller	“void”	Name of the pedestrian controller.
target_speed	0.0	Walking speed of the pedestrian.
trajectory	[]	Walking path of the pedestrian.

Table 6. Table 6: Simulation Entity: Fog.

Class Name: WebotsFog
Property	Default	Description
def_name	“FOG”	Name as it appears in Webots simulation tree.
fog_type	“LINEAR”	Defines the type of the fog gradient.
color	[0.93, 0.96, 1.0]	R,G,B values of the fog color in the range [0, 1].
visibility_range	1000	Visibility range of the fog (m).

Table 7. Table 7: Simulation Entity: Road Disturbance Object.

Class Name: WebotsRoadDisturbance
Property	Default	Description
disturbance_id	1	Object ID for later reference.
disturbance_type	INTERLEAVED	Enumerated type of the disturbance.
		INTERLEAVED,
		FULL_LANE_LENGTH,
		ONLY_LEFT, ONLY_RIGHT$>$
rotation	[0, 1, 0, 0]	Rotation of the object.
position	[0, 0, 0]	x,y,z values of the position.
length	100	Length of the bumpy road surface (m).
width	3.5	Width of the corresponding lane.
height	0.06	Height of the disturbance (m).
surface_height	0.02	Height of the corresponding road surface (m).
inter_object_spacing	1.0	Distance between repeating solid objects on the road (m).

Table 8. Table 8: Simulation Entity: Generic Simulation Object.

Class Name: WebotsSimObject
Property	Default	Description
def_name	“”	Name as it appears in Webots simulation tree.
object_name	“Tree”	Type name of the object. Must be same as the name used by Webots.
object_parameters	[]	List of (field name, field value) tuples as strings. Names must be same as the field names used by Webots.

Table 9. Table 9: Controller parameters.

Class Name: WebotsControllerParameter
Property	Default	Description
vehicle_id	None	ID of the corresponding vehicle.
parameter_name	“”	Name of the parameter as string.
parameter_data	[]	Parameter data.

Table 10. Table 10: Heartbeat Configuration.

Class Name: WebotsSimObject
Property	Default	Description
sync_type	NO_HEART_BEAT	NO_HEART_BEAT: Do not report simulation status.
		WITHOUT_SYNC: Report the status and continue simulation.
		WITH_SYNC: Wait for new commands after each heartbeat.
period_ms	10	Period of the simulation status reporting.

Table 11. Table 11: Data Item Description.

Class Name: ItemDescription
Property	Default	Description
item_type	None	Type of the corresponding simulation entity.
		TIME, VEHICLE, PEDESTRIAN$>$
item_index	None	Index of the corresponding simulation entity.
item_state_index	None	Index of the state that will be recorded.

Table 12. Table 12: Initial State Configuration.

Class Name: InitialStateConfig
Property	Default	Description
item	None	Data item for the corresponding state as an ItemDescription object.
value	None	Initial value of the corresponding state.

Table 13. Table 13: Viewpoint Configuration.

Class Name: ViewFollowConfig
Property	Default	Description
item_type	None	Type of the corresponding simulation entity.
		TIME, VEHICLE, PEDESTRIAN$>$
item_index	None	Index of the corresponding simulation entity.
position	None	$[x,y,z]$ values of the initial position of the viewpoint.
rotation	None	$[x,y,z,\theta ]$ Rotation of the viewpoint.

Table 14. Table 14: Simulation Configuration.

Class Name: SimulationConfig
Property	Default	Description
world_file	“../Webots_Projects/	Name of the base Webots
	worlds/test_world_1.wbt”	world file.
server_port	10021	Port number for connecting to Simulation Supervisor.
server_ip	“127.0.0.1”	Port number for connecting to Simulation Supervisor.
sim_duration_ms	50000	Simulation duration (ms).
sim_step_size	10	Simulation time step (ms).
run_config_arr	[]	List of run configurations for supporting multiple parallel simulation executions (RunConfig objects).
Class Name: RunConfig
Property	Default	Description
simulation_run_mode	FAST_NO_GRAPHICS	Webots run mode (simulation speed).
		REAL_TIME:
		Real-time speed,
		FAST_RUN:
		As fast as possible with visualization,
		FAST_NO_GRAPHICS:
		As fast as possible without visualization.

Table 15. Table 15: A list of Sim-ATAV methods for loading test cases from csv files.

load_experiment_data(file_name, header_line_count=6, index_col=None)
Description:	Loads test cases from a csv file.
Arguments:	file_name: Name of the csv file
	header_line_count: Number of header lines to skip. (default: 6)
	index_col: Index of the column that is used for data indexing. (default: None)
Return:	A pandas data frame (a tabular data structure) containing all test cases.
get_experiment_all_fields(exp_data_frame, exp_ind)
Description:	Returns all parameters for a test case (one row of the test table).
Arguments:	exp_data_frame: Data frame that was read using
	load_experiment_data
	exp_ind: Index of the experiment
Return:	A row from the test table that contains the requested test case.
get_field_value_for_current_experiment(cur_experiment, field_name)
Description:	Returns the value of the requested test parameter.
Arguments:	cur_experiment: Current test case that was read using
	get_experiment_all_fields.
	field_name: Name of the test parameter.
Return:	Value of the requested parameter for the given test case.

Table 16. Table 16 : Describing test parameters for creating a covering array.

Parameter	Name in ACTS	Possible Values
ego_init_speed_m_s	ego_init_speed	0, 5, 10, 15
ego_x_pos	ego_x_position	15, 20, 25
pedestrian_speed	pedestrian_speed	2,3,4,5

Full text

1 An Overview of Sim-ATAV

Sim-ATAV [6, 7, 8] is a tool developed for experimenting with different test generation techniques in simulation environments for research purposes as described in [8]. It is mainly developed in Python and it uses the open-source robotics toolbox Webots [2] for 3D scene generation, vehicle and sensor modeling and simulation. Sim-ATAV can be interfaced with covering array generation tools like ACTS [4] and with falsification tools like S-TaLiRo [3], which is a MATLAB toolbox. Sim-ATAV is publicly available as an open-source project [6].

Figure 1 provides a high-level overview of the framework. The main functionality of Sim-ATAV is provided by Simulation configurator and Simulation Supervisor blocks. Simulation configurator block represents the Sim-ATAV API for the user script to create a simulation and to receive the results. Simulation Supervisor block represents the part of the framework that executes inside the robotics simulation toolbox Webots. It uses the Webots API to (1) modify the simulation environment, e.g., add/configure vehicles, roads, pedestrians, provide vehicle controllers, (2) execute the simulation, and (3) collect information, e.g., the evolution of vehicle states over the simulation time. Simulation Supervisor receives the requested simulation configuration from the Simulation configurator over socket communication. When the simulation environment is set up, Simulation Supervisor requests Webots to start the simulation. User-provided controllers control the motion of the vehicles and pedestrians. At the end of the simulation, Simulation Supervisor sends the collected simulation trace to Simulation configurator.

External tools like ACTS can be used to generate combinatorial tests with covering arrays. Sim-ATAV provides functions to read covering array test scenarios from csv files. For falsification, S-TaLiRo is used to iteratively sample new configurations until a failure case is detected. Optionally, initial samples for the configurations can be read by S-TaLiRo from the covering array csv files. The sampled configurations in S-TaLiRo are passed as parameters to test generation functions using the Python interface provided in MATLAB, i.e., by calling Python functions directly inside MATLAB. The test generation functions return the simulation trace back to MATLAB (and to S-TaLiRo) as return values. More detail on falsification methods is available in [8, 3]. A running example of test generation is provided in the upcoming sections.

Although they are not essential for the test generation purposes, Sim-ATAV also comes with some vehicle controller implementations as well as some basic perception system, sensor fusion, path planning and control algorithms that can be utilized by the user for Ego or Agent vehicle control.

2 Installation Instructions

Sim-ATAV requires Python 3.7 and Webots for basic functionality. For some controllers and test generation approaches, there are other requirements like MATLAB®, S-TaLiRo, TensorFlow, SqueezeDet etc. The framework has been tested in Windows® 10 with specific versions of the required packages.

Firstly, Sim-ATAV should be either downloaded or cloned using a git client 111Sim-ATAV: https://cpslab.assembla.com/spaces/sim-atav. For Windows®, the preferred installation approach is to use setup_for_windows.bat. Once executed, it will guide the user through the installation process and automate the process as much as possible. This script is not advanced and may fail for some systems. If the script fails, please try the steps below for a manual installation.

Below are the steps for the manual installation. All the paths are given relative to the root folder for the Sim-ATAV distribution. In case any problems are experienced during the installation of the packages, most of the packages can also be found in Christoph Gohlke’s website222Christoph Gohlke’s website: https://www.lfd.uci.edu/ gohlke/pythonlibs/:

Install Python 3.7-64 Bit

• 2.

Install Webots r2019a.

• 3.

(optional) If the system has CUDA-enabled GPU and it will be utilized for an increased performance:

• a.

Install CUDA Toolkit 10.0

• b.

Install CUDNN 7.3.1

• 4.

Download Python_Dependencies from http://www.public.asu.edu/ etuncali/downloads/ and unzip it next to this installation script. The Python wheel (.whl) files should be directly under ./Python_Dependencies/

• 5.

Install commonly used Python packages:

• a.

Install Numpy+MKL 1.14.6 either from Python_Dependencies folder or from Christoph Gohlke’s website:

⬇

pip3 install –upgrade Python_Dependencies/numpy-1.14.6+mkl-cp37-cp37m-win_amd64.whl

• b.

Install scipy 1.2.0:

⬇

pip3 install scipy==1.2.0

• c.

Install scikit-learn:

⬇

pip3 install scikit-learn

• d.

Install pandas:

⬇

pip3 install pandas

• e.

Install Absl Py:

⬇

pip3 install absl-py

• f.

Install matplotlib:

⬇

pip3 install matplotlib

• g.

Install pykalman:

⬇

pip3 install pykalman

• h.

Install Shapely:

⬇

pip3 install Shapely

• !

If any problems are experienced during the installation of Shapely, it can be installed from Python_Dependencies folder:

⬇

pip3 install Python_Dependencies/Shapely-1.6.4.post1-cp37-cp37m-win_amd64.whl

• i.

Install dubins:

⬇

pip3 install dubins

• !

If any problems are experienced when installing pydubins:

One option is to go into the folder Python_Dependencies/pydubins and execute python setup.py install. Another option is the following:

• (i)

Download pydubins from github.com/AndrewWalker/pydubins.

• (ii)

Do the following changes in dubins/src/dubins.c:

⬇

#ifndef M_PI

#define M_PI 3.14159265358979323846

#endif

• (iii)

Call python setup.py install inside pydubins folder.

• 6.

For controllers with DNN (Deep Neural Network) object detection:

*! Currently, Python 3.7 support for Tensorflow is provided by a 3rd party (only for Windows). Installation wheels are provided under Python_Dependencies folder. *

Check if the system CPU supports AVX2 (for increased performance) 333A list of CPUs with AVX2 is available at:

https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2.

• a.

If the system GPU has CUDA cores, CUDNN is installed and the system CPU supports AVX 2: Install Tensorflow-gpu with AVX2 support.

⬇

pip3 install –upgrade Python_Dependencies/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl

• b.

If the system GPU has CUDA cores, CUDNN is installed and the system CPU does NOT support AVX 2: Install Tensorflow-gpu without AVX2 support.

⬇

pip3 install –upgrade Python_Dependencies/sse2/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl

• c.

If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU supports AVX 2: Install Tensorflow with AVX2 support.

⬇

pip3 install –user –upgrade Python_Dependencies/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl

• d.

If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU does NOT support AVX 2: Install Tensorflow without AVX2 support.

⬇

pip3 install –user –upgrade Python_Dependencies/sse2/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl

• 7.

Install Python Dependencies of SqueezeDet (if the existing controllers that use SqueezeDet will be used). There is no need to install SqueezeDet separately, as it is provided in the framework.

• a.

Install joblib:

⬇

pip3 install –upgrade joblib

• b.

Install opencv:

⬇

pip3 install –upgrade opencv-contrib-python

• c.

Install pillow:

⬇

pip3 install –upgrade Pillow

• d.

Install easydict:

⬇

pip3 install –upgrade easydict==1.7

! If any problems are experienced while installing easydict, try the following:

⬇

cd Python_Dependencies/easydict-1.7\

python setup.py install

cd ../..

• 8.

To design Covering Array Tests: Please request a copy and install ACTS from NIST444ACTS tool can be requested from

https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software/downloadable-tools.

• 9.

To do robustness-guided falsification, MATLAB® and S-TaLiRo are needed:

• a.

Install MATLAB from Mathworks(tested with r2017b).

• b.

Install S-TaLiRo 555S-TaLiRo is available at https://sites.google.com/a/asu.edu/s-taliro/s-taliro.

• 10.

After the installation is finished, the Python package wheel files that are under the folder named Python_Dependencies can be deleted to save some disk space.

Setting to Utilize GPU:

If the system has a CUDA-enabled GPU, and CUDA Toolkit, CUDNN are installed, the variable has_gpu should be set to True in the following file to make the experiments use the system GPU for SqueezeDet:

Sim_ATAV/classifier/classifier_interface/gpu_check.py.

3 Reference Manual

3.1 Simulation Entities

Sim-ATAV starts building a testing scenario from an existing Webots world file provided by the user, which we will call as base world file. The user has the option to have all or some of the simulation entities, i.e., roads, vehicles, etc., saved in the base world file before the test generation time. The user can use the functionality provided by Sim-ATAV to add more simulation entities to the world at the time of test generation. This functionality is especially useful when the search space for the tests contain some parameters of the simulation entities such as road width, number of lanes, positions of the vehicles.

This section describes the most commonly used simulation entities that can be programmatically added into the simulation world at the time of test generation. Note that Sim-ATAV may not provide the functionality to add all possible types of simulation objects that are supported by Webots. In this section, the class names and properties with their default values for the simulation entities supported by Sim-ATAV are provided. Test Generation Script which is developed by the user typically creates instances of required simulation entities and uses the provided functions to add those entities into the test scenario. For modifying the simulation environment beyond the capabilities of Sim-ATAV, the user can do the changes manually and save in the base world file or modify the source code of Sim-ATAV to add or change the capabilities as needed. For a deeper understanding of the possibilities, the reader is advised to get familiar with the Webots simulation environment and details of available simulation entities [2].

The class properties for the simulation entities are provided in the tables below. The first row of each table gives the class name. Other rows start with the property name, gives the default value of the property (the value used if not explicitly changed by the user), and a brief description of the property. For each class, a related source code snippet from a running example is provided. The original source code for the running example can be found as tests/tutorial_example_1.py in the Sim-ATAV distribution.

Figure 2 is an image from the scenario described in the running example. A yellow Ego vehicle is behind a pedestrian walking in the middle of a 3-lane road, and an agent vehicle is approaching from the opposite direction in the next lane. There is a bumpy road surface for a short distance, and a stop sign placed on the right side of the road. This is only a simple example to illustrate how to use Sim-ATAV.

General information:

In Webots, all simulation entities are kept in a simulation tree and they can be easily accessed using their DEF_NAME property. Position fields are 3D arrays as $[x,y,z]$, keeping the value of the position at each axis in Webots coordinate system, where $y$ is typically the vertical axis to the ground (in practice, this depends on the Webots world file provided by the user). Rotation fields are 4D arrays as $[x,y,z,\theta]$, where $x,y,z$ represents a rotation vector and $\theta$ represents the rotation around this vector in clockwise direction.

3.1.1 Simulation Environment

Once an object is created for a simulation entity, the user can utilize the corresponding function provided by Sim-ATAV to add the entity to the scenario. An easier alternative is to utilize the SimEnvironment class provided by Sim-ATAV. An object of this class can be populated with the necessary simulation entities and passed to the simulation environment with a single function call. Table 1 summarizes the SimEnvironment class.

Example 1

For the running example, we start with an empty simulation environment. Listing 1 creates an empty SimEnvironment object that will later keep the required simulation entities.

3.1.2 Road

A user-configurable road structure to use in the simulation environment is given in Table 2.

Example 2

Listing 2 provides a source code snippet that creates a 3-lane straight road segment lying between 1000m and -1000m along the x-axis.

3.1.3 Vehicle

A user-configurable vehicle class to use in the simulation environment in Table 3.

The vehicle_model field of a vehicle object should match with the models available in Webots (or any custom models added to Webots by the user). Webots r2019a version provides vehicle model options ToyotaPrius, CitroenCZero, BmwX5, RangeRoverSportSVR, LincolnMKZ, TeslaModel3 as well as truck, motorcycle and tractor models. When is_controller_name_absolute is set to true, Webots will load the given controller from Webots_Projects/controllers folder, otherwise, it will load the controller named vehicle_controller which is located under the same folder but will take the controller name as an argument.

Example 3

We can now create vehicles and place them on the road that was created above. Listing 3 provides an example source code snippet that creates an Ego vehicle at the position $x=20m$, $y=0$, and an agent vehicle at the position $x=300m$, $y=3.5m$, vehicles facing toward each other. The controllers for the vehicles are set and the controller arguments are provided. The arguments accepted are controller-specific. The vehicle controllers used in this example can be found under the folder Webots_Projects/controllers.

3.1.4 Sensor

Table 4 provides an overview of the WebotsSensor class that can be used to describe a sensor. Webots vehicle models have specific sensor slots on the vehicles. These are typically TOP, CENTER, FRONT, RIGHT, LEFT. The property sensor_type accepts the type of the sensor which should match the type used by Webots. The translation field of the sensor can be used to place the sensor to a different position relative to its corresponding sensor slot. As sensors can vary a lot in terms or parameters, sensor_fields property is provided to accept names and values of the desired parameters as a list of WebotsSensorField objects for flexible configuration of sensors.

Example 4

An example source code of adding sensors to vehicles is provided in Listing 4. In this example, we add a compass and a GPS device that are used by the controllers for path following. The receiver device added to the vehicles is used by the controllers to receive new commands from Simulation Supervisor. We add the receivers because we will later need them to update target paths of the vehicles. Note that although the necessary infrastructure for this approach is provided by Sim-ATAV, it is implementation specific and not mandated. In this example, we also add a radar device to Ego vehicle for collision avoidance.

3.1.5 Pedestrian

The user-configurable pedestrian class, WebotsPedestrian, to use in the simulation environment is described in Table 5. Target speed and target path (trajectory) of the pedestrian are automatically passed as arguments to the given controller.

Example 5

Listing 5 provides an example code snippet to create a pedestrian object, and provide a target speed and a path to define the motion of the pedestrian.

3.1.6 Fog

User-configurable fog class to use in the simulation environment is summarized in Table 6.

Example 6

No camera is involved in this scenario, hence fog will not impact the performance of the controller. However, an example source code snippet is provided in Listing 6 for reference.

3.1.7 Road Disturbance

Road disturbance objects are solid triangular objects placed on the road surface to emulate a bumpy road surface. WebotsRoadDisturbance, which is described in Table 7, contains the properties to describe how the solid objects will be placed to create a bumpy road surface.

Example 7

An example road disturbance object creation is given in Listing 7.

3.1.8 Generic Simulation Object

Generic simulation object is for adding any type of object into the simulation which is not covered above. For these objects, there are no checks performed or there are no limitations on the field values. The user can manually create any possible Webots object by setting all of its non-default field values.

Example 8

In Listing 8, although it is not expected to have an impact on the controller performance, a Stop Sign object is created as a generic simulation object example for reference.

3.2 Configuring the Simulation Execution

3.2.1 Additional Controller Parameters

Depending on the application and implementation details, controller parameters can be directly given to the controllers or they can be sent in the run-time. To emulate runtime inputs, such as human commands, Sim-ATAV can transmit controller commands over virtual radio communication. The controller should be able to read those commands from a radio receiver and a receiver object should be added to one of the sensor slots of the vehicles. This is an optional approach and the user is free to use other approaches such as reading data from a file, using socket communications etc.

Example 9

An example controller parameter creation.

3.2.2 Heartbeat Configuration

Simulation Supervisor can periodically report the status of the simulation execution to Simulation Configurator with heartbeats. Simulation Configurator can further modify the simulation environment on the run time by responding to the heartbeats.

Example 10

An example heartbeat configuration.

3.2.3 Data Log Item

Data log items are the states that will be collected into the simulation trace by Simulation Supervisor.

Example 11

An example list of data log items for simulation trajectory generation.

3.2.4 Initial State Configuration

Initial state configuration objects are for setting an initial state ofa simulation entity.

Example 12

An example initial state configuration.

3.2.5 Viewpoint Configuration

Webots viewpoint can automatically follow a simulation entity throughout the simulation. Viewpoint configuration is used to describe which object to follow if desired.

Example 13

An example viewpoint configuration.

3.2.6 Simulation Configuration

Simulation configuration provides the information necessary to execute a simulation through TCP/IP connection between Simulation Configurator and Simulation Supervisor.

Example 14

An example simulation configuration creation.

3.3 Executing the Simulation

To start execution of a scenario, it is advised to first start Webots with the base world file manually. If Webots crashes or communication is lost, Sim-ATAV can restart Webots when necessary (only is Webots is executing in the same system). To start the simulation, Simulation Configurator should be used to connect to the Simulation Supervisor, send the simulation environment and configuration details, start the simulation, and finally collect the simulation trace. Below is an example source code for this step.

Example 15

An example for execution of the simulation.

4 Combinatorial Testing

Sim-ATAV provides basic functionality to read test scenarios from csv files, in which, each row represents a test case and each column holds the values of a test parameter. This feature can be used to automatically execute a set of predefined test cases. The user can create csv files that contain a set of test cases either manually or by using a tool. Table 15 provides a list of methods provided in covering_array_utilities.py.

For combinatorial testing, ACTS from NIST [4] can be used as a tool for generating covering arrays. Providing a complete guide on covering arrays and how to use ACTS is out of the scope of this chapter. The user of ACTS creates a system definition by defining the system parameters and providing the possible values for each parameter. The system definition is saved as an xml file. ACTS can generate a covering array of desired strength and export the outputs in a csv file. An example of a system definition and a corresponding 2-way covering array are provided in Sim-ATAV distribution in the files tests/examples/TutorialExampleSystemACTS.xml and

tests/examples/TutorialExample_CA_2way.csv, respectively. Below is an example of reading test cases from a csv file and running each test one by one. Note that, in the example, simulation trajectories are not evaluated. Outputs of each test case should be evaluated to decide the test result.

Example 16

An example for running covering array test cases.

In this example, a 2-way covering array for the test parameters is generated in ACTS. Table 16 shows the original test parameter name from the file tutorial_example_1.py, the name used in ACTS, and the possible values for each parameter (corresponding ACTS file is TutorialExampleSystemACTS.xml).

After entering the parameter descriptions in Table 16 to ACTS, a 2-way covering array is generated and exported as a csv file. Listing 16 shows the contents of the output csv file. The file is available as: tests/examples/TutorialExample_CA_2way.csv.

Below is the source code that is reading the test cases from the csv file using the functions listed in Table 15 and running each test case one by one.

5 Falsification / Search-based Testing

For performing falsification, a CPS falsification tool like S-TaLiRo [3] can be used. As S-TaLiRo is in MATLAB, we need to call the Sim-ATAV test cases which are developed in Python from MATLAB. In this section, first a simple approach is described to call the test cases from MATLAB, then a simple S-TaLiRo setup is described to perform falsification. This chapter does not provide a complete guide to S-TaLiRo. Reader is referred to S-TaLiRo website for further details 666S-TaLiRo iwebsite: https://sites.google.com/a/asu.edu/s-taliro/s-taliro.

5.1 Connecting to MATLAB

MATLAB® has built-in support to call Python functions. However, type conversions are required both in Sim-ATAV and in the MATLAB code 777The user is free to use a different approach than what is described here.. Sim-ATAV provides basic functionality to convert simulation trace to a format (single dimensional list) that can be easily interpreted in MATLAB. Listing 18 shows the necessary updates on the Python side. First, sim_duration argument is added to run_test function allow S-TaLiRo run different length of simulations. Next, for_matlab argument is added to tell the function is called from MATLAB so that experiment_tools.npArray2Matlab method from Sim-ATAV is used to convert simulation trace to a single list for MATLAB.

On the MATLAB side, we need a wrapper function to call the function run_test. Listing 19 provides an example to such a wrapper function. This MATLAB code can be found in Sim-ATAV distribution in the file /tests/examples/run_tutorial_example_from_matlab.m. Unused return parameters YT, LT, CLG, GRD and input arguments steptime, InpSignal are to maintain compatibility with S-TaLiRo function call format.

5.2 Connecting to S-TaLiRo

To perform falsification with S-TaLiRo, an MTL specification should be defined, ranges for test parameters and the MATLAB code, which is given in Listing 19, for running a simulation should be provided to S-TaLiRo as the model under test. Listing 20 gives a very simple example for falsification with an MTL requirement that checks x and y coordinates of Ego and agent vehicles to decide a collision. This MATLAB code can be found in Sim-ATAV distribution in /tests/examples/run_falsification.m. The requirement is not to collide onto the agent vehicle. Note that, this requirement is very simplified to provide a simple example source code. A more complete check for collision would require incorporating further vehicle details and/or extracting collision information from the simulation.

6 Other Remarks

A user guide for Sim-ATAV is provided with a running example. The source code for the running example used in this chapter is available in Sim-ATAV distribution under tests/examples folder. As Sim-ATAV is a research tool that is not directly targeted for production-level systems special caution should be taken before using it for testing any critical functionality. Sim-ATAV is still evolving and it may contain a number of bugs or parts that are open to optimization. Here, only major functionality provided by Sim-ATAV is discussed. It provides further functionality like a number of sample vehicle controller subsystems, additional computations on simulation trajectory such as collision detection, and functionality toward evaluating perception-system performance. For a deeper understanding of Sim-ATAV’s capabilities, the reader is encouraged to go through the source code for examples and experiments that are used as case studies for publications.