Surface Recognition for e-Scooter Using Smartphone IMU Sensor

TL;DR

This paper presents a data-driven deep learning approach using smartphone IMU sensors to accurately distinguish between road and sidewalk surfaces for e-scooter safety enhancement.

Contribution

It introduces a novel method employing deep neural networks with smartphone IMU data for surface recognition, improving safety and stability in e-scooter operation.

Findings

Deep neural networks outperform classical machine learning models in surface classification.

The method achieves high accuracy in distinguishing road from sidewalk surfaces.

Smartphone IMU sensors are effective for real-time surface recognition in e-scooters.

Abstract

In recent years, as the use of micromobility gained popularity, technological challenges connected to e-scooters became increasingly important. This paper focuses on road surface recognition, an important task in this area. A reliable and accurate method for road surface recognition can help improve the safety and stability of the vehicle. Here a data-driven method is proposed to recognize if an e-scooter is on a road or a sidewalk. The proposed method uses only the widely available inertial measurement unit (IMU) sensors on a smartphone device. deep neural networks (DNNs) are used to infer whether an e-scooteris driving on a road or on a sidewalk by solving a binary classification problem. A data set is collected and several different deep models as well as classical machine learning approaches for the binary classification problem are applied and compared. Experiment results on a route containing the two surfaces are presented demonstrating the DNNs ability to distinguish between them.

Tables

Table 1. TABLE I: Performance metrics for each model on the validation set for different window sizes.

Model	Window Size [s]	Accuracy	F1	FPR
CNN	1	0.78	0.72	0.22
	2	0.83	0.83	0.15
	3	0.78	0.78	0.16
LSTM	1	0.86	0.88	0.15
	2	0.85	0.86	0.17
	3	0.52	0.47	0.23
LSTM-CNN	1	0.88	0.88	0.14
	2	0.90	0.90	0.13
	3	0.93	0.93	0.13
SVM	1	0.73	0.70	0.16
	2	0.72	0.68	0.16
	3	0.71	0.66	0.18
Random Forest	1	0.79	0.79	0.23
	2	0.78	0.79	0.22
	3	0.80	0.82	0.20

Equations

$$\ell (p_i,Y_i)=-\left[Y_i\cdot \log (p_i)+(1-Y_i)\cdot \log (1-p_i)\right],$$

$$L=\frac{1}{N}\sum _{i=1}^N\ell (p_i,Y_i),$$

Full text

I Introduction

Road surface recognition is the process of identifying and classifying different types of road surfaces that a vehicle may encounter during operation [1, 2, 3, 4]. This work focuses on road surface recognition for e-scooters. A mode of transportation that has become increasingly popular in urban areas in recent years. In particular, this work presents a method for distinguishing between paved asphalt roads and sidewalks for e-scooters using only a mobile device.

Accurate road surface recognition is important for several reasons. First, different road surfaces can have different impacts on the stability and handling of an e-scooter [5]. For example, a smooth, paved road may provide a stable and comfortable ride, while a textured, rough sidewalk may be more difficult to navigate. By identifying the road surface, e-scooters can adjust their performance accordingly, ensuring a safe and enjoyable ride for the user. Additionally, road surface recognition can be used to improve the efficiency of e-scooters. Different road surfaces can have different levels of resistance, which can impact the amount of energy required to travel a given distance. By accurately identifying the road surface, e-scooters can optimize their power output and energy consumption, extending the range of the vehicle and improving the overall user experience. Finally, driving an e-scooter on a sidewalk is illegal in many places. An accurate method that determines if a rider is on a sidewalk or on a road can provide fleet managers and micromobility companies with a valuable tool.

In recent years, several approaches were developed to address the problem of road surface recognition for different transportation modes. In[6] the authors present a road surface recognition system for bicycles using several sensors attached to the bicycle including a camera an inertial measurement unit (IMU) and a speedometer. In [7], the authors described a method to estimate the roughness of a surface on which a car is traversing using LiDAR as well as a global positioning system (GPS) receiver and IMU. In [8] the authors describe a method of surface recognition for pedestrians using an IMU and a microphone attached to the users’ shoes. A method to estimate the condition of the road surface using lasers mounted on a vehicle was proposed in [9]. In [10] a deep classifier that estimates the road surface was trained using data collected by cameras. In [1] an algorithm is presented that extracts features from an IMU sensor on a car and uses K nearest neighbors algorithm to detect abnormal road surface conditions.

Other related works where machine learning algorithms, and in particular deep neural networks, were applied to IMU sensor data include [11] where DNN-based multi-models together with an EKF were proposed to deal with the global navigation satellite system (GNSS) outage. In [12] it was demonstrated that integration of DNN into classical signal processing methods can boost navigation performance. In [13] the authors demonstrate how a DNN model can estimate a mounting yaw angle of a smartphone strapped to a vehicle. Another related work is [14] where DNNs were used to learn linear velocities from IMU sensors in cars. To the best of the author’s knowledge, this work is the first to present a road surface recognition algorithm, including a DNN, for e-scooters.

In this paper, a novel approach to road surface recognition for e-scooters using IMU sensors on smartphones is developed. The method is data-driven and relies on a DNN binary classifier. IMU sensors are widely available in smartphones. By mounting a smartphone on an e-scooter and collecting data from the IMU sensors, and applying a DNN-based algorithm, it is possible to infer the type of road surface the e-scooter is traveling on. The DNN is able to detect the signature associated with each type of road surface and discern between roads and sidewalks. The proposed approach has several advantages: It is cost-effective, as it utilizes sensors that are already present in many smartphones; It is also lightweight and compact, making it well-suited for use on e-scooters; Additionally, smartphones are increasingly being used as onboard computers for e-scooters, providing a convenient platform for integrating the suggested road surface recognition system.

The rest of the paper is organized as follows: In Section II the data collection and dataset generation processes are presented. In Section III the proposed ML algorithms are detailed. In Section IV the experimental results are described. In particular, the results of several algorithms on a validation set are presented. An experiment on a route containing both types of surfaces is also presented. Finally, in Section V conclusions are given.

II Data set creation

Any data-driven method requires an offline collection of data and model training. This section contains a description of the data collection and processing. A schematic view of the data process can be seen in Fig. (2).

II-A Raw data collection

IMU acceleration and gyroscope data was collected with an iPhone 13 strapped to an e-scooter’s handle. Data was recorded at the maximum rate of 100 [Hz]. A total of 100 minutes were collected for this task in 12 distinct driving sessions conducted by the same rider. Different driving sessions had distinct routes to increase diversity in the data set. See Fig. (1) for an image of the route and the collecting device (smartphone). Of the routes, 7 were entirely on a sidewalk, and the rest were on the road. To create a validation set, 2 of the drives were set aside. All in all, 10 drives totaling 80 minutes were used for training and the two drives, one on a sidewalk and one on the road, for validation.

II-B Pre-processing

The collected data consists of acceleration $[m/s^{2}]$ and angular velocity $[rad/s]$ in three axes. Before passing the acceleration and gyroscope data to the classifier, the following pre-processing tasks are applied:

Apply a low-pass filter with a cut-off frequency of 10 [Hz]. 2. 2.

Down-sample the data to 20 [Hz]. 3. 3.

Rotate (only roll and pitch) the measured acceleration and angular velocity vectors to the navigation frame where the first component points to the e-scooter forward direction and the third component is parallel to the gravity vector. 4. 4.

Subtract the gravity component from the third acceleration component.

These steps are applied both offline when training the model and online as a part of the classification algorithm. The goal of the described processing is to simplify the signal and avoid unnecessary noise and other information that can affect the performance of the classifier. It is noted that the above processing procedure has appeared in various prior works e.g. [15, 16, 13]. It is provided in Figure 2 for this work to be self-contained.

II-C Data generation

To create the dataset the recorded drives were partitioned to windows of $S$ seconds. In this work the value of $S$ was considered a hyper-parameter in the set $\{1,2,3\}$. The overall data set consists of pairs $X\in\mathrm{R}^{20\cdot S\times 6}$ of IMU measurements of $S[s]$ sampled in 20 [Hz] and $Y\in\{0,1\}$ with $1$ indicating a sidewalk and [math] indicating a paved road.

III Classifiers

Using the dataset collected as described above we train a total of 5 ML models. Each of the models outputs one or zero signifying sidewalk or road.

III-A Models

Five ML models where tested. The most common DNN-based models for this type of task are convolutional neural network (CNN) [17] and long short-term memory (LSTM) [18]. Each one of them were used as a basis for a DNN architecture and both of them were combined in a fused architecture. In addition, two classical ML models were tested: support vector machine (SVM) [19] and random forest [20]. A description of each model is provided:

CNN - A convolutional neural network with two convolution layers with 8 channels and 3 kernels followed by dropout and mean pooling with a rectified linear unit (ReLU). One convolution layer followed by dropout and max pooling with ReLU. Two linear layers with ReLU with 20 and 1 channels, respectively. 2. 2.

LSTM - A simplified LSTM layer with hidden size of 100 followed by a dense layer. 3. 3.

LSTM-CNN - An LSTM layer with hidden size 100 followed by two convolution layers with 8 channels and 3 kernels each, followed by dropout and mean pooling with ReLU. The model ends with a single linear layer, as presented in Fig.3. 4. 4.

SVM - Each sample $X\in\mathbb{R}^{20\cdot S\times 6}$ was flattened to a vector $\tilde{X}\in\mathbb{R}^{120\cdot S}$ and a classifier was trained on it. 5. 5.

Random Forest - Each sample $X\in\mathbb{R}^{20\cdot S\times 6}$ was flattened to a vector $\tilde{X}\in\mathbb{R}^{120\cdot S}$ and an ensemble random forest classifier was trained. The maximum depth allowed per tree was set to 100.

III-B DNNs loss function and training hyperparameters

The loss function and training criteria for the DNN-based models are described in this section.

III-B1 Loss function for DNN-based models

All of the DNNs considered in this work were trained using the binary cross entropy loss, the most common loss function used for binary classification problems. It is calculated using the following formula:

[TABLE]

where the output of the DNN is $p_{i}=p(X_{i},{\bf{\Theta})}\in[0,1]$, which depends on the sample $X_{i}$ and the weights $\bf{\Theta}$ and $Y_{i}\in\{0,1\}$ is the true label of example $i$. The entire loss is

[TABLE]

where $N$ is the number of samples in the train set.

III-B2 Training DNN-based models

The DNN-based models were all trained for 30 epochs using the ADAM optimizer [21] with a learning rate of $0.001$, $\beta$ coefficients of $\beta_{1}=0.9$, $\beta_{2}=0.999$, and batch size $6$. We trained the model using a Tesla K80 GPU. We trained for approximately 3.5 minutes on average in each combination of architecture and window size, 8 seconds per epoch.

IV Results

This section contains the results of each of the considered classifiers on the validation set. The models were evaluated for windows of $S=\{1,2,3\}$ seconds. The model which obtained the best results on the validation set was then tested on a separate drive containing both road and sidewalk.

IV-A Evaluation Metrics

Three evaluation metrics for the performance of the models were considered.

Accuracy: accuracy is the proportion of true results among the total number of cases examined. 2. 2.

F1 Score: the F1 score is a number between 0 and 1 and represents the harmonic mean of precision and recall. 3. 3.

False Positive Rate (FPR): FPR is the proportion of samples classified as 1 when the true label was 0.

Each of these metrics was computed for the validation set for all the ML models.

IV-B Best performing model

The optimal solution for the classification problem on the validation set is the LSTM-CNN model. The benefit of this model is its both spatially (IMU signals information) and temporally (time) properties. The architecture layers are presented in Fig. 3. The best results on the validation set were obtained for a window of $3[s]$. Table I presents performance metrics for each of the ML models considered in this work. Note that the model outperforms all other models where it achieves maximal accuracy of $0.93$, maximal F1 of $0.93$, and minimal FPR of $0.13$.

IV-C The performance of classical models

The two classical models considered were the SVM and Random Forest. In contrast to the LSTM and CNN-based models, the classical approaches do not treat the input signal as time series but as a flattened vector of features. Table I shows a surprising result that both classical models performed well and closely approached the metrics of the DNNs. This may suggest that the data is separable as is even without significant effort in handcrafting features. However, it is important to note that the IMU data was filtered, downsampled, rotated, and the gravity component was subtracted.

IV-D Field experiment

An experiment was conducted on a single drive containing both road and sidewalk segments. The route is shown in Fig. (4) and it was not used in the training or validation sets. The light green segment is a sidewalk, the light gray is an asphalt road, and the red points show false classification. The route begins on a sidewalk and at a certain point, switches to an asphalt road.

The trained LSTM-CNN model with a $3[s]$ window was chosen for this task as it performed best on the validation set. A recording of raw IMU measurement was passed through the pre-processing described in Section II-B and fed to the classifier. The model was activated every $5[s]$. The classifier has successfully classified $90\%$ of the route. In addition, a strong wind was present during the recording of this experiment, which was not the case for the training data. The wind has led to elevated levels of noise and vibration which were not present in the training or validation datasets. However, the results of this experiment agree well with the performance observed on the validation set.

V Conclusions

A data-driven approach to the road surface recognition problem for e-scooters was suggested. For e-scooters it is specifically important to distinguish between roads and sidewalks. A binary classifier was trained to perform this task. Several models both based on deep learning and classical approaches were tested. Experimental results showed that a DNN-based model produces good results. Future research directions include developing a multi-class classifier that distinguishes between several types of surfaces and developing a real-time system that incorporates data from several sensors using a mobile device to deduce the road surface.