Conditional Generative Adversarial Networks Based Inertial Signal Translation

TL;DR

This paper introduces a method using Conditional GANs to translate wrist-worn inertial signals into shoe-mounted sensor signals, facilitating gait analysis with existing wearable devices.

Contribution

It demonstrates the effectiveness of GAN-based translation models for inertial signal conversion, enabling gait analysis without specialized sensors.

Findings

Wasserstein GANs outperform traditional GANs in translation accuracy.

Convolutional U-Net architecture yields better results than autoencoders.

The approach enables accurate gait analysis using wrist-worn sensors.

Abstract

The paper presents an approach in which inertial signals measured with a wrist-worn sensor (e.g., a smartwatch) are translated into those that would be recorded using a shoe-mounted sensor, enabling the use of state-of-the-art gait analysis methods. In the study, the signals are translated using Conditional Generative Adversarial Networks (GANs). Two different GAN versions are used for experimental verification: traditional ones trained using binary cross-entropy loss and Wasserstein GANs (WGANs). For the generator, two architectures, a convolutional autoencoder, and a convolutional U-Net, are tested. The experiment results have shown that the proposed approach allows for an accurate translation, enabling the use of wrist sensor inertial signals for efficient, every-day gait analysis.

Tables

Table 1. TABLE I: Generator networks architectures

Autoencoder Generator		U-Net Generator
Layer	Output size	Layer	Output size
Input	(1, 6, 256)	Input	(1, 6, 256)
Conv. 1D	(1, 64, 256)	Conv. 1D	(1, 64, 256)
LeakyReLU	(1, 64, 256)	LeakyReLU: x1	(1, 64, 256)
MaxPool1D	(1, 64, 128)	MaxPool1D	(1, 64, 128)
Conv. 1D	(1, 128, 128)	Conv. 1D	(1, 128, 128)
LeakyReLU	(1, 128, 128)	LeakyReLU: x2	(1, 128, 128)
MaxPool1D	(1, 128, 64)	MaxPool1D	(1, 128, 64)
Conv. 1D	(1, 256, 64)	Conv. 1D	(1, 256, 64)
LeakyReLU	(1, 256, 64)	LeakyReLU	(1, 256, 64)
MaxPool1D	(1, 256, 32)	ConvTrans. 1D	(1, 128, 128)
ConvTrans.1D	(1, 256, 64)	LeakyReLU	(1, 128, 128)
LeakyReLU	(1, 256, 64)	Concat(x, x2)	(1, 256, 128)
ConvTrans. 1D	(1, 256, 128)	ConvTrans. 1D	(1, 64, 256)
LeakyReLU	(1, 256, 128)	LeakyReLU	(1, 64, 256)
ConvTrans.1Da	(1, N, 256)	Concat(x, x1)	(1, 128, 256)
Sigmoida	(1, N, 256)	Conv. 1D	(1, N, 256)
		Tanh	(1, N, 256)
Parameters no.	247 938	Parameters no.	272 898
a N is the number of translated signal channels (6 or 2)

Table 2. TABLE II: The Discriminator Network architecture and training parameters

Discriminator		Training parameters
Layer	Output size	Parameter	Value
Inputa	(1,6+N,256)	epochs	10,000
Conv. 1D	(1, 64, 254)	optimizer	Adam
LeakyReLU	(1, 64, 254)	learning rate GAN	$1e-4$
MaxPool1D	(1, 64, 127)	learning rate WGAN	$3e-5$
Conv. 1D	(1, 64, 125)	Gradient Penalty $\lambda$	15
LeakyReLU	(1, 64, 125)
MaxPool1D	(1, 64, 62)
Conv. 1D	(1, 64, 60)
LeakyReLU	(1, 64, 60)
MaxPool1D	(1, 64, 30)
Flatten	(1, 7680)
Linear	(1, 128)
LeakyRelu	(1, 128)
Dropout	(1, 128)
Linear	(1, 1)
Sigmoid	(1, 1)
Parameters no.	247 938
a N is the number of translated signal channels (6 or 2)

Table 3. TABLE III: Wrist to shoe-sensor signal translation errors

model	6-channel												2-channel
	RMSE						MAE						RMSE		MAE
	$a_x$	$a_y$	$a_z$	${\omega }_x$	${\omega }_y$	${\omega }_z$	$a_x$	$a_y$	$a_z$	${\omega }_x$	${\omega }_y$	${\omega }_z$	${\omega }_{\mathrm{tot}}$	${\omega }_y$	${\omega }_{\mathrm{tot}}$	${\omega }_y$
CNN AE GAN	0.133	0.080	0.102	0.022	0.030	0.025	0.069	0.047	0.055	0.015	0.021	0.018	0.099	0.023	0.062	0.015
CNN AE WGAN	0.123	0.078	0.099	0.024	0.029	0.018	0.065	0.045	0.053	0.016	0.020	0.012	0.182	0.090	0.129	0.071
U-Net GAN	0.123	0.082	0.095	0.020	0.027	0.019	0.062	0.045	0.050	0.013	0.018	0.012	0.096	0.024	0.061	0.015
U-Net WGAN	0.142	0.087	0.114	0.049	0.047	0.032	0.078	0.051	0.067	0.039	0.032	0.020	0.116	0.040	0.075	0.029
aThe error values are presented for the output 0–1 range. Acceleration and angular velocity were measured in $\pm 4g$ and $\pm 2000dps$ ranges.

Equations

$$\begin{gathered} L_{\mathrm{adv}}={\mathbb{E}}_{y\sim p_{\mathrm{data}}}\log \left[D(x,y)\right] \\ +{\mathbb{E}}_{y\sim \mathrm{model}}\log \left[1-D(x,G(x))\right] \end{gathered}$$

$$L_{\mathrm{G}}=-\frac{1}{N}\sum _{i}D(x_i,G(x_i))$$

$$L_{\mathrm{D}}=\frac{1}{N}\sum _{i}D(x_i,G(x_i))-\frac{1}{N}\sum _{i}D(x_i,y_i)+\lambda L_{\mathrm{GP}}$$

$$L_{\mathrm{GP}}=\frac{1}{N}\sum _i((\Vert {\nabla }_{{\breve{y}}_i}D(x_i,{\breve{y}}_i){\Vert }_2-1{)}^2)$$

$${\breve{y}}_i=\alpha y_i+(1-\alpha )G(x_i)\text{where}\alpha \sim \mathcal{U}[0,1]$$

Full text

I Introduction

In clinical settings, gait analysis is performed using laboratory-based motion capture systems. Unfortunately, despite their high accuracy, they are unsuitable for everyday use due to their high cost and stationary nature. A possible solution for everyday gait monitoring is using sensors equipped with inertial measurement units (IMUs). Those sensors can be placed in several areas of the human body, with lower back, feet, and shins being the most popular among the studies [1]. Unfortunately, wearing the sensors in those places may be uncomfortable and impractical. For example, shoe sensors may require sensor frequent reattachment when changing shoes or battery charging.

A more practical solution would be to use popular wrist wearables, e.g., smartwatches or smart bands. However, the validation studies reported that the methods used for wrist-based gait analysis are less accurate than those employed for sensors located in the lower body area [1]. The solution to that problem might be to translate the wrist-worn IMU measurement results to those that would be collected if the sensor was worn in the lower body area, e.g., strapped to a shoe. The obtained signals might then be processed with well-established lower body-area methods.

Translating the signals between the upper and lower limbs is a challenging task. Even simplified movement models take into account detailed, not easily measurable data such as limb segments’ mass and dimensions [2]. Ither options involve specialized software [3], which can not be run on wearables. Therefore, most of modern signal translation solutions rely on machine learning.

There are several works concerning time-series translation, but only a few directly deal with inertial signals. In [4], the autoencoders are used to predict the output of a paired sensor worn in another body area to generate keys protecting the body area network from eavesdropping. The autoencoders are also utilized in [5] to tackle person recognition problems originating from inconsistencies between signals registered using sensors worn differently.

Inertial signal translation for motion analysis was presented in [6], where smartphone-based measurement results were translated to ones collected in the lower-body area using neural networks. The models used for that purpose included feed-forward dense neural networks and convolution-Long Short-Term Memory (CNN-LSTM) architectures. Although the translation was very accurate, the proposed models were large (more than 12 million parameters), and it might be problematic to implement them on constrained devices.

Given the rising popularity of generative methods, recent works employ Generative Adversarial Networks (GANs) for signal translation and generation. In [7], GANs are used to transform noisy inertial signals recorded with a cheaper sensor into those that would be obtained using a more expensive one. Another example of GAN-based inertial signal generation is presented in [8], where GANs are used to generate inertial signals corresponding to selected physical activities.

The work presented in the following paper is the continuation of research described in [9], where the inertial signals were translated between the ones obtained with wrist-worn devices and devices strapped to shins and shoes. The previously tested neural network architectures included a dense neural network, convolutional autonecoder (CNN AE), CNN-based U-Net, and an LSTM network trained in a supervised setting. The main contributions of the paper are the following:

• •

The Conditional Generative Adversarial Networks architectures are tested in an inertial signal translation scenario.

• •

In addition to translating all signals, a different scenario, in which only two required components (mediolateral component of angular speed and its total value), is tested.

The rest of the paper is organized as follows. Section II describes the inertial signal translation scenario. The employed ML models and experiment results are presented in Sections III and IV, respectively. Section V concludes the paper.

II Gait analysis using inertial signals

In a typical scenario, a gait sensor is equipped with an IMU containing a 3-axis accelerometer and 3-axis gyroscope, resulting in six signal components that can be used for gait analysis. Exemplary signals recorded using such sensors are presented in Fig. 1.

There is a significant difference between the signals recorded using wrist sensors and sensors placed in the lower body areas. Due to large forces during heel strikes, the acceleration and angular velocity changes are much more abrupt. It enables more precise timing measurements than in the case of wrist signals. In the proposed concept, the signals registered using a wrist-worn sensor are translated to signals that would be obtained if the sensor was worn in the lower-body area. It enables the use established gait analysis methods without specialized shoe/shin gait sensors.

In the paper, two approaches are tested. In the first one, all of the signal components are translated, yielding a complete set of IMU measurement results. The second approach assumes generating only two signals: the angular velocity in the mediolateral axis (ankle rotation axis, where, according to gait studies [10], the most prominent movements happen - $\omega_{y}$ in Fig. 1) and the total value of the angular velocity $\omega_{\mathrm{tot}}=\sqrt{\omega_{x}^{2}+\omega_{y}^{2}+\omega_{z}^{2}}$. In most cases, those two components are sufficient for detecting heel strike, flat foot, and toe-off phases of the gait cycle [10].

III Conditional GAN-based signal translation

III-A Training set-up

The signal translation is performed using conditional GAN networks. A typical GAN network consists of two separate models, the Generator and the Discriminator, trained in an adversarial manner as presented in Fig. 2.

The architecture proposed for the IMU signal translation task follows the guidelines outlined in the seminal conditional GAN paper [11]. The Generator network is responsible for generating (translating) new data based on supplied labels (in this case, a set of six inertial signals). The Discriminator network then evaluates both real and translated signals, judging whether they they are real or fake. To evaluate translation quality, the Discriminator also takes in the original set of signals. The classification errors are backpropagated to update the weights of both the Discriminator and the Generator.

In classical setup, the networks are trained in an adversarial manner, trying to minimize (with respect to the Discriminator) and maximize (with respect to the Generator) the adversarial loss function $L_{\mathrm{adv}}$:

[TABLE]

where $\mathbb{E}_{y\sim p_{\mathrm{data}}}$ and $\mathbb{E}_{y\sim\mathrm{model}}$ are expected values with respect to the real and generated samples respectively; $G(x)$, $D(x,y)$ are Generator and Discriminator outputs; $x$ are inertial signals registered using the wrist sensor and $y$ is a corresponding label (shin/shoe signals).

The paper also tests another approach - Wasserstein GANs with gradient penalty (GP-WGANs). In such a setup, the networks have different loss functions $L_{\mathrm{G}}$ and $L_{\mathrm{D}}$:

[TABLE]

The Generator loss ($L_{\mathrm{G}}$) depends on how well the discriminator performs when classifying generated signals $G(x_{i})$. The loss for the Discriminator ($L_{\mathrm{D}}$) includes an additional gradient penalty computed based on its performance on interpolated signals $\breve{y}_{i}$ being a randomly weighted sum of the real and generated examples. The influence of the gradient penalty can be controlled using the $\lambda$ parameter.

III-B Proposed generator and discriminator architectures

Gait analysis requires a relatively high sampling rate (at least 50 Hz but typically 100 Hz or more) resulting in large volumes of data. As transferring large volumes of data is impractical, in real-use scenarios, it would be preferable to translate the signals and perform the analysis locally on the wrist sensor. Therefore, the translation models should have a limited complexity to optimize the processing time and energy efficiency. As most available AI-enabled chips are optimized for CNNs, the tested generator architectures are CNN-based without using the recurrent layers typically employed for temporal data processing. The details of the generator and discriminator architectures are presented in Tables I and II.

For the generator, two architectures were tested: Convolutional Autoencoder (CNN AE) and U-Net. The generator models, as an input, accept six channel time series comprising acceleration and angular velocity measured with the wrist sensor. The input signals are scaled into the 0–1 range. The input sequence length is 256 samples corresponding to 5.12 s (the sampling rate was set to 50 Hz). The network contains solely convolutional layers followed with Leaky ReLU activation (negative slope parameter equal to 0.2) and MaxPooling. Depending on the translation scenario, the generators output six or two channels. The size of the models is moderate and does not exceed 300,000 parameters.

The Discriminator is a convolutional network following the pattern similar to the generators. To obtain better generalization, a 20 percent dropout is used.

IV Experimental verification

The proposed setup was experimentally tested based on data gathered with custom sensors developed at Warsaw University of Technology (WUT). The devices were equipped with Bosch Sensortec BMI270 IMUs allowing for 3-axial acceleration (in $\pm 4\mathrm{g}$ range) and angular velocity (in $\pm 2000\mathrm{dps}$ range) measurements. The synchronization between the signals was attained by triggering the data collection start over the BLE-radio link (the devices were controlled with BLE-enabled nRF52833 Nordic Semiconductor microcontrollers).

The data used in the study is publicly available at [12]. The dataset contains 856 examples comprised of wrist-shoe inertial signal sequence pairs of 5.12 s length (50 Hz sampling rate). Both input and output signals were scaled into the 0–1 range. The data were divided into training and validation datasets following a 0.9–0.1 split. The proposed network architectures were then trained using the the parameters listed in Table II.

The translation accuracy was evaluated based on the root mean squared error (RMSE) and mean absolute error (MAE). The obtained values are listed in Table III. The selected translated signals are presented in Fig. 3.

For the 6-channel translation, the best results were achieved using the CNN AE WGAN network closely followed by the U-Net GAN. The translation errors were larger for the acceleration signals due to higher noise present in such measurements. In the case of the 2-channel translation, the mediolateral angular velocity component ($\omega_{y}$) reconstruction accuracy was slightly better than when translating all of the signals.

The visual analysis of the translated signals shows that, as in the earlier study [12], the networks struggle with reconstruction of sharp signal peaks (especially noticeable for the acceleration signals). However, the GAN-based reconstruction of crucial signal areas (like valleys in $\omega_{y}$ indicating toe-offs and heel-strikes) is much better which would lead to more accurate gait timing analysis.

V Conclusions

In the paper, conditional GAN networks were tested in the scenario of inertial signal translation between two sensor locations. The tested networks comprised convolutional networks of moderate complexities. The experiment results have shown that the translation accuracy is sufficient for the signals to be used in gait analysis scenarios. Using the GAN training setup allows the translating network to better grasp the shoe sensor signal characteristics which leads to better results than in the case of standard autoencoder networks [12].

The translation errors might be reduced by enlarging the dataset or using semi-supervised methods. Another possible development direction is to introduce loss functions taking into account the timing between the moments corresponding to typical gait events in both real and translated signals.