Privacy-Enhancing Fall Detection from Remote Sensor Data Using Multi-Party Computation

TL;DR

This paper presents a privacy-preserving fall detection system using multi-party computation on IMU data, achieving real-time performance and state-of-the-art accuracy in a cloud-based setting.

Contribution

It introduces a novel application of multi-party computation for privacy-preserving fall detection from IMU sensors, demonstrating practicality and high accuracy.

Findings

MPC-based fall detection is practical with real-time performance.

The system achieves state-of-the-art error rates.

Execution time is within the required 750ms window.

Abstract

Motion-based fall detection systems are concerned with detecting falls from vulnerable users, which is typically performed by classifying measurements from a body-worn inertial measurement unit (IMU) using machine learning. Such systems, however, necessitate the collection of high-resolution measurements that may violate users' privacy, such as revealing their gait, activities of daily living (ADLs), and relative position using dead reckoning. In this paper, we investigate the application of multi-party computation (MPC) to IMU-based fall detection for protecting device measurement confidentiality. Our system is evaluated in a cloud-based setting that precludes parties from learning the underlying data using multiple, disparate cloud instances deployed in three geographical configurations. Using a publicly-available dataset, we demonstrate that MPC-based fall detection from IMU measurements is practical while achieving state-of-the-art error rates. In the best case, our system executes in 365.2 milliseconds, which falls well within the required time window for on-device data acquisition (750ms).

Tables

Table 1. Table 1. Comparison of BWD-based fall detection systems.

Proposals	Algorithms	Error Rates
SmartFall (Mauldin et al., 2018)	NB, SVM, RNN	0.37–0.79 (P), 0.55–1.0 (R), 0.65–0.99 (A)
Liu & Cheng (Liu and Cheng, 2012)	SVM	0.9927 (P), 0.9740 (R), 0.9838 (A)
S-R et al. (Santoyo-Ramón et al., 2018)	SVM, kNN, NB, DT	0.614–0.999 (GMPR)
Doukas et al. (Doukas et al., 2007)	SVM	0.982 (A)
E-FallD (Cao et al., 2012)	Thresholds	GT: 0.8675 (P), 0.855 (R) CT: 0.9275 (P), 0.8675 (R)
Vilarinho et al. (Vilarinho et al., 2015)	Thresholds	0.78 (P), 0.63 (R), 0.68 (A)
Yavuz et al. (Yavuz et al., 2010)	Thresholds	0.88–0.90 (P), 0.46–0.95 (R)
Albert et al. (Albert et al., 2012)	SVM, LR, NB, DT, kNN	0.982 (A)
Musci et al. (Musci et al., 2018)	RNN, LSTM	0.9286–0.9716 (A)
Luštrek & Kaluža (Luštrek and Kaluža, 2009)	SVM, DT, kNN, NB, RF, Adaboost, Bagging	0.732–0.963 (A)

Table 2. Table 2. Communication cost for MPC functions using arithmetic secret sharing  (Chen, 2012 ) .

Function	Communication Cost (in rounds)
Multiplication	1
Inner Product	1
Inverse	1
Random element	1
Random bit	3
Equality ($=$)	2
$\ge$	$2m+4$
$\le$	$2m+4$
AND, OR	$2\log \left(k\right)$

Table 3. Table 3. MPC server location configurations using AWS.

Config	Party 1	Party 2	Party 3
A	US_Oregon	Canada_Central	US_Ohio
B	US_Oregon	Canada_Central	Europe_London
C	US_Oregon	Europe_London	Asia_Singapore

Table 4. Table 4. Classification error with SmartFall features.

Algorithm	Accuracy	Precision	Recall	F1-Score
LR	0.933	0.780	0.688	0.731
NB	0.926	0.704	0.762	0.732
SVM	0.936	0.819	0.663	0.733

Table 5. Table 5. Classification error with derivative-based features.

Algorithm	Accuracy	Precision	Recall	F1-Score
LR	0.936	0.771	0.812	0.791
NB	0.934	0.725	0.903	0.804
SVM	0.937	0.762	0.835	0.797

Table 6. Table 6. Mean feature extraction and inferencing times (milliseconds) using SmartFall features for each configuration; S.D. shown in brackets.

	Config A		Config B		Config C
Alg.	FE	IN	FE	IN	FE	IN
LR	—	308 (91)	—	383 (104)	—	507 (150)
SVM	779 (119)†	—	1005 (125)	—	1204 (194)	—
NB	—	509 (103)	—	557 (226)	—	692 (114)

Table 7. Table 7. Mean feature extraction and inferencing times (milliseconds) using derivative-based features.

	Config A		Config B		Config C
Alg.	FE	IN	FE	IN	FE	IN
LR	—	297 (106)	—	365 (95)	—	490 (104)
SVM	68 (15)†	—	102 (19)	—	111 (24)	—
NB	—	552 (103)	—	602 (106)	—	742 (113)

Equations

$$|\vec{v}|=\sqrt{x^2+y^2+z^2}$$

$$\Delta s=\max (|\overrightarrow{v_i}|)-\min (|\overrightarrow{v_j}|),i\ne j$$

$$\text{D}=\left(\begin{array}{ccccc}[A_x{]}_0& [A_x{]}_1& [A_x{]}_2& \cdots & [A_x{]}_n\\ [A_y{]}_0& [A_y{]}_1& [A_y{]}_2& \cdots & [A_y{]}_n\\ [A_z{]}_0& [A_z{]}_1& [A_z{]}_2& \cdots & [A_z{]}_n\end{array}\right)$$

$$f=\sum\frac{dx}{dt},\sum\frac{dy}{dt},\sum\frac{dz}{dt},\sum\big{(}\frac{dx}{dt}\big{)}^{2},\sum\big{(}\frac{dy}{dt}\big{)}^{2},\sum\big{(}\frac{dz}{dt}\big{)}^{2}$$

$$\frac{du}{dt}=A_u\ast h$$

$$C=\{\begin{array}{ll}1,& \text{if}\vec{\theta }\cdot \vec{x}+b\ge 0\\ 0,& \text{otherwise}\end{array}$$

$$g_i(\vec{x})=\ln (P(C_i))-\sum _{j=1}^n\frac{||x_j-{\mu }_{ij}|{|}^2}{2{\sigma }_{ij}^2}$$

Full text

Fall detection, multi-party computation (MPC), Internet of Things (IoT), cloud computing

††journalyear: 2019††copyright: acmlicensed††conference: Proceedings of the 14th International Conference on Availability, Reliability and Security (ARES 2019); August 26–29, 2019; Canterbury, United Kingdom††booktitle: Proceedings of the 14th International Conference on Availability, Reliability and Security (ARES ’19), August 26–29, 2019, Canterbury, United Kingdom††price: 15.00††doi: 10.1145/3339252.3340500††isbn: 978-1-4503-7164-3/19/08††ccs: Security and privacy Security services††ccs: Security and privacy Distributed systems security††ccs: Security and privacy Domain-specific security and privacy architectures††ccs: Computer systems organization Cloud computing

1. Introduction

Falls are the leading cause of fatal injury and the most common cause of nonfatal, trauma-related hospital admissions among older adults, with 20–30% of mild-to-severe injuries and 40% of injury-related fatalities (Musci et al., 2018). The annual medical costs attributable to fatal and nonfatal falls was estimated at $50bn (USD) in 2015 (Florence et al., 2018). Fall detection systems are concerned with automatically detecting the occurrence of falls within the homes of elderly and disabled users, with the aim of minimising response times to potential injuries and facilitating independent living (Mubashir et al., 2013). These systems involve the collection of data, such as accelerometer data from a body-worn device (Mauldin et al., 2018; Liu and Cheng, 2012; Santoyo-Ramón et al., 2018; Doukas et al., 2007; Yavuz et al., 2010; Albert et al., 2012; Luštrek and Kaluža, 2009), video imagery from cameras in the home (Zhang et al., 2012; Mastorakis and Makris, 2014; Kwolek and Kepski, 2014; Edgcomb and Vahid, 2012), and proximity sensors in ceilings (Tao et al., 2012). This data is typically classified with respect to a dataset of known fall and non-fall data using machine learning.

The deployment challenges surrounding fall detection systems have, to date, remained largely out-of-scope in current work, which implicitly assume a local deployment or focus solely on classification performance using off-line analyses. Recent work has touted the benefits of a cloud-based, software-as-a-service (SaaS) approach for mobility-based sensing systems, where applications and their underlying platform can be updated, upgraded, monitored and secured with greater flexibility (Doukas and Maglogiannis, 2011; Fortino et al., 2014; Gravina et al., 2017). This is important in managing large numbers of remote devices, where patch penetration remains a significant challenge (Palani et al., 2016). Indeed, failing to update devices could lead to potentially unsafe misclassification errors due to accuracy discrepancies between implementations.

This principle of algorithm agility provides greater flexibility in changing the underlying classification algorithm in a cloud-based scenario without potential penetration issues of patching and updating large numbers of devices in the field. Another benefit lies in the separation of the platform development, e.g. device hardware, firmware and software, from the algorithm implementation itself. In this paradigm, individual device OEMs111OEM: Original Equipment Manufacturer. are relinquished from the responsibility of implementing and maintaining the detection algorithm. Rather, this can be performed by a dedicated third-party with the potential of supporting devices from multiple OEMs in a service-oriented architecture, which we focus upon in this work.

A myriad of security and privacy concerns remain, however, regarding large volumes of data being harvested, used or, indeed, misused by SaaS providers (Zhang et al., 2014). At worst, sensor measurements collected from users’ devices could be exploited to infer their activity patterns and whereabouts, with major privacy implications. This is compounded by the risks of disclosing plaintext measurements, say, after the exploitation of a vulnerable service, such as insecure AWS S3 buckets (Continella et al., 2018). A large corpus of work has already demonstrated identifying users based on their gait from accelerometer and other inertial measurement unit (IMU) data (Gafurov et al., 2007; Thang et al., 2012; Mantyjarvi et al., 2005; Juefei-Xu et al., 2012; Gafurov, 2007); their position using dead reckoning (Pratama et al., 2012; Kang and Han, 2015; Wang et al., 2018); and determining activities of daily living (ADLs), like whether the user is walking, sleeping, and sitting (Stikic et al., 2008; Hong et al., 2010). Existing work on privacy-enhancing fall detection has focussed on non-cryptographic techniques for video-based methods, including image blurring, silhouetting and foreground extraction. However, these still reveal significant information about users, such as their presence. Moreover, they do not generalise to the significant proportion of fall detection proposals using time-series IMU measurements from personal devices, such as smartphones and smartwatches (Mauldin et al., 2018; Santoyo-Ramón et al., 2018; Cao et al., 2012; Vilarinho et al., 2015; Albert et al., 2012).

A promising solution is multi-party computation (MPC), where measurements are split using a linear secret sharing scheme (LSSS) and distributed to mutually distrusting parties. After this, functions can be jointly computed without parties disclosing their plaintext inputs to others. Existing literature work has already shown that machine learning inferencing, upon which many fall detection systems rely, can be performed efficiently using MPC. This includes MPC-based image recognition using support vector machines (SVMs) (Makri et al., 2017) and remote rehabilitation treatment classification from patient data using decision trees (de Hoogh et al., 2014).

In this work, we present the implementation and evaluation of the first system for perform MPC-based fall detection from IMU device data in order to protect measurement confidentiality. MPC is used to perform the requisite feature extraction and classification without the parties learning the contents of IMU measurements received from the device. We evaluate the system, illustrated in Figure 1, using an off-the-shelf device and three MPC parties deployed using a public cloud service. In short, this work presents the following contributions:

• •

The first framework for performing privacy-enhancing fall detection from device IMU measurements using MPC. The proposal preserves measurement confidentiality in a cloud-based model while performing feature extraction and classification using machine learning.

• •

A two-part evaluation with experiments conducted using a publicly-available fall detection dataset. Our inferencing framework executes faster than the time necessary for on-device measurement acquisition, and achieves error rates commensurate with state-of-the-art schemes.

1.1. Document Structure

This paper proceeds with a general review of fall detection systems in Section 2, which is followed by a discussion of the privacy challenges associated with existing systems. In Section 3, we present the assumptions and threat model in greater detail, before introducing and motivating the use of MPC as a solution to the aforementioned challenges. This section also develops the design of the proposed framework after which, in Section 4, the implementation details are discussed. Section 5 presents an evaluation of the proposed framework, comprising a discussion of the methodology and experimental results in the form of classification and computational performance. Lastly, we conclude our work in Section 6, including a discussion of future research directions.

2. Fall Detection

Fall detection systems fall broadly into two categories: 1), those using devices situated in the user’s ambient environment; and, 2), those using body-worn devices (BWDs), e.g. smartwatches. We now describe prominent existing schemes and their operation. This section is not an exhaustive analysis—for this, the reader is referred to Mubashir et al. (Mubashir et al., 2013) and Delahoz et al. (Delahoz and Labrador, 2014)—but, rather, we aim to summarise the salient approaches and results.

2.1. Ambient Device-based Systems

Privacy-enhancing fall detection has focused almost entirely on non-cryptographic techniques from ambient-based devices, such as silhouetting, blurring and foreground masking from camera data. Mastorakis and Makris (Mastorakis and Makris, 2014) and Zhang et al. (Zhang et al., 2012) employ RGBD data from a Microsoft Kinect to detect five ADLs, using only the depth values and foreground mask of the RGB data, thus occluding users’ facial features. Tao et al. (Tao et al., 2012) study the use of a ceiling-based sensor network comprising infrared cameras for fall detection in a home environment. The sensors output binary values to indicate the existence of persons underneath, which are used to model a map of the user’s home. Edgcomb and Vahid (Edgcomb and Vahid, 2012) explore four techniques for privacy-enhancing video-based fall detection: blurring (of the user), silhouetting, and covering the user with a graphical box and oval. Raw video imagery is first taken from a stationary in-home camera, before applying one of these techniques.

2.2. BWD-based Systems

The use of ambient-based devices has inherent issues associated with detecting falls in occluded areas: fitting multiple cameras to cover all possible locations in the user’s living environment—bathroom, bedroom, kitchen, and so on—may impose significant costs relating to unit installation and maintenance. Another paradigm employs devices likely to be already owned and carried by the user, such as a smartwatch or smartphone. Unlike the previous section, we are unaware of proposals that investigate privacy-enhancing fall detection from BWDs to the best of our knowledge. As such, we provide a general review of highly-cited and state-of-the-art work from the literature.

Mauldin et al. (Mauldin et al., 2018) present SmartFall, which uses accelerometer data sampled at 31.25Hz from a user-worn smartwatch to detect user falls. Smartwatch data is streamed to the user’s smartphone that extracts features pertaining to the vector magnitude of the accelerometer data and the difference between the maximum and minimum magnitudes over a 750ms sliding window (Equations 1 and 2). Naïve Bayes, Support Vector Machine (SVM) and Recurrent Neural Network (RNN) classifiers are evaluated using labelled data from seven volunteers between 21–55 years old, alongside the Farseeing dataset (Mellone et al., 2012) with data from 2,000 participants. Results of 0.37–0.79 precision, 0.55–1.0 recall, and 0.65–0.99 accuracy are reported, depending on the chosen dataset and classifier.

[TABLE]

[TABLE]

Liu and Cheng (Liu and Cheng, 2012) investigate IMUs placed at users’ waists that collects accelerometer measurements at 200Hz. The vector magnitude is computed, along with the fast changed vector, vertical acceleration and posture angle between the vertical acceleration and gravity over a 0.1s sliding window. The features are inputted to an SVM with labelled fall data from 15 volunteers who performed 10 simulated falls and 11 ADLs repeated 10 times. The proposal yields error rates of 98.38% accuracy, 97.40% recall, and 99.27% precision.

Santoyo-Ramón et al. (Santoyo-Ramón et al., 2018) investigate IMUs attached to the user’s chest, waist, wrist and thigh—the data from which is aggregated into 15 second segments by a smartphone. A sliding window of 0.5s is used in which the fast changed vector of accelerometer measurements is computed alongside the magnitudes’ mean, standard deviation and mean absolute distance, and the mean rotation angle and mean module of the $(Y,Z)$ and $(X,Z)$ components. The features are inputted to SVM, k-Nearest Neighbour, Naïve Bayes and Decision Tree classifiers, and evaluated using data from 19 subjects who performed 746 ADLs and falls. The geometric mean of the precision and recall is used as the performance metric, with results of 0.614–0.999 depending on the algorithm and sensor combination.

Doukas et al. (Doukas et al., 2007) investigate the use of an accelerometer attached to the user’s foot for fall detection. The sensor data is transmitted to a receiving device, such as a laptop, where it is normalised and inputted to an SVM classifier in raw form after applying a Kalman filter. The evaluation reports a classification accuracy of 98.2% using a limited set of labelled data from two volunteers.

Cao et al. (Cao et al., 2012) present E-FallD, which uses accelerometer data from an Android smartphone. The vector magnitude of the accelerometer components is computed and a global threshold is fixed for identifying falls based on the phone’s measured acceleration. The work also investigates multiple thresholds customised to the user’s gender, age, and body mass index (BMI). The authors recruit 20 participants of varying gender, age and BMI, who perform a total of 400 falls and 1200 ADLs. The approach yields results of 86.75% precision and 85.5% recall using global thresholds, and 92.75% precision and 86.75% recall with customised thresholds.

Similarly, Vilarinho et al. (Vilarinho et al., 2015) use accelerometer data from an Android smartphone and smartwatch. Here, a rule-based system is devised for threshold-based detection using the vector magnitude, fall index, and absolute vertical acceleration as features over a 0.8s sliding window. The approach is evaluated using data from three participants, who performed 12 falls and 7 ADLs each, with a reported 0.68 classification accuracy, 0.78 precision, and 0.63 recall.

Yavuz et al. (Yavuz et al., 2010) propose another system based on accelerometer values from an Android smartphone. In contrast, the authors use solely frequency domain features, namely the wavelet coefficients after applying a discrete wavelet transform. A threshold-based approach is used to test the coefficients generated from the phone against those of previously observed falls. 100 fall sequences from five volunteers were collected to evaluate the system, with results of 46-95% precision and 88-90% recall.

Albert et al. (Albert et al., 2012) use accelerometer values from an LG G1 Android smartphone located at the user’s pelvic region. 178 features across several categories are subsequently extracted: moments, e.g. kurtosis and skew; moments between successive samples; smoothed root mean squares; extremities, e.g. min and max; histogram values; Fourier components; mean acceleration magnitude; and cross products. The features are inputted to five classifiers—SVM, Logistic Regression, Naïve Bayes, Decision Tree and kNN—and evaluated using data from 18 simulated falls from 15 subjects, resulting in 98% classification accuracy.

Musci et al. (Musci et al., 2018) evaluate the use of recurrent neural networks (RNNs) and long short-term memory units (LSTMs), and propose a deep learning architecture for ternary fall detection for whether or not a fall has occurred, and whether one is about to occur. The proposal is evaluated using the SisFall dataset from (Sucerquia et al., 2018), comprising accelerometer and gyroscope measurements from a smartphone and a custom IMU attached to the waists of 38 users; the dataset is labelled with respect to the three aforementioned states. A confusion matrix of the three classes reports 92.86–97.16% classification accuracy following hyperparameter optimisation.

Luštrek and Kaluža (Luštrek and Kaluža, 2009) explore the use of a body area network (BAN) comprising 12 IMUs at the user’s shoulders, wrists, hips, knees and ankles. Features are collected regarding the coordinates of units in a body coordinate system, their velocities, absolute distance, and relative angles. The features are used as input to seven machine learning algorithms—SVM, DT, kNN, NB, Random Forest, Adaboost, and bagging—implemented in the Weka library. Three participants were recruited who performed a total of 45 activities: 15 falls and 10 lying down, sitting down and walking ADLs each. 73.4–96.3% classification accuracy is reported depending on the algorithm used, with SVM yielding the best case.

2.3. Discussion

Generally, fall detection systems rely upon traditional supervised learning algorithms and, more recently, deep neural networks (LSTMs and RNNs), trained over labelled fall data collected a priori. The majority of existing work models binary classification, i.e. ‘fall’ or ‘not fall’, while other work also models whether a fall may be about to occur and classification of general ADLs, such as sitting, walking, and lying down.

Additionally, the generation of global models that generalise to all users, rather than individuals, is also a common facet of current work. For BWD-based systems, accelerometer measurements are used ubiquitously from which a variety of features are extracted prior to classification, which is illustrated in Figure 2 for the binary case. Most work focuses on time-domain features, such as the vector magnitude, arithmetic mean, standard deviation, and maximum and minimum values. The choice of algorithm remains varied, although SVMs tend to exhibit the best performance with 0.9838 (Liu and Cheng, 2012), 0.982 (Doukas et al., 2007) and 0.73–0.99 (Mauldin et al., 2018) accuracy. Recent work (Mauldin et al., 2018; Musci et al., 2018) has begun to explore the application of deep learning using LSTMs and RNNs with promising results—0.9286–0.9716 (Musci et al., 2018) and 0.85–0.99 (Mauldin et al., 2018) accuracy—but work in the area remains limited.

Rather than using machine learning for the discovery of optimal decision boundaries, another widely-used approach has been the use of manually set threshold values, which are discovered experimentally. Proponents of this approach have touted faster classification as an advantage over machine learning-based systems, and thus more suitable for constrained devices; however, such approaches generally yield higher error rates and are evaluated over limited datasets, e.g. with five (Yavuz et al., 2010) and three (Vilarinho et al., 2015) participants. We present a summary of BWD-based systems in Table 1.

3. Privacy-Enhancing BWD-Based Fall Detection

To the best of our knowledge, privacy-enhancing BWD-based fall detection is yet to be investigated. Accelerometers, used in the bulk of current proposals, measure fine-grained movements at high frequency rates (typically $>$30Hz) on commodity devices, such as smartphones and smartwatches. A wealth of literature exists surrounding the identification of users based on their gait using accelerometer measurements alone from such devices (Gafurov et al., 2007; Thang et al., 2012; Mantyjarvi et al., 2005; Juefei-Xu et al., 2012; Gafurov, 2007). Furthermore, we are also aware of IMU measurements being used for recognising non-fall ADLs (Stikic et al., 2008; Hong et al., 2010) and performing user position tracking using dead reckoning (Pratama et al., 2012; Kang and Han, 2015; Wang et al., 2018), which could be exploited to violate user privacy.

3.1. Applying Multi-Party Computation (MPC)

We observe that MPC can be applied to acquire data, extract features, and perform fall classification in a way that preserves IMU measurement confidentiality. In this work, we tackle the base case of MPC using arithmetic secret sharing from Shamir’s secret sharing scheme (SSSS) (Shamir, 1979), which is briefly described as follows.

3.1.1. Shamir’s Secret Sharing Scheme

SSSS divides a secret $\mathcal{S}$ into $n$ shares, $\textbf{s}=\{s_{1},s_{2},\dots,s_{n}\}$, such that knowing $t$ shares from s makes $\mathcal{S}$ easy to compute, but knowing fewer shares than $t$ renders $\mathcal{S}$ undetermined. The scheme relies on the fact that $t$ points are required to construct a polynomial of degree $t-1$, which is defined in the field $\mathbb{F}_{P}$. Next, $t-1$ positive integers, $a_{1},\dots,a_{t-1}$ are chosen with $a_{i}<P$, with the secret assigned as $a_{0}=\mathcal{S}$. The polynomial $f(x)=a_{0}+a_{1}x+a_{2}x^{2}+a_{3}x^{3}+\cdots+a_{t-1}x^{t-1}$ is built with which $n$ points are constructed from $i=1,\dots,n$, to retrieve pairs $(i,f(i))$. Any subset with $t$ distinct pairs can be used to recover the constant term, $a_{0}$, i.e. the secret, which is found using Lagrange interpolation.

3.1.2. MPC from Arithmetic Secret Sharing

An important observation is that SSSS allows each party to locally compute linear combinations of secrets and public values. Firstly, let us define $[x]_{i}$ as the $i^{th}$ share of data item, $x$. SSSS allows parties to locally compute the following arithmetic operations over their shares (Chen, 2012):

• •

Addition ($[c]\leftarrow[a]+[b]$): Each party $p_{i}$ can compute a new share, $[c]_{i}$, that is the addition of two other shares, i.e. $[c]_{i}=[a]_{i}+[b]_{i}$.

• •

Addition of a secret and public value ($[c]\leftarrow[a]+\alpha,\alpha\in\mathbb{F}_{P}$): Party $p_{i}$ can compute a new share from the addition of a held share and a public value $\alpha\in\mathbb{F}_{P}$; that is, $[c]_{i}=[a]_{i}+\alpha$.

• •

Multiplication of a secret and public value ($[c]\leftarrow[a]\cdot\alpha,\alpha\in\mathbb{F}_{P}$): $p_{i}$ can compute a new share of the multiplication of a held share with a public value, $[c]_{i}=[a]_{i}\cdot\alpha$.

Computing a new share that is the multiplication of two others, $[c]=[a]\cdot[b]$, is less straightforward: given that SSSS uses polynomials of degree $t$, then $[a\cdot b]\neq[a]_{i}\cdot[b]_{i}$, as the resulting polynomial has degree $2t$. A method for computing $t$-degree polynomials representing $[c]\leftarrow[a]\cdot[b]$ was presented in (Ben-Or et al., 1988). As opposed to entirely local computation for addition of shares and the addition and multiplication of a share with a constant value, multiplying shares necessitates one round of communication. While many functions can be computed using only these operations, they preclude integer comparison, e.g. $=$, $\neq$, $<$, and $>$, and Boolean operations, e.g. AND and OR. The reader is referred to (Chen, 2012) for a comprehensive description and analysis of these protocols; the differences in asymptotic communication complexity are reproduced from (Chen, 2012) in Table 2.

3.2. Assumptions and Threat Model

An outsourcing model is assumed whereby a body-worn device streams IMU measurements to a cloud-based service that performs the feature extraction and classification. The focus is on binary classification for inferring the occurrence of falls using a globally-trained model in line with the bulk of existing work from Section 2. We divide the entities involved into two categories.

3.2.1. Source device

We assume the existence of a body-worn device—a smartphone in the user’s pocket, smartwatch, a sensor placed on the user’s belt, or otherwise—with an IMU. The device is also assumed to possess the ability to split measurements into shares using SSSS and stream them to an Internet-facing service over a secure channel, such as TLS using a pre-installed certificate. In this work, the IMU measurements are considered to be trusted; the focus is on the unauthorised disclosure of measurements after departing the device over a network to a cloud-based service.

3.2.2. Cloud service

Comprises the software components—database instance, REST APIs, and so on—for receiving, storing and classifying falls from IMU sensor data on a public cloud platform. The primary threat is the exfiltration and disclosure of raw IMU measurements from a remote attacker, such as from an unsecured, public-facing database, e.g. AWS S3 bucket, which could be used to model and infer users’ general ADLs, position, or gait for user identification. This work tackles the base case for MPC where parties operate under an honest-but-curious model, whereby the protocol specification is executed as intended. We also assume that the output, i.e. whether or not a fall occurred, is learned by the cloud services in order to contact an emergency number or provide alternative remediation. We note that some information may be revealed by the communication between the device and the cloud service, such as the absence of receiving protected IMU data at certain times of day, e.g. at night where the user deactivates their smartphone or smartwatch before sleeping. However, this is orthogonal to the ultimate goal of this work of preventing either the service or an external adversary from learning additional information from IMU measurements beyond that intended for detecting falls.

3.3. Workflow

The principle stages of the proposed system were illustrated previously in Figure 1 and described as follows:

1. (1)

Data acquisition. The collection of tri-axial IMU data on the device, e.g. smartphone, at an appropriate sampling frequency. This may be unimodal input, i.e. only accelerometer (Liu and Cheng, 2012; Mauldin et al., 2018; Cao et al., 2012; Doukas et al., 2007; Yavuz et al., 2010); multi-modal, e.g. accelerometer and gyroscope together (Santoyo-Ramón et al., 2018); or aggregating measurements from multiple sensors in a body-area network (Luštrek and Kaluža, 2009). 2. (2)

Secret sharing. Given a tri-axial IMU sample, $[A_{x},A_{y},A_{z}]$, the master device splits each component into $n$ shares using SSSS. In this work, we set $n$ equal to the number of authorised communicating parties, while the reconstruction threshold is fixed at $t=3$. In short, IMU samples are split as follows:

[TABLE]

Where the rows comprise the individual shares of a particular component, while each column represents the $i^{th}$ share of all components. 3. (3)

Share distribution. The device individually transmits the column elements from D to each cloud service instance; that is, $[A_{x}]_{0}$, $[A_{y}]_{0}$ and $[A_{z}]_{0}$, is sent to party $P_{0}$, $[A_{x}]_{1}$, $[A_{y}]_{1}$ and $[A_{z}]_{1}$ to $P_{1}$, and so on. By sharing each sample independently, no single cloud-based party may recover the underlying IMU samples assuming the reconstruction threshold above. 4. (4)

MPC-based inferencing. The parties use arithmetic operations upon the received shares to perform fall detection feature extraction, such as computing the vector magnitude of IMU samples, and subsequent classification using a desired machine learning classifiers, e.g. support vector machine (SVM), logistic regression (LR), and naïve Bayes (NB). 5. (5)

Classification decision. The resulting label, $\textbf{L}\in\{0,1\}$, is used to denote the occurrence of a fall in a binary fashion. This value may be used to raise an alarm or telephone an emergency contact number.

4. Implementation

This section describes the implementation of the device- and cloud-side systems introduced in the previous section.

4.1. Device Application

A Python application was developed that reads CSV IMU data line-by-line from file and creates shares of each sensor modality component using SSSS. The resulting shares are base 64-encoded and JSON-formatted, and transmitted over TLS to their respective cloud parties using the method described in Section 3.3. The application was deployed on a Raspberry Pi 3 Model B, with a Broadcom BCM2837 system-on-chip with a quad-core ARM Cortex-A53 at 1.2GHz, 1GB RAM, and Broadcom BCM43438 WiFi module (802.11N, 150 Mbit/s at 2.4GHz). The board was connected to a corporate WiFi network over which all communications occurred. Benchmarking procedures were also implemented using the Python time module, which wraps functions from the GNU C standard library, for measuring the upload time for transmitting the shares and receiving an acknowledgement from the server.

4.2. Cloud Framework

We developed a cloud-based framework for performing MPC using arithmetic secret sharing comprising three applications.

Web server. A Python web server application that implements a public-facing service for retrieving JSON-formatted shares from the device application; performs database initialisation, storage, and retrieval; and bootstraps the launching of MPC-based operations with other parties and returns the result to the device.

MPyC instance. We make use of MPyC—an open-source Python library developed by Schoenmakers (Schoenmakers, 2018) intended for the rapid prototyping of MPC-based systems. MPyC is based on the VIFF framework by Damgård et al. (Damgård et al., 2009), which implements numerous MPC operation protocols using arithmetic secret sharing, and abstracts the complexity of network management, e.g. socket connections, message formatting and transmission, and reconstructing and computing upon shares between the parties.

Database. Stores time-series IMU shares received from the device application; our implementation uses MongoDB222MongoDB: https://www.mongodb.com, a JSON document-based NoSQL database.

4.2.1. Deployment

We deploy our architecture using three MPC parties deployed in geographically disparate locations available via AWS: US West (Oregon), US East (Ohio), Central Canada, Europe (London), and South East Asia (Singapore). These configurations are listed in Table 3, representing a strictly North American deployment (Config A), North America and Europe (Config B), and North America, Europe and Asia (Config C). The three server-side applications, listed above, were deployed collectively on separate AWS EC2 instances in their respective region; each instance was also pre-loaded with the other instances’ TLS certificates and static IP addresses in configuration files. We use three t2.micro AWS instances, featuring a single-core Intel Xeon Scalable Processor at 3.3 GHz, with 1GB RAM and up to 0.72Gbit/s network throughput. At the time of publication, this incurs an operating cost of $0.0116 (USD) per hour, or$8.47 per month per instance (Amazon, 2019).

4.2.2. Feature Extraction

In this work, we implement MPC-based feature extraction for replicating the SmartFall proposal by Mauldin et al. (Mauldin et al., 2018), which uses two features: the samples’ vector magnitudes, and the difference between the maximum and minimum magnitudes, $\Delta s$ (Equations 1 and 2 respectively), over a sliding window of 750ms. This is managed in our framework by aggregating shares in an array of size 24, assuming a sampling frequency of 31.25Hz as in (Mauldin et al., 2018). The vector magnitude is found for each sample via the Newton-Raphson method for computing the square root, while $\Delta s$ relies upon MPyC’s maximum and minimum functions of a list of secret-shared elements. These are internally implemented by recursively finding the maximum (or minimum) of each list half, computing their difference, and performing a $\geq$ or $\leq$ operation with zero—the costs of which were listed in Table 2.

An important observation is that the choice of features dramatically affects the asymptotic cost of MPC-based operations. Features reliant upon large numbers of multiplication, comparison or Boolean operations will necessitate many rounds of inter-party communication versus features that can be computed locally, such as additions or multiplications with public values, as noted in Section 3.1.2. We note that the SmartFall features, i.e. the vector magnitude and computing $\Delta s$ in a sliding window, imposes significant cost for the square root and minimum and maximum functions respectively.

To address this, we also explore derivative-based features, which are used extensively for feature extraction in computer vision tasks (Lowe, 2004; Bay et al., 2008; Dalal and Triggs, 2005). In particular, inspired by the SURF features used in Bay et al. (Bay et al., 2008), we use the sum of the derivatives and the sum of the squared derivatives for each data channel within the window, resulting in a six-dimensional feature vector, $f$, listed in Equation 3.

[TABLE]

Where $\frac{dx}{dt}$, $\frac{dy}{dt}$, and $\frac{dz}{dt}$ represent the differentials of the $x$, $y$ an $z$ measurement components respectively. The derivative is computed by convolution as follows:

[TABLE]

Where $A_{u}$ is the $u$-th component of an accelerometer measurement and $h$ is a filter kernel with impulse response $[-0.5,0,0.5]$. This feature extraction scheme is substantially more efficient, requiring only a single round of inter-party communication for each derivative squaring operation; the summing operations can be performed locally without communication, as can the convolution operation using a filter kernel that performs multiplications with constant values.

4.2.3. Privacy-Enhancing Inferencing

We developed a range of MPC-based functions for privacy-preserving inferencing from measurement shares, which were implemented using the fundamental operations from Section 3.1.2. At present, we have developed MPC-based implementations of the following classifiers.

Logistic Regression (LR) is a linear model of the probability of a binary response given a feature vector, $\vec{x}=[x_{1},x_{2},\dots,x_{n}]$. LR applies the logit function to $\vec{x}$ and parameters, $\vec{\theta}$, that minimise some cost function, e.g. ordinary least squares. The decision boundary, $h$, is represented by a linear combination of the parameters and features; that is, $h(\vec{\theta},\vec{x})=\sum^{n}_{i=0}\theta_{i}x_{i}$.

A Support Vector Machine (SVM) attempts to find a hyperplane that maximises the margin (distance) between linearly separable data points of classes in the training set. We implement a traditional SVM with linear kernel for binary classification, whereby samples are classified using a given set of weights, $\vec{w}$, feature vector, $\vec{x}$, bias, $b$, and evaluating Equation 5. Note that, for inferencing, both SVM and LR reduce to a single inner product. With MPC, a naïve approach requires $m$ rounds of communication to compute each $\theta_{i}x_{i}$; however, (de Hoogh, 2012) demonstrated how to compute the inner product of two vectors using LSSS requiring only a single round of communication with (non-interactive) local summations, which is reproduced in Algorithm 1 for two shared vectors, $[\textbf{a}]$ and $[\textbf{b}]$.

[TABLE]

Naïve Bayes (NB) is a probabilistic classifier that uses Bayes’ Theorem with a strong independence assumption between features. A feature vector is subsequently classified by maximising the posterior probability for each class label using the prior and conditional probabilities computed in the training phase. In our case, we use Gaussian NB to compute the log-likelihoods of real-valued inputs, parameterised by mean $\mu$ with variance $\sigma^{2}$ and evaluated as follows:

[TABLE]

Where $n$ is the number of features and $P(C_{i})$ is the probability of the occurrence of class $C_{i}$.

5. Evaluation

This section presents a two-part evaluation comprising: 1), classification error rates achieved by our framework using MPC-based implementations of SmartFall and derivative-based features with the three inferencing algorithms; and 2), computational performance, for evaluating the time complexity of the framework.

5.1. Classification Error Rates

We now present error rates achieved by the models used in our MPC-based implementations of SmartFall (Mauldin et al., 2018) using the standard and derivative-based features described in Section 4.2.2. The authors of (Mauldin et al., 2018) provide a publicly-available dataset comprising labelled IMU accelerometer measurements. These correspond to binary fall events from seven participants between 21–55 years old wearing a Microsoft Band 2 smartwatch.

An off-line Python application was used to generate the model parameters. Firstly, a sliding window of 750ms was used with data sampled at a frequency of 31.25Hz, over which the vector magnitude and $\Delta s$ were computed. This was repeated for the derivative-based feature extraction. Next, six classifiers—Gaussian NB, SVM (with linear kernel), and logistic regression models for both feature types—were trained independently using the Scikit-Learn library (Pedregosa et al., 2011). The inclusion of logistic regression goes beyond the classifiers evaluated in (Mauldin et al., 2018). The dataset was split using a 80:20 training-test set ratio, and five-fold cross-validation was used to select the best performing model, whose final error rate was measured using the test set. The parameters of the best performing models were subsequently exported and implemented within the inferencing procedures detailed in Section 4.2.3.

Tables 4 and 5 presents the error rates for each model with respect to classification accuracy, precision, recall, and F1-scores averaged across all participants for the SmartFall and derivative-based features respectively. The achieved results, i.e. $93.6\%$ accuracy and $0.733$ F1-score in the best case for SVM, are commensurate with the original SmartFall proposal and other fall detection work listed previously in Table 1. We note that the derivative features outperform the original features in terms of precision, recall and F1-score, e.g. $0.804$ F1-score for NB versus $0.732$, and $0.835$ recall for SVM versus $0.663$; however, the classification accuracy is broadly similar for both feature types.

5.2. Computational Performance

This suite of experiments evaluates the latencies incurred by the device and cloud entities.

5.2.1. Device-side Latency

Here, we measure the average time to both construct the shares using SSSS and their transmission time to $n=3$ parties, with a reconstruction threshold of $t=3$. This is measured for the sharing and transmission of samples in a sliding window of 750ms (21 samples at 31.25Hz). The mean share construction time, averaged across all windows in the dataset, took 6.1ms ($\sigma=0.9$), with the upload time taking 398ms ($\sigma=23.9$ms), 404ms ($\sigma=25.9$ms) and 493.2ms ($\sigma=33.9$ms) for configurations A, B and C respectively. These timings are measured by the device-side Python application using the time module, which executed upon a Raspberry Pi 3 located in Belgium, Europe. The device specifications were listed previously in Section 4.1.

5.2.2. Cloud-side Latency

We measure the average wall-clock times of the MPC-based feature extraction and inferencing procedures, which were described in Sections 4.2.2 and 4.2.3. The total time for classifying each window can be considered as the sum of these two values. These were measured within the server-side Python application also via the time module. We report the times for each cloud configuration using the standard SmartFall features in Table 6 and the derivative-based features in Table 7. The results are also illustrated in Figures 3 and 4.

5.3. Discussion

One concern with MPC, raised previously in Section 3.1.2, is the inter-party communication cost for performing multiplication- and comparison-heavy functions, which were used in the feature extraction and inferencing procedures. With the original SmartFall features, inferencing and feature extraction takes approximately 1900ms in the worst case with Naive Bayes under Config C, to $\sim$1100ms in the best case using LR with Config A. We also observe that the feature extraction stage dominates the execution times using this feature type, accounting for approximately two-thirds of each tested algorithm’s total time. In contrast, this inverts with the use of derivative features, which require fewer rounds of inter-party communication; the inferencing stage becomes the dominant factor for all algorithms, rather than the feature extraction. The execution time is also substantially reduced for the derivative-based features; reducing from 1087ms to 365ms using LR with Config A, for example, between the SmartFall and derivative-based features respectively. In general, the derivative-based features are extracted $\sim$11x faster than the originally proposed SmartFall features in (Mauldin et al., 2018); the total execution time, including inferencing, exhibits a $\sim$2.8x increase between derivative- and SmartFall-based experiments.

We also see that the SVM and LR algorithms execute in similar times due to the mathematical similarities of inferencing, requiring an inner product operation that necessitates only a single round of communication. NB, however, is relatively expensive from requiring two multiplications for each feature in the window: one for finding the square of the feature subtracted from the mean, and the inverse multiplication (division) of this value by the variance. The choice of server configuration, i.e. the parties’ geographical distribution, also has a significant effect on the total execution time; increasing, for example, from approximately 1087ms to 1711ms using SmartFall features with SVM-based inferencing, corresponding to a $\sim$57% increase from Config A to C. Similar results are exhibited for all SmartFall- and derivative-based experiments.

Lastly, we observe that, in some instances, the total execution time falls within that required for acquiring the measurements within the window (750ms). This applies for all derivative-based prediction using configurations A and B, but only the SVM and LR inferencing algorithms using Config C. This leads us to believe that these would be best suited for real-time prediction.

6. Conclusion

A plethora of fall detection systems rely upon IMU data from user-owned commodity devices. However, as we discussed in Section 2, these proposals necessitate the collection of data that can be used to identify users’ gait, their position using dead reckoning, and determining their activities of daily living (ADLs). In light of this, we presented the first investigation of the application of multi-party computation for IMU-based fall detection with the aim of preserving the confidentiality of sensor measurements.

We presented the design and implementation of our system in Sections 3 and 4, which was subsequently deployed using three AWS EC2 instances in geographically disparate locations under three configurations. Each instance acted as an MPC participant for jointly computing the feature extraction and inferencing procedures using weights from pre-trained global models. The system was evaluated with the original SmartFall features alongside an exploration of derivative-based features for reducing the communication complexity imposed by MPC operations.

In Section 5, we presented the results from a two-part evaluation of the system’s classification error and computational performance. This was conducted using a publicly-available dataset comprising data from seven participants wearing an off-the-shelf device. The performance results indicate that state-of-the-art error rates can be achieved for MPC-based fall detection—achieving 0.926–0.937 classification accuracy and 0.731–0.804 F1-score depending on the chosen algorithm and feature set.

We also showed experimentally that the choices of inferencing algorithm, feature extraction, and server location have substantial effects on computational performance. The worst cases increase the total execution total by almost three-fold versus the best performers, which is dominated principally by the choice of features. In the best cases—for example, using an SVM classifier with derivative-based features in a cloud configuration in the same continent—the total prediction time is less than the on-device data acquisition time. This provides an indication that real-time prediction is feasible.

To conclude, we aim to pursue the following research directions in future work:

• •

MPC-based fall detection using deep learning. Existing fall detection systems rely heavily on traditional machine learning classifiers; a small selection of recent work, however, has begun to explore the use of deep neural networks (DNNs), such as RNNs (Musci et al., 2018). As such, a future avenue lies in exploring MPC-based DNNs and the overhead imposed by evaluating large numbers of intermediate layers with potentially hundreds or thousands of nodes.

• •

Video-based fall detection. A separate paradigm of fall detection systems, described in Section 2, relies upon video camera data from devices situated in the user’s ambient environment. This data, however, is of significantly greater dimension than the IMU data explored in this work; a future direction is investigating the extent to which MPC-based fall detection can be performed with high-frequency, high-dimensionality data.