Effectiveness Assessment of Cyber-Physical Systems

TL;DR

This paper introduces a formal measure of effectiveness for Cyber-Physical Systems using measure theory, TBM, and Ev-IOHMM, enabling in vivo evaluation and benchmarking of system performance under uncertainties.

Contribution

It develops a novel formal framework combining measure theory, Transferable Belief Model, and Ev-IOHMM to assess CPS effectiveness considering epistemic and aleatory uncertainties.

Findings

The measure of effectiveness can evaluate autonomous vehicle controllers.

The approach enables benchmarking against safety and well-being constraints.

Application to autonomous vehicles demonstrates practical utility.

Abstract

By achieving their purposes through interactions with the physical world, Cyber-Physical Systems (CPS) pose new challenges in terms of dependability. Indeed, the evolution of the physical systems they control with transducers can be affected by surrounding physical processes over which they have no control and which may potentially hamper the achievement of their purposes. While it is illusory to hope for a comprehensive model of the physical environment at design time to anticipate and remove faults that may occur once these systems are deployed, it becomes necessary to evaluate their degree of effectiveness in vivo. In this paper, the degree of effectiveness is formally defined and generalized in the context of the measure theory. The measure is developed in the context of the Transferable Belief Model (TBM), an elaboration on the Dempster-Shafer Theory (DST) of evidence so as to handle epistemic and aleatory uncertainties respectively pertaining the users' expectations and the natural variability of the physical environment. The TBM is used in conjunction with the Input/Output Hidden Markov Modeling framework (we denote by Ev-IOHMM) to specify the expected evolution of the physical system controlled by the CPS and the tolerances towards uncertainties. The measure of effectiveness is then obtained from the forward algorithm, leveraging the conflict entailed by the successive combinations of the beliefs obtained from observations of the physical system and the beliefs corresponding to its expected evolution. The proposed approach is applied to autonomous vehicles and show how the degree of effectiveness can be used for bench-marking their controller relative to the highway code speed limitations and passengers' well-being constraints, both modeled through an Ev-IOHMM.

Figures

Tables

Table 1. Table 9: Speed limitations in force and their dependency on several factors.

Localization	Weather condition	Road	Improved road	Highway
Outside urban area	No precipitation	90 km/h	110 km/h	130 km/h
	Rainy	80 km/h	100 km/h	110 km/h
	Visibility <50m	50 km/h	50 km/h	50 km/h
Urban area	30 km/h area	30 km/h	30 km/h	N/A
	General case	50 km/h	50 km/h	90 km/h (urban highway)
	Improved section	70 km/h	70 km/h	N/A

Table 2. Table 10: Speed limitation values are retrieved at each time t 𝑡 t from OSM specific infrastructure features given the vehicle position (longitude/latitude). The weather web services provides us with precipitation and visibility information.

OSM Road infrastructure max speed features			Weather data
From values	maxspeed:<value>	value	hour.precip	mm
From localization	zone:maxspeed:FR:30	30 km/h	hour.vis	km
	maxspeed:type:FR:urban	50 km/h
	maxspeed:type:FR:rural	90 km/h
	maxspeed:type:FR:trunk	110 km/h
	maxspeed:type:FR:motorway	130 km/h
	highway:living_street	50 km/h
	highway:residential	50 km/h
	highway:primary	90 km/h
	highway:secondary	70 km/h
	highway:tertiary	50 km/h
	highway:trunk	110 km/h
	highway:motorway	130 km/h

Equations

$$p\left((x_t{)}_{t=1}^T|({\vec{y}}_t{)}_{t=1}^T\right)=p(x_1)\times p(x_1|{\vec{y}}_1)\times \left[\prod _{t=2}^T\left(p(x_{(t)}|x_{(t-1)})\times p(x_{(t)}|{\vec{y}}_{(t)})\right)\right]$$

$$p\left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=p(x_1)\times p(x_1|{\vec{y}}_1)\times \left[\prod _{t=2}^T\left(p(x_{(t)}|x_{(t-1)},{\vec{u}}_{(t-1)})\times p(x_{(t)}|{\vec{y}}_{(t)})\right)\right]$$

$$\left\{\begin{array}[]{l l l}x_{(t)}&=\Gamma(x_{(t-1)},\vec{u}_{(t-1)})&\text{({State prediction})}\\ \psi_{(t)}&=G(x_{(t)})&\text{({State emission})}\\ U_{(t)}&=\mathcal{U}(x_{(t-1)},x_{(t)})&\text{(Finite set of inputs)}\\ Y_{(t)}&=\mathcal{Y}(x_{(t)})&\text{(Finite set of outputs)}\end{array}\right.$$

$$\text{mathds}{1}_{x_{(t)}}(x_{(t-1)},{\vec{u}}_{(t-1)})=$$

$$\text{mathds}{1}_{x_{(t)}}({\vec{y}}_{(t)})=$$

$$\boxed{Test(\underset{\text{Expected result}}{\underbrace{(x_t{)}_{t=1}^T}},\underset{Inputs}{\underbrace{({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T}})=\prod _{t=2}^T\left(\text{mathds}{1}_{x_{(t)}}(x_{(t-1)},{\vec{u}}_{(t-1)})\times \text{mathds}{1}_{x_{(t)}}({\vec{y}}_{(t)})\right)}$$

$$K_S:(\Omega \times \mathbb{U})\times {\Sigma }_{\Omega }\to [0,1]$$

$$K_E:\Omega \times {\Sigma }_{\mathbb{Y}}\to [0,1]$$

$$m^{\Omega }:2^{\Omega }\to [0,1]$$

$$m^{\Omega }(\{x_1,x_2\})\ne m^{\Omega }(\{x_1\})+m^{\Omega }(\{x_2\})$$

$$\sum _{A\in 2^{\Omega }}{m}^{\Omega }(A)=1$$

$$p{l}^{\Omega }(A)=\sum _{B\cap A\ne 0}{m}^{\Omega }(B),\forall A,B\in 2^{\Omega }$$

$$m^{\Omega }(A)=\sum _{B\subseteq A}(-1{)}^{|A|-|B|+1}p{l}^{\Omega }(\bar{B}),\forall A,B\in 2^{\Omega }$$

$$be{l}^{\Omega }(A)=\sum _{\varnothing \ne B\subseteq A}{m}^{\Omega }(B),\forall A,B\in 2^{\Omega }\text{, with mΩ normal, i.e. mΩ(∅)=0.}$$

$$q^{\Omega }(A)=\sum _{B\supseteq A}{m}^{\Omega }(B),\forall A,B\in 2^{\Omega }$$

$$m^{\Omega }(A)=\sum _{A\subseteq B}(-1{)}^{|B|-|A|}{q}^{\Omega }(B),\forall A,B\in 2^{\Omega }$$

$$m_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cap \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(A)=\sum _{B\cap C=A}{m}_1^{\Omega }(B).m_2^{\Omega }(C)\text{, }A\in 2^{\Omega }$$

$$q_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cap \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(A)=q_1^{\Omega }(A).q_2^{\Omega }(A)$$

$$m_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cap \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(\varnothing )=\sum _{A\cap B=\varnothing }{m}_1^{\Omega }(A).m_2^{\Omega }(B)\text{, }A,B\in 2^{\Omega }$$

$$m_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cap \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(\varnothing )=1+\sum _{A\in 2^{\Omega },A\ne \varnothing }(-1{)}^{|A|}\cdot q_1^{\Omega }(A).q_2^{\Omega }(A)$$

$$m_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cup \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(A)=\sum _{B\cup C=A}{m}_1^{\Omega }(B).m_2^{\Omega }(C)\text{, }A\in 2^{\Omega }$$

$$be{l}_{1\text{scalerel}*\text{leavevmode}to12.98pt\text{vbox}to12.98pt\text{pgfpicture}\text{makeatletter}\text{lower}-6.4876ptto0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\text{pgfsys@setlinewidth}0.4pt\text{pgfsys@invoke}\text{nullfont}to0.0pt\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@moveto}6.2876pt0.0pt\text{pgfsys@curveto}6.2876pt3.47258pt3.47258pt6.2876pt0.0pt6.2876pt\text{pgfsys@curveto}-3.47258pt6.2876pt-6.2876pt3.47258pt-6.2876pt0.0pt\text{pgfsys@curveto}-6.2876pt-3.47258pt-3.47258pt-6.2876pt0.0pt-6.2876pt\text{pgfsys@curveto}3.47258pt-6.2876pt6.2876pt-3.47258pt6.2876pt0.0pt\text{pgfsys@closepath}\text{pgfsys@moveto}0.0pt0.0pt\text{pgfsys@stroke}\text{pgfsys@invoke}\text{pgfsys@beginscope}\text{pgfsys@invoke}\text{pgfsys@transformcm}1.00.00.01.0-2.66667pt-2.22223pt\text{pgfsys@invoke}\text{definecolor}pgfstrokecolorrgb\mathrm{0,0,0}\text{pgfsys@color@rgb@stroke}000\text{pgfsys@invoke}\text{pgfsys@color@rgb@fill}000\text{pgfsys@invoke}\cup \text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{pgfsys@discardpath}\text{pgfsys@invoke}\text{lxSVG@closescope}\text{pgfsys@endscope}\text{hss}\text{lxSVG@closescope}\text{endpgfpicture}\sum 2}^{\Omega }(A)=be{l}_1^{\Omega }(A).be{l}_2^{\Omega }(A)$$

$$\boxed{\delta \left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=\underset{\text{Initialization}}{\underbrace{\pi (x_1)\otimes K_E(x_1,{\vec{y}}_1)}}\otimes \left[\underset{\text{prediction - update mechanism}}{\underbrace{\underset{t=2}{\overset{T}{⨂}}{K}_S\left((x_{(t-1)},{\vec{u}}_{(t-1)}),x_{(t)}\right)\otimes K_E\left(x_{(t)},{\vec{y}}_{(t)}\right)}}\right]}$$

$$\boxed{\delta \left(({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=\underset{(x_t{)}_{t=1}^T}{\max }\delta \left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)}$$

$$\delta \left(({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=1\text{ ,}\forall T\in {\mathbb{N}}^{\star }$$

$$\delta \left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=1$$

$$0<\delta \left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)<1$$

$$\delta \left((x_t{)}_{t=1}^T,({\vec{u}}_t,{\vec{y}}_t{)}_{t=1}^T\right)=0$$

$$\boxed{\begin{minipage}{0.0pt} $\delta^{\Omega_{T}}[(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}]=\max_{(x_{t})_{t=1}^{T}}$\\ $\left(m^{\Omega}_{\pi}(\{x_{1}\})\times m^{\Omega_{1}}[\vec{y}_{1}](\{x_{1}\})\times\left[\prod_{t=2}^{T}m^{\Omega_{(t)}|\Omega_{(t-1)}}[\{x_{(t-1)}\},\vec{u}_{(t-1)}](\{x_{(t)}\})\times m^{\Omega_{(t)}}[\vec{y}_{(t)}](\{x_{(t)}\})\right]\right)$ \end{minipage}}$$

$$q^{\Omega }[\vec{o}](A)=\prod _{x_i\in A}\mathcal{L}(x_i|\vec{o}),\forall A\in 2^{\Omega }$$

$$p{l}^{\Omega }[\vec{o}](\{x_i\})$$

Full text

††journal: International Journal of Approximate Reasoning

1 Introduction

Generally, computing systems are understood as being purposeful processing units, directed to produce expected results by means of computational resources manipulating data through controlled computational environments.

At the infrastructure level, some hardware and software mechanisms ensure correct operation of the computing resources (e.g. power-on self-test, etc.), integrity and persistence of the data (e.g. Cyclic Redundancy Check (CRC), memory content refresh, etc.). At the system level, accesses to the computational resources are made safe by an operating system or a middleware. The computational environments being controlled, the production and the persistence of the expected results are guaranteed ”by design” solely provided that the computer program issues the right commands to the computational resources. In this sense, a computer program is a perfect deterministic model of a computing system and the question does not even arise that, x being a variable, the execution of the following code snippet will lead its value to be set to 6 into the memory.

Let us now consider Cyber-Physical Systems (CPS) as being orchestrations of distributed computing and physical systems [1]. CPS can be understood as being ”cyber” physical processes where some properties of a physical system of interest are purposefully modified by means of computational resources manipulating them through transducers (e.g. sensors and actuators). For instance, let us keep the template of the preceding code snippet by considering that the variable to be modified now corresponds to a physical property of the physical system (e.g. the temperature in a living room).

What trust can we have that the temperature in the living room is going to be changed to \SI23\degC? In other words, can one consider the above code snippet as a perfect deterministic model of the physical system? Considering that the living room is a non-isolated physical system, the answer is ”no”. Such systems are driven by non-deterministic dynamics, at any time, the temperature of the living room can be affected by surrounding processes over which the computing system has no control [2] [3] [4]. This situation is aggravated for the Internet of Things (IoT)-based CPS whose underlying infrastructure is volatile. Indeed, their structural components being embedded into physical things, their availability cannot be ensured over time. Consequently, the attainment of the CPS purposes cannot be guaranteed solely ”by design” [1].

As a solution to this problem, we propose to quantitatively assess, at run-time, to which extent the CPS purposes are met. In other words, it is about providing the degree of effectiveness of the CPS as a measure of the concrete evolution of the physical system according to the expected evolution. To be more precise about the measure and the meaning we seek to give it as an assessment of the degree of effectiveness of the CPS, we borrow some terminology employed in the viability theory [5]. Let us assume that the expected evolution of the physical system can be specified as a deterministic model, free from uncertainties, where (1) state transitions are determined by contextual events (stimuli), (2) states are qualified by the expected physical effects resulting from actuators over which the computing system has control. Zones of Viability extend this deterministic point of view with tolerances accounting for aleatory uncertainties pertaining the natural variability of actuators effects and sensors readings and for epistemic uncertainties relative to the users’ satisfaction towards the concrete evolution of the physical system.

In this paper, we propose to generalize the deterministic model in the framework of the measure theory. Doing so, one can leverage the set of measures (probabilities, possibilities, etc.) as a means of defining zones of viability from which one can reason in order to obtain the degree of effectiveness. By obtaining a quantitative measure of the degree of effectiveness, (1) one can leverage this measure within a feedback loop so as for the controller of the system to minimize the behavioral drift (e.g. negative feedback control systems [6]), (2) one can use this measure as a bench-marking tool used to compare algorithms deployed for controlling CPS.

2 Related work and contributions

The work presented in this paper is closely related to the dependability of the computing systems [7]. Within computer science, this term refers to the trust that can justifiably be placed in the service delivered by computing systems and covers all their critical quality aspects [8]. In other words, it reflects users’ degree of trust in these systems. Among the attributes of dependability [9], availability (i.e. readiness for correct service), reliability (i.e. continuity of correct service), safety (i.e. absence of consequences on the users and the environment) and integrity (i.e. absence of improper system alterations) characterize the immunity of computing systems towards uncontrolled physical processes and associated uncertainties (i.e. threats that can affect computing systems operation and undermine their dependability [9]).

The assessment of the dependability can be done at design time through analytic metrics using models of the systems and, whenever possible, the known uncertainties (e.g. U-Test [10]). Run-time monitoring involves direct and indirect empirical metrics, respectively measuring the system itself through probes (whenever possible) and its effects within the physical environment through sensors.

While methodologies involved at design time (e.g. Model-based design) and at testing phase (e.g. Model checking, simulation, etc.) are respectively devoted to fault prevention and fault removal, run-time monitoring is devoted to automatic fault and anomaly detection [11]. The most common formulation of the anomaly detection problem is to determine if a given test sequence is anomalous with respect to normal sequences. More formally, given a set of $n$ normal sequences $S=\{(S_{k})_{1,k\in\mathbb{N}^{\star}},\ldots,(S_{k})_{n,k\in\mathbb{N}^{\star}}\}$ and a test sequence $(\mathfrak{S}_{k})_{k\in\mathbb{N}^{\star}}$, it is about computing an anomaly score for $(\mathfrak{S}_{k})_{k\in\mathbb{N}^{\star}}$, with respect to $S$. It is assumed that test sequences might be misaligned in time and space w.r.t the normal sequences. We do also consider complex, and collective anomalies. On the one hand, when contextual attributes can be associated with observations (e.g. time, location, etc.), contextual anomalies are corresponding to behaviors that are valid under some conditions but are abnormal in others. For instance, in European countries, normally high temperatures during the summer can be considered as contextual anomalies if they occur during the winter (time-based contextual anomaly). On the other hand, collective anomalies correspond to a collection of consecutive behaviors which are not abnormal by themselves but are abnormal when they occur together as a collection [12]. Approaches that address these anomalies fall into three categories described hereafter.

2.1 Prediction-based approaches

**These approaches consist in modeling legitimate behavior through a parametric model learned from observations and further used for predicting observation at each time $t$. Abnormal behaviors are those whose real observations differ from the predicted ones.

**

In [13], authors use stacked Long Short-Term Memory (LSTM) networks [14] for anomaly/fault detection in time series. A network is trained on non-anomalous data and used as a predictor over a number of time steps. The resulting prediction errors are modeled as a multivariate Gaussian distribution, which is used to assess the likelihood of anomalous behavior. In [15], authors present an unsupervised approach to detect cyber-attacks in Cyber-Physical Systems (CPS). A Recurrent Neural Network (RNN) [16] is used as a time series predictor. The Cumulative Sum method is further used to identify anomalies in a replicate of a water treatment plant.

pros & cons: these models are difficult to train [17] and are generally hardly interpretable, their intrinsic structure and parameters making unclear the mapping between the variables and the observations [18]. For instance, such models, once learned make difficult, if not impossible, the modification of their intrinsic parameters in order to tune a posteriori the tolerances pertaining the epistemic uncertainties. More importantly, learning a comprehensive model of the CPS behavior based on observations is often impracticable with regards to their complexity [1].

2.2 Model drift-based approaches

**These approaches are relative to the anomalous evolution of the model parameters. The basic idea is to build a parametric behavioral test model from test sequences as they arrive and compare it with the normal behavioral model. Dissimilarities between models give the anomaly score.

**

Authors in [19] focus on the quantitative measure of concept drift and introduce the notion of drift magnitude whose value can be quantified through distance functions such as Kullback-Leibler Divergence or Hellinger Distance. Close to the idea of concept drift is the notion of Bayesian Surprise [20]. A surprise quantifies how data affects an observer. It quantifies a mismatch between an expectation and what is actually observed by measuring the difference between posterior and prior beliefs of the observer. In [21] authors propose using Bayesian surprise as a measure of the learning progress of reinforcement learning agents.

pros & cons: being based on the distance between prior and posterior beliefs, the main disadvantage of these approaches concerns the speed of convergence to an accurate test model, highly dependent on the number of observations needed to learn it. Hereby, a short time anomalous behavior might be ”attenuated” or even not detected. These approaches are mainly leveraged in autonomic computing and the models@run-time community [22] where an initial model is updated over time taking into account unanticipated evolutions of the environment. In this context, above a given threshold, the quantitative drift value is used to trigger the update of the model with the newly learned parameters, assuming it represents the correct behavior.

2.3 Likelihood-based approaches

**These approaches consist in modeling legitimate behavior through a parametric model and considering abnormal behaviors as those having low ”likelihood” to have been generated by the model.

**

In this category, Dynamic Bayesian networks (DBN) and derivatives ($n$-order Markov chains) are widely used where tolerances towards uncertainties are generally described through probability density functions (pdf). An extension of the Markovian models, denoted by Hidden Markov Models (HMM), consists in considering the case where states of the model are ”hidden” [23], i.e. not directly observable, or partially hidden [24]. Such models are particularly well suited in the context of this paper where it is assumed that while the expected behavior of a CPS can be described a priori, the prior knowledge of its concrete internals and surrounding environment is unlikely available [1]. In this context, the likelihood of a given observation sequence $(\vec{y}_{t})_{t=1}^{T}$ is inferred from the model of the expected behavior of the system by using the probabilistic forward algorithm. This algorithm computes the likelihood of all the possible sequences of hidden states given the observation sequence $(\vec{y}_{t})_{t=1}^{T}$. The likelihood of a particular sequence of hidden states $(x_{t})_{t=1}^{T}$ given the observation sequence $(\vec{y}_{t})_{t=1}^{T}$ is given by:

[TABLE]

Some works have extended the HMM in the framework of the Transferable Belief Model (TBM) [25], an elaboration on the Dempster-Shafer Theory (DST) of evidence where tolerances towards uncertainties are neither described by probabilities but by belief functions. In [26], the author describes previous works in using HMM with TBM [27],[28] in the context of analyzing time series and denoted as Evidential HMM (EvHMM). Probability-based HMM is built upon the Closed World Assumption (CWA), i.e. probabilities are spread on the states $\{x_{1},\ldots,x_{N}\}$ defined in the model with $\sum_{i=1}^{N}p(x_{i})=1$, i.e. $p(\Omega)=1$ and $p(\emptyset)=0$. TBM, on his side, is built upon the Open World Assumption (OWA). It allows to associate a belief value (mass of conflict $m$) to the empty set, i.e. $m(\emptyset)\geq 0$, meant to quantify the degree of inconsistency of the observations with regards to the model. This is coherent with the meaning we seek to give to the measure of effectiveness. In this context, it is proven in [29] that the plausibility of the observation sequence to have been produced by the model, i.e. the plausibility of the model, is given by $pl(\Omega)=1-m(\emptyset)$ obtained from the evidential forward algorithm, likewise the likelihood obtained from the probabilistic forward algorithm.

Close to the concern of CPS behavioral analysis, the case of Evidential HMM with application to dynamical system analysis is described in [30]. However, HMM-based methods do not consider state-transitions probabilities governed by inputs necessary in modeling CPS expected behavior.

A way to cope with this limitation is to use the Input/Output HMM (IOHMM), first introduced in [31]. With this modeling framework, state-transitions probabilities are not hardcoded as it is the case with HMMs. Instead, the probability of a state-transition to occur depends on some input values. In this context, the observation sequence consists in an input sequence $(\vec{u}_{t})_{t=1}^{T}$ and an output sequence $(\vec{y}_{t})_{t=1}^{T}$. In this context, the likelihood of a particular sequence of hidden states $(x_{t})_{t=1}^{T}$ given the sequences $(\vec{u}_{t})_{t=1}^{T}$ and $(\vec{y}_{t})_{t=1}^{T}$ is given by :

[TABLE]

pros & cons: a key advantage here is that these models are interpretable, making clear (1) the mapping between the variables and the observations, (2) the description of the zones of viability through probabilities or belief functions. In this category, HMM-based modeling frameworks and more particularly the IOHMM where state-transitions probabilities depend on some input values, are well suited for representing dynamical systems [32][33]. Moreover, such models assume that the internals and the environment of the systems considered are not necessarily known a priori. This makes sense in the context of CPS that, with regards to their complexity, are unlikely to be comprehensively modeled. At best, one can define their expected behavior through the effects they are supposed to produce in response to some events. By assuming OWA, the Ev-IOHMM would be a good candidate so as to compute the degree of effectiveness of CPS. However, to date, no effort has been put on elaborating on such modeling framework.

2.4 Contributions

In this paper, we do extend previous works on the probabilistic and the possibilistic IOHMM likelihood-based approaches respectively described in [34] and [35] in the framework of the TBM (we denote Ev-IOHMM). The work done in [36],[37] and [38] being considered as the starting points, the main contributions of this paper are the following:

The degree of effectiveness is formally defined and generalized in the context of the measure and the viability theories, 2. 2.

The probabilistic IOHMM described in [31] is extended into the TBM framework, resulting in the Ev-IOHMM. To this end, we do rely on previous contributions done on extending HMM to EvHMM [29][27]. The associated evidential forward algorithm is provided and used for inferring the likelihood of the input/output observations to have been generated by the model whose zones of viability are neither defined through probabilities [34] nor possibilities [35] but by belief functions. 3. 3.

The Evaluation of the approach is carried out on a simple yet revealing example, complemented with a list of use-cases emphasizing its interest. Among these use-cases, we do elaborate on a use-case in the domain of autonomous vehicles. The idea is to leverage the proposed approach as means for designers to benchmark the control systems of these vehicles by relying on the measure of their effectiveness against constraints of the highway code relative to speed limitations and passengers well-being, both modeled through an Ev-IOHMM.

3 Mathematical background

3.1 Deterministic model of the expected behavior

In this paper, we do consider physical systems whose expected evolution under a CPS control can be constrained through a deterministic model $\lambda$ whose state transitions are determined by contextual events (stimuli) while states are qualified by the expected physical effects resulting from actuators over which the computing system has control. This model is formally defined by:

[TABLE]

with:

• –

$t\in\mathbb{N}^{\star}$,

• –

$x_{(t=1)}$ is the known initial state,

• –

$\Omega=\{x_{1},\ldots,x_{N}\}$ is the finite set of states,

• –

$\Gamma:\Omega\times\mathbb{R}^{m_{(t)}}\to\Omega$, $m_{(t)}=|U_{(t)}|\in\mathbb{N}^{\star}$, is a state-transition function mapping a state $x_{(t)}\in\Omega$ and an input vector $\vec{u}_{(t)}\in\mathbb{R}^{m_{(t)}}$ to a next state $x_{(t+1)}\in\Omega$. Each element of $\vec{u}_{(t)}$ qualifies the observation of an event supposed to act on the state $x_{(t)}$ to yield $x_{(t+1)}$. In this context, $\sqcup_{(x_{(t)}|x_{(t-1)})}=\{\vec{u}_{1},\ldots,\vec{u}_{n}\}$ (denoted $\sqcup_{(t)}$ in the sequel) represents the set of input vectors whose values are supposed to trigger a state-transition from the state $x_{(t)}$ to the state $x_{(t+1)}$.

• –

$G:\Omega\leadsto\psi_{\Omega}$, is a set-valued output function mapping each state $x_{(t)}$ to a set $\psi_{x_{(t)}}=\{\vec{y}_{1},\ldots,\vec{y}_{j}\}$ of expected observations while being in state $x_{(t)}$. The $p$ elements of $\vec{y}_{i}\in\mathbb{R}^{p_{(t)}}$, $p_{(t)}=|Y_{(t)}|$, $1\leq i\leq j$, qualifies an expected physical effect while being in state $x_{(t)}$.

• –

$\mathcal{U}:\Omega\times\Omega\leadsto U$ is a function mapping a state-transition $(x_{(t-1)},x_{(t)})$ to the set of inputs $U_{(t)}$ needed to qualify this state-transition,

• –

$\mathcal{Y}:\Omega\leadsto Y$ is a function mapping a state $x_{(t)}$ to the set of outputs $Y_{(t)}$ needed to qualify this state.

For instance, Fig.1 depicts the expected behavior of a simple CPS whose purpose is to adjust the luminosity (physical property) of a room (the physical system) according to whether an inhabitant is present or not. While no inhabitant is present in the room (characterized by $pres<3$) then the value provided from the luminosity sensor should be less than 5 (characterized by $lum<5$). Here $m=p=1$.

Here, one may see a parallel with unit tests performed in software engineering for validating an algorithm. Some inputs are provided to the algorithm. The output, resulting from the treatment of these inputs by the algorithm, is compared with an expected output value. In this context, let us imagine one want to test that the sequence $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$ leads the sequence of states $(x_{t})_{t=1}^{T}$ : the algorithm under test is the one controlling the CPS considered. The deterministic model of the expected behavior, here, plays the role of an unit test defined as follows :

[TABLE]

The result of the test is then computed by :

[TABLE]

without room for tolerance towards uncertainties, the test result can only be PASS or FAIL, i.e. $\in\{0,1\}$.

However, without being perfect, the luminosity level at 22.8 (state $x_{2}$ in Fig.1) when an inhabitant is present may be still acceptable and effectiveness $\in[0,1]$. So, one needs to extend the deterministic model allowing to define tolerances pertaining the following uncertainties:

• –

The aleatory uncertainties which are most likely objective and relative to the natural variability of the physical properties of interest whose values over time are most likely distributed around an average value,

• –

The epistemic uncertainties which are most likely subjective and relative to users’ satisfaction towards the physical system evolution.

Besides these uncertainties, one may also consider reliability uncertainties such as:

• –

The spatial uncertainties relative to the sensors location with respect to the physical property of interest,

• –

The hardware uncertainties relative to the sensors accuracy and resolution,

• –

The model uncertainties relative to the designer of the model and its expertise on the application domain.

3.2 Towards its formalization into the measure theory

So as to handle the uncertainties previously described, we propose to generalize the deterministic model in the framework of the measure theory. Doing so, one can leverage the set of measures (probabilities, possibilities, beliefs, etc.) as a means of defining zones of viability from which one can reason in order to obtain the degree of effectiveness.

3.2.1 Background

Before formally generalizing the deterministic model in the framework of the measure theory and defining the degree of effectiveness, let us first review some key concepts of the measure theory. The reader is referred to the literature for details on this theory [39].

Definition 1**.**

(Measure) Let $(\mathsf{X},\Sigma_{\mathsf{X}})$ be a measurable space where $\mathsf{X}$ is a countable set and $\Sigma_{\mathsf{X}}$ is a $\sigma$-algebra over $\mathsf{X}$. A function $\mu:\Sigma_{\mathsf{X}}\to\mathbb{R}_{\geq 0}$ is **[40]**:

Monotone if $\forall A,B\in\Sigma_{\mathsf{X}},A\subseteq B$ implies $\mu(A)\leq\mu(B)$, 2. 2.

Normalized if $\mu(\Sigma_{\mathsf{X}})=1$, 3. 3.

Non-negative if $\forall A\in\Sigma_{\mathsf{X}}$, $\mu(A)\geq 0$, 4. 4.

Additive if $\forall A,B\in\Sigma_{\mathsf{X}}$ where $(A\cap B)=\emptyset$ then $\mu(A\cup B)=\mu(A)+\mu(B)$,

The function $\mu$ is said to be an additive measure if it is monotone, non-negative, additive and $\mu(\emptyset)=0$. It is said to be a non-additive measure if it is monotone, non-negative, non-additive and $\mu(\emptyset)=0$. A measure is said to be a sub-measure if $\forall A,B\in\Sigma_{\mathsf{X}}$, $\mu(A\cup B)\leq\mu(A)+\mu(B)$.

Definition 2**.**

(Measurable Function) Let $(\mathsf{X},\Sigma_{X})$ and $(\mathsf{Y},\Sigma_{Y})$ be measurable spaces where $\mathsf{X}$ and $\mathsf{Y}$ are countable sets and where $\Sigma_{X}$ and $\Sigma_{Y}$ are finite $\sigma$-algebras. A function $f:(\mathsf{X},\Sigma_{X})\to(\mathsf{Y},\Sigma_{Y})$ is said measurable if $f^{-1}(A)\in\Sigma_{X}$ $\forall A\in\Sigma_{Y}$.

For instance, let $\mathsf{X}=\{a,b,c,d\}$, and $\Sigma_{X}=\{\{a,b\},\{c,d\},\mathsf{X},\emptyset\}$. Let $\mathsf{Y}=\{1,2,3\}$, and $\Sigma_{Y}=\{\{1\},\{2,3\},\{2\},\{1,3\},\{3\},\{1,2\},\mathsf{Y},\emptyset\}$. The function $f$, defined by $f(a)=1$, $f(b)=1$, $f(c)=2$ and $f(d)=2$, is measurable. Indeed, $f^{-1}(\{1\})=\{a,b\}\in\Sigma_{X}$, $f^{-1}(\{2,3\})=\{c,d\}\in\Sigma_{X}$, $f^{-1}(\{3\})=\emptyset\in\Sigma_{X}$, $f^{-1}(\{1,2\})=\mathsf{X}\in\Sigma_{X}$, etc.

Definition 3**.**

(Kernel) Let $(\mathsf{X},\Sigma_{X})$ and $(\mathsf{Y},\Sigma_{Y})$ be measurable spaces where $\mathsf{X}$ and $\mathsf{Y}$ are countable sets and where $\Sigma_{X}$ and $\Sigma_{Y}$ are finite $\sigma$-algebras. A finite kernel from $\mathsf{X}$ to $\mathsf{Y}$ is a function $K:\mathsf{X}\times\Sigma_{Y}\to\mathbb{R}_{\geq 0}$ that satisfies:

• –

$\forall x\in\mathsf{X},K(x,Y)$* is a measure on $(\mathsf{Y},\Sigma_{Y})$,*

• –

$\forall Y\in\Sigma_{Y},x\mapsto K(x,Y)$* is measurable.*

$\mathsf{X}$ and $\mathsf{Y}$ being countable sets, the kernel can be specified as a matrix $\{k(x,y):(x,y)\in\mathsf{X}\times\mathsf{Y}\}$. One can think of $k(x,y)$ as providing the conditional measure of $y$ given $x$. The kernel is referred to as a stochastic kernel (a.k.a. Markov kernel or probability kernel) when $K:\mathsf{X}\times\Sigma_{Y}\to[0,1]$ and $\sum_{y\in\mathsf{Y}}k(x,y)=1,\forall x\in\mathsf{X}$, i.e. $K(x,\mathsf{Y})=1$ $\forall x\in\mathsf{X}$.

Definition 4**.**

(Kernel Product) Let $(\mathsf{X},\Sigma_{X})$, $(\mathsf{Y},\Sigma_{Y})$ and $(\mathsf{Z},\Sigma_{Z})$ be measurable spaces. Let $k_{1}:\mathsf{X}\times\Sigma_{Y}\to\mathbb{R}_{+}$ and $k_{2}:(\mathsf{X}\times\mathsf{Y})\times\Sigma_{Z}\to\mathbb{R}_{+}$.

Then, one can define the kernel product $k_{1}\otimes k_{2}:\mathsf{X}\times(\Sigma_{Y}\otimes\Sigma_{Z})\to\mathbb{R}_{+}$ as a function of $k_{1}$ and $k_{2}$ [41] where $\otimes$ is a product operator111In the literature, this operator is also known as fusion operator [42].

Theorem 5**.**

(Ionescu-Tulcea Extension Theorem) **[43]** Let us consider a sequence of measurable spaces $(\mathsf{X}_{n},\Sigma_{n})_{n\in\mathbb{N}^{\star}}$. Let assume that for each $n$, there exists a kernel $K_{n}$ from $\times_{k=1}^{k=n-1}X_{k}$ to $X_{n}$. Then, for every sequence $(S_{n})_{n\in\mathbb{N}^{\star}}$ taking values in $(X_{n},\Sigma_{n})$ there exists a unique measure $\mu(S_{1},S_{2},\ldots,S_{n})=\otimes_{k=1}^{n}K_{k}$.

With these key concepts defined, one can generalize the deterministic model described by Eq.3 in the measure theory framework.

3.2.2 Generalizing the function $\Gamma$ to the finite kernel $K_{S}$

Let us consider the measurable spaces $(\Omega\times\mathbb{U},\Sigma_{\Omega\mathbb{U}})$ and $(\Omega,\Sigma_{\Omega})$ where $\Omega$ is the finite set of states, $\mathbb{U}\subseteq\mathbb{R}^{m}$ is the input vector, $\Sigma_{\Omega\mathbb{U}}$ is a finite $\sigma$-algebra on $\Omega\times\mathbb{U}$ and $\Sigma_{\Omega}$ is a finite $\sigma$-algebra on $\Omega$. A finite kernel $K_{S}$ from $\Omega\times\mathbb{U}$ to $\Omega$ is defined by (Definition.3):

[TABLE]

$\Omega$ being a countable set, the kernel $K_{S}$ can be specified as a matrix $\{k_{S}((x,\vec{u}),x^{\prime}):((x,\vec{u}),x^{\prime})\in(\Omega\times\mathbb{U})\times\Omega\}$. Think of $k_{S}((x,\vec{u}),A)$ as the conditional measure that the process will be in the state $A\subset\Omega$ at time $t$ given its state at time $t-1$ is $x\in\Omega$ and the input vector is $\vec{u}$. Here, it is assumed that the state at time $t$ depends on the state at time $t-1$ and not on the previous states $t-2,t-3,\ldots,t_{1}$ (first order Markov property). Thus, ${\{\mathfrak{X}_{t},\mathfrak{U}_{t}\}}_{t\in\mathbb{N}^{\star}}$, where $\mathfrak{X}_{t}$ and $\mathfrak{U}_{t}$ are random variables taking values in $\Omega$ and $\mathbb{U}$ respectively, is a chain with kernels ${K_{S}}_{(t\geq 2)}$ and initial distribution $\pi$ where $\pi:\Sigma_{\Omega}\to[0,1]$ is a measure on $(\Omega,\Sigma_{\Omega})$ at $t=1$.

3.2.3 Generalizing the function $G$ to the finite kernel $K_{E}$

Let us consider the measurable spaces $(\Omega,\Sigma_{\Omega})$ and $(\mathbb{Y},\Sigma_{\mathbb{Y}})$ where $\mathbb{Y}\subseteq\mathbb{R}^{p}$ is the output vector and $\Sigma_{\mathbb{Y}}$ is a $\sigma$-algeba on $\mathsf{\mathbb{Y}}$. A finite kernel $K_{E}$ from $(\Omega,\Sigma_{\Omega})$ to $(\mathbb{Y},\Sigma_{\mathbb{Y}})$ is defined by:

[TABLE]

Think of ${K_{E}}_{(t)}(x,\vec{y})$ as the conditional measure that the process is in the state $x$ at time $t$ given the output vector $\vec{y}\subset\mathbb{Y}$ at time $t$.

Per Definition.4, at each time $t$, the Markov transition kernel $K_{S}\otimes K_{E}:(\Omega\times\mathbb{U})\times(\Sigma_{\Omega}\otimes\Sigma_{\mathbb{Y}})\to[0,1]$, is a function of $K_{S}$ and $K_{E}$ (Fig.2). Think of $K_{S}\otimes K_{E}$ as the conditional measure of $x_{(t)}$ given $\vec{y}_{(t)}$, $\vec{u}_{(t-1)}$ and $x_{(t-1)}$.

3.3 Transferable Belief Model (TBM)

3.3.1 Basic definitions and notations

Let us consider $\Omega=\{x_{1},x_{2},\ldots,x_{N}\}$ the discrete frame of discernment (FoD) representing the states of a physical system where $x_{1},x_{2},\ldots$ are hypothesis. In this paper, hypothesis are supposed to be exhaustive and exclusive, i.e. the system cannot be in two states at once. A mass function $m^{\Omega}$, a.k.a. Basic Belief Assignment (BBA) is defined by:

[TABLE]

where $2^{\Omega}=\{\emptyset,\{x_{1}\},\{x_{2}\},\{x_{1},x_{2}\},\{x_{3}\},\{x_{1},x_{3}\},\{x_{2},x_{3}\},\{x_{1},x_{2},x_{3}\},...,\Omega\}$. A BBA is non-additive, i.e

[TABLE]

This is a fundamental difference with probability theory. A proposition $A=\{x_{1},x_{2}\}\in 2^{\Omega}$ explicitly represents the doubt between hypothesis composing $A$ and the mass of belief $m^{\Omega}(A)$ assigned to $A$ is not informative regarding the elements of $A$.

A BBA is a set of belief masses concerning propositions $A\in 2^{\Omega}$ verifying:

[TABLE]

$A\in 2^{\Omega}$ is a focal element of the BBA if $m^{\Omega}(A)>0$.

In the Dempster-Shafer theory of evidence $m^{\Omega}(\emptyset)$ is constrained to 0. This constraint is relaxed in TBM [25] where $m^{\Omega}(\emptyset)>0$ is given different interpretations [44]:

Inaccuracy of the sensors measurements (Observations),

Incompleteness of the model leading to non-exhaustive FoD.

3.3.2 Belief functions

BBAs can be transformed to one-to-one relationships [25] representing the same information (a.k.a. belief functions), albeit in different forms. Some are described hereafter.

• –

Plausibility $pl$ where

[TABLE]

and reversely

[TABLE]

• –

Belief $bel$ where

[TABLE]

• –

Commonality $q$ where

[TABLE]

and reversely

[TABLE]

3.3.3 CRC/DRC combination rules

There have been many combination rules proposed in the literature [45]. In the sequel, we do consider the Conjunctive Rule of Combination (CRC) and the Disjunctive Rule of Combination (DRC).

Definition 6**.**

**Conjunctive Rule of Combination (CRC). Let us consider two BBAs defined by $m^{\Omega}_{1}$ and $m^{\Omega}_{2}$.

Assuming their sources are independent and reliable then the unnormalized conjunctive rule of combination (CRC \operatorname*{\scalerel*{\leavevmode\hbox to14.7pt{\vbox to14.7pt{\pgfpicture\makeatletter\hbox{\hskip 7.34819pt\lower-7.34819pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.1482pt}{0.0pt}\pgfsys@curveto{7.1482pt}{3.94788pt}{3.94788pt}{7.1482pt}{0.0pt}{7.1482pt}\pgfsys@curveto{-3.94788pt}{7.1482pt}{-7.1482pt}{3.94788pt}{-7.1482pt}{0.0pt}\pgfsys@curveto{-7.1482pt}{-3.94788pt}{-3.94788pt}{-7.1482pt}{0.0pt}{-7.1482pt}\pgfsys@curveto{3.94788pt}{-7.1482pt}{7.1482pt}{-3.94788pt}{7.1482pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.33334pt}{-2.77779pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{\cap}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\textstyle\sum}}) can be used as follows [37]: **

$\forall A\in 2^{\Omega}$, by:

[TABLE]

[TABLE]

This combination may result in a sub-normal BBA, i.e. $m^{\Omega}(\emptyset)>0$. The mass of conflict is given by:

[TABLE]

It is worth noting that the CRC can be computed from commonality functions:

[TABLE]

Definition 7**.**

**Disjunctive Rule of Combination (DRC).

Let us consider two BBAs defined by $m^{\Omega}_{1}$ and $m^{\Omega}_{2}$.

Assuming their sources are independent and at least one source is reliable, then the unnormalized disjunctive rule of combination (DRC \operatorname*{\scalerel*{\leavevmode\hbox to14.7pt{\vbox to14.7pt{\pgfpicture\makeatletter\hbox{\hskip 7.34819pt\lower-7.34819pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.1482pt}{0.0pt}\pgfsys@curveto{7.1482pt}{3.94788pt}{3.94788pt}{7.1482pt}{0.0pt}{7.1482pt}\pgfsys@curveto{-3.94788pt}{7.1482pt}{-7.1482pt}{3.94788pt}{-7.1482pt}{0.0pt}\pgfsys@curveto{-7.1482pt}{-3.94788pt}{-3.94788pt}{-7.1482pt}{0.0pt}{-7.1482pt}\pgfsys@curveto{3.94788pt}{-7.1482pt}{7.1482pt}{-3.94788pt}{7.1482pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.33334pt}{-2.77779pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{\cup}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\textstyle\sum}}) can be used as follows [37]: **

[TABLE]

[TABLE]

4 Degree of effectiveness

4.1 Formalization in the measure theory

On the basis of the formalization of the deterministic model of the CPS expected behavior in the measure theory described in 3.2 and extending Eq.4, the degree of effectiveness can be formulated as follows:

Definition 8**.**

The degree of effectiveness is a function $\delta:\Omega_{(t)}\times{(\mathbb{U}\times\mathbb{Y})}_{(t)}\to[0,1],t\in\mathbb{N}^{\star}$ such that given the state sequence $(x_{t})_{t=1}^{T}$ and the observation sequence $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$, the degree of effectiveness $\delta\left((x_{t})_{t=1}^{T},(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}\right)$ is given by:

[TABLE]

The process consists in propagating the measure over the state sequence $(x_{t})_{t=1}^{T}$. At each time $t$, it satisfies a ”prediction ($K_{S}$) - update ($K_{E}$)” mechanism. It can be understood as the ’likelihood’ of the state sequence $(x_{t})_{t=1}^{T}$ to have been produced by the observation sequence $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$. $\pi(x_{i})$ gives the measure of the state sequence to start by the state $x_{i}$.

As Eq.21 provides the degree of effectiveness for one possible sequence of states, one needs to find the sequence of states leading the highest degree of effectiveness over all the possible sequences of states given the observation sequence, i.e.

[TABLE]

Note 1**.**

The chain ${\{\mathfrak{X}_{t},\mathfrak{U}_{t},\mathfrak{Y}_{t}\}}_{t\in\mathbb{N}^{\star}}$ where $\mathfrak{X}_{t}$,$\mathfrak{U}_{t}$* and $\mathfrak{Y}_{t}$ are random variables taking values in $\Omega$, $\mathbb{R}^{m_{(t)}}$ and $\mathbb{R}^{p_{(t)}}$ respectively, with transition kernel $K_{S}\otimes K_{E}$ and initial distribution $\pi\otimes K_{E}$, is an Input/Output Hidden Markov Model (IOHMM)222In this paper we do assume that the state at time $t$ only depends on the state at time $t-1$ and not on the previous states $t-1,t-2,\ldots,t_{1}$ (first order Markov chain). [46] (claim derived from [47]).*

Following this definition, an observation sequence $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$ is said perfect when

[TABLE]

To be more precise about the meaning we seek to give to the degree of effectiveness, we borrow some terminology employed in the viability theory [5]. Let us consider that the constraints the physical system evolution has to comply with are encoded into information $I$. Then, the following definitions are adopted:

Definition 9**.**

A zone of comfort $\textnormal{{Cfrt}}_{(t)}(I)$ associated to an event $E\in\Sigma_{\Omega}$ at time $t$ corresponds to the set $\mathcal{C}\subset\mathbb{U}\times\mathbb{Y}$ of values for which the event $E$ is certain according to I, such that:

[TABLE]

Definition 10**.**

A zone of tolerance $\textnormal{{Tol}}_{(t)}(I)$ associated to an event $E\in\Sigma_{\Omega}$ at time $t$ corresponds to the set $\mathcal{T}\subset\mathbb{U}\times\mathbb{Y}$ of values for which the event $E$ is uncertain according to I, such that:

[TABLE]

Definition 11**.**

A zone of viability $\textnormal{{Viab}}_{(t)}(I)$ associated to an event $E\in\Sigma_{\Omega}$ at time $t$ corresponds to the union of $\textnormal{{Cfrt}}_{(t)}(I)$ and $\textnormal{{Tol}}_{(t)}(I)$. Note that outside the zone of viability, the event $E$ is impossible according to I:

[TABLE]

Thus, the degree of effectiveness determines zones of viability according to the model, i.e. it determines the boundaries of the states defined in the model. When $\delta(.)=0$, one faces a model breakdown, i.e. the state of the system is outside the boundaries of the states defined in the model.

The Fig.3 provides an illustrative example. Here, the event E can be stated as ”the passengers of the ship are safe”. An input of the model might be the geographic position of the ship (latitude/longitude), while the output might be the heart rate of the passengers. Within the zone of comfort one can be certain that the passengers are safe, i.e. their heart rate is at the expected level. Within the zone of tolerance, passengers may suffer from disturbances and their safety is at risk, i.e. their heart rate is higher than expected. The ship is not supposed to go outside the boundary of the zone of viability…

4.2 Application to the Transferable Belief Model

Per Eq.21 and Eq.22, by replacing kernels $K_{S}$ and $K_{E}$ with BBAs, the computation of the degree of effectiveness can be factored as follows :

[TABLE]

where $m^{\Omega_{(t)}|\Omega_{(t-1)}}[A](B)$ represents a belief function defined on $\Omega_{(t)}$ conditionally to the subset $A\subseteq\Omega_{(t-1)}$. For the sake of simplicity, $m^{\Omega_{(t)}|\Omega_{(t-1)}}[A](B)$ is replaced by $m^{\Omega_{(t)}}[A](B)$ in the sequel. It is worth noting that in Eq.26 the masses involved in the computation are supposed to be known.

4.3 Evidential Input/Output Hidden Markov Model (Ev-IOHMM)

The work presented in this paper extends works done on Evidential HMM (Ev-HMM) [26][27] and probabilistic Input/Output HMM (IOHMM) [31]. In the sequel, we do assume the reader is familiar with basics in HMM.

Formally, an Evidential IOHMM (Ev-IOHMM) is defined by the tuple $\lambda=<\Omega,A,\vec{B},m^{\Omega}_{\pi}>$ where:

• –

$\mathbm{\Omega}=\{x_{1},x_{2},\ldots,x_{N}\}$ is the finite set of hidden states, i.e. the frame of discernment,

• –

$\mathbm{\vec{B}}$ is the $2^{|\Omega|}$ emission vector whose elements represent the beliefs conditional to the output value $\vec{y}$. For instance, $m^{\Omega_{(t)}}[\vec{y}_{(t)}](\{x_{1}\}_{(t)})$ represents the belief in $\{x_{1}\}$ at time $t$ given the output observation $\vec{y}_{(t)}$ at time t.

• –

$\mathbm{A}$ is the $|\Omega|\times 2^{|\Omega|}$ state-transition matrix. There is one row per singleton $\in\Omega$. Each row of the matrix is a BBA whose elements represent the belief in transiting from the singleton to this element. For instance, in Ev-HMM, $m^{\Omega_{(t)}}[\{x_{1}\}_{(t-1)}](\{x_{2}\}_{(t)})$ represents the belief in transitioning to state $\{x_{2}\}$ at time $t$ given the state at time $t-1$ was $\{x_{2}\}$. Here the belief is only conditional to the previous state. In Ev-IOHMM, the belief in transitioning from one state to another is also conditional to an input. For instance, $m^{\Omega_{(t)}}[\{x_{1}\}_{(t-1)},\vec{u}_{(t-1)}](\{x_{2}\}_{(t)})$ represents the belief in transitioning to state $\{x_{2}\}$ at time $t$ given the state at time $t-1$ was $\{x_{2}\}$ and the input value was $\vec{u}$..

• –

$\mathbm{m^{\Omega}_{\pi}}$ is a vacuous BBA, i.e. $m^{\Omega}_{\pi}(\Omega)=1$ meant to indicate that one has no information on the initial state of the system.

Note 2**.**

In real life applications, BBAs are often not directly available. Only the probability or the possibility values computed from observations are available on the singletons. So, the model is extended with a vector $\vec{B}^{\prime}$ and a matrix $A^{\prime}$ whose elements describe probability density functions or distributions of possibility :

• –

$\mathbm{\vec{B}^{\prime}}$* is a $|\Omega|$ vector where each element $b_{i}$, $(1\leq i\leq|\Omega|)$ is a probability density function or distribution of possibility. For instance, ${b_{i}}(\vec{y})=p(\vec{y}_{(t)}=\vec{y}|x_{(t)}=i)$ denotes the probability of observing the output vector $\vec{y}$ at time $t$ given the state is $x_{i}$ at time $t$.*

• –

$\mathbm{A^{\prime}}$* is a $|\Omega|\times|\Omega|$ matrix where each element $a_{ij}$, $(1\leq i,j\leq|\Omega|)$ is a probability density function or distribution of possibility. For instance, $a_{ij}(\vec{u})=p(x_{(t+1)}=j|x_{(t)}=i,\vec{u}_{(t)}=\vec{u})$ denotes the probability of transiting to $x_{(t+1)}$ at time $t+1$, given the state is $x_{i}$ at time $t$ and the input vector is $\vec{y}$ at time $t$.*

The distributions in $A^{\prime}$ and $B^{\prime}$ can be defined by the designer of the model when distributions represent, for instance, users’ preferences or specific behavioral requirements/constraints. However, to date, no effort has been put on learning the model parameters from observations (following what has been done in **[26]** on the Ev-HMM or in **[49]** for the IO-HMM).

The HMM modeling framework and derivatives rely on computationally efficient reasoning algorithms [50]. Among these algorithms, the forward algorithm offers a solution to the evaluation problem. It computes the ”likelihood” of the observation sequences $(\vec{u}_{t})_{t=1}^{T}$ and $(\vec{y}_{t})_{t=1}^{T}$ to have been produced by the model by taking into account all the possible underlying state sequences. In other words, it provides a solution to the equation Eq.26.

4.3.1 State prediction

Given this model, let us now detail the basic mechanics of the Ev-IOHMM state-prediction. Let us consider the two states model depicted in Fig.4 extending the model depicted in Fig.1 with constraints taking into account uncertainties described as distributions of possibility as depicted in Fig.5.

Let us assume that the input $pres$ at time $t-1$ was 3.5. Recall that constraints are encoded in the form of probability or possibility functions. So, one needs to compute the possibility value of the input value for each state-transition in the matrix $\mathbm{A^{\prime}}$. It gives:

[TABLE]

Now, beliefs allocated to the subsets $\in 2^{\Omega}$, i.e. elements of the matrix A, can be deduced from beliefs on the singletons obtained from the observations and the distributions of possibility described in matrix A’ as follows.

• –

When beliefs on singletons are obtained from probability density functions (likelihoods $\mathcal{L}(x_{i}|\vec{o})$), one can obtain commonality $q^{\Omega}$ by [51]:

[TABLE]

• –

When beliefs on singletons are obtained from possibility distributions $\sqcap_{x_{i}}$, one can obtain plausibility $pl^{\Omega}$ by [52]:

[TABLE]

By applying Eq.29 and then Eq.11 for transforming $pl^{\Omega}$ to $m^{\Omega}$, one obtains:

[TABLE]

[TABLE]

For the time being, only the BBAs conditional to the singletons $[x_{1_{(t-1)}}]$ and $[x_{2_{(t-1)}}]$ are available. So, one needs to compute beliefs conditional to the subsets $[\emptyset_{(t-1)}]$ and $[\Omega_{(t-1)}]$. Per [37], this can be achieved by applying a DRC on BBAs conditional to the singletons as follows:

[TABLE]

For instance, m^{\Omega_{(t)}}_{a}[\Omega]=m^{\Omega_{(t)}}_{a}[x_{1_{(t-1)}}]\operatorname*{\scalerel*{\leavevmode\hbox to14.7pt{\vbox to14.7pt{\pgfpicture\makeatletter\hbox{\hskip 7.34819pt\lower-7.34819pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.1482pt}{0.0pt}\pgfsys@curveto{7.1482pt}{3.94788pt}{3.94788pt}{7.1482pt}{0.0pt}{7.1482pt}\pgfsys@curveto{-3.94788pt}{7.1482pt}{-7.1482pt}{3.94788pt}{-7.1482pt}{0.0pt}\pgfsys@curveto{-7.1482pt}{-3.94788pt}{-3.94788pt}{-7.1482pt}{0.0pt}{-7.1482pt}\pgfsys@curveto{3.94788pt}{-7.1482pt}{7.1482pt}{-3.94788pt}{7.1482pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.33334pt}{-2.77779pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{\cup}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\textstyle\sum}}m^{\Omega_{(t)}}_{a}[x_{2_{(t-1)}}]. Results are given in Table.4.

[TABLE]

We are now ready to compute states prediction at time $t$ given states at time $t-1$. The prediction is obtained using the following generalized conjunctive form [37]:

[TABLE]

where $q^{\Omega_{(t)}}_{a}[X,\vec{u}_{(t-1)}](A)$ corresponds to the matrix given in Table.4. Without an a priori on the previous states, i.e. $m^{\Omega_{(t-1)}}_{\alpha}(\Omega_{(t-1)})=1$, the predicted BBA is given from Eq.31 further transformed to $m$:

[TABLE]

4.3.2 State emission

Let us also assume that the output $lum$ at time $t$ is 2.34. So, one needs to compute the possibility value of the output value for each state from the vector $\mathbm{\vec{B}^{\prime}}$ from which the BBA can further be computed. It gives:

[TABLE]

By applying Eq.29 and then Eq.11 for transforming $pl^{\Omega}$ to $m^{\Omega}$, one obtains:

[TABLE]

[TABLE]

In the sequel, the Ev-IOHMM forward algorithm is detailed. This algorithm computes the likelihood of the observation sequences $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$ in the form of a BBA from which the degree of effectiveness is computed.

Note 3**.**

The forward algorithm described in the next section makes use of the CRC (7) for propagating beliefs. Other combination rules such as the Cautious Conjunctive Rule of Combination (CCRC) and the Bold Disjunctive Rule of Combination (BDRC) have been introduced in the TBM framework [53]. However, it is shown that the CRC is the only rule satisfying the Shafer-Shenoy axioms for belief functions propagation [54][55].

{comment}

4.3.3 The Ev-IOHMM Forward algorithm

The Ev-IOHMM forward algorithm is close to the Ev-HMM forward algorithm described in [26] and [30]. The main difference consists in conditioning the state-transition not only on the previous state $x_{(t-1)}$ but also on the input observation $\vec{u}_{(t-1)}$. Thus, the forward algorithm is given by :

**Initialization

**no a priori is given to the initial state of the system, i.e. $m^{\Omega}_{pi}(\Omega)=1$. Thus, $\forall X_{(t=1)}\in 2^{\Omega}$

[TABLE]

**Induction

$\forall X_{(t)}\in 2^{\Omega}$**, $2\leq t\leq T$

[TABLE]

It is worth noting that, at each time $t$, the resulting $q^{\Omega_{(t)}}_{\alpha}(X_{(t)})$ has to be transformed to $m^{\Omega_{(t)}}_{\alpha}(X_{(t)})$ by using Eq.14. Thus, one obtain the BBA $m^{\Omega_{(t)}}_{\alpha}$ resulting from the combination of the belief at time $t$ with the previous beliefs combined together. However, successive combinations lead the conflict to increase over time, i.e.

[TABLE]

So as to cope with this problem, the BBA $m^{\Omega_{(t)}}_{\alpha}$ is normalized at each time $t$ (from $t=1$) by redistributing the conflict over propositions $\in 2^{\Omega}$. Several strategies have been defined for redistributing the conflict [56] so as to keep $m^{\Omega_{(t)}}_{\alpha}(\emptyset)=0$ at each time $t$. For instance, assuming sources are equally reliable and {m_{1\operatorname*{\scalerel*{\leavevmode\hbox to12.08pt{\vbox to12.08pt{\pgfpicture\makeatletter\hbox{\hskip 6.03838pt\lower-6.03838pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.83838pt}{0.0pt}\pgfsys@curveto{5.83838pt}{3.22449pt}{3.22449pt}{5.83838pt}{0.0pt}{5.83838pt}\pgfsys@curveto{-3.22449pt}{5.83838pt}{-5.83838pt}{3.22449pt}{-5.83838pt}{0.0pt}\pgfsys@curveto{-5.83838pt}{-3.22449pt}{-3.22449pt}{-5.83838pt}{0.0pt}{-5.83838pt}\pgfsys@curveto{3.22449pt}{-5.83838pt}{5.83838pt}{-3.22449pt}{5.83838pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.33333pt}{-1.94444pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{\cap}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\textstyle\sum}}2}^{\Omega}}(\emptyset)<1, the Dempster’s normalization rule redistributes conflict on the focal elements, i.e. $\forall A\in 2^{\Omega},A\neq\emptyset$ and {m_{1\operatorname*{\scalerel*{\leavevmode\hbox to12.08pt{\vbox to12.08pt{\pgfpicture\makeatletter\hbox{\hskip 6.03838pt\lower-6.03838pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.83838pt}{0.0pt}\pgfsys@curveto{5.83838pt}{3.22449pt}{3.22449pt}{5.83838pt}{0.0pt}{5.83838pt}\pgfsys@curveto{-3.22449pt}{5.83838pt}{-5.83838pt}{3.22449pt}{-5.83838pt}{0.0pt}\pgfsys@curveto{-5.83838pt}{-3.22449pt}{-3.22449pt}{-5.83838pt}{0.0pt}{-5.83838pt}\pgfsys@curveto{3.22449pt}{-5.83838pt}{5.83838pt}{-3.22449pt}{5.83838pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.33333pt}{-1.94444pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{\cap}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\textstyle\sum}}2}^{\Omega}}(A)>0,

[TABLE]

Other normalization rules exist [56]. For instance, Dubois-Prade normalization rule assumes that at least one source is reliable in case of conflict. $\forall A\in 2^{\Omega},A\neq\emptyset$, the rule is defined by:

[TABLE]

Termination:

In this paper, we do consider leveraging the conflict as a means to provide the degree of effectiveness as done in [26]. Following the proof given in [26] in the context of Ev-HMM where :

[TABLE]

the degree of effectiveness given the Ev-IOHMM $\Theta$ and the observation sequence $(\vec{u}_{t},\vec{y}_{t})_{t=1}^{T}$ is given by :

[TABLE]

As described in [30], one needs to record the value of the conflict at each time $t$ before normalizing the BBA.

Note 4**.**

The degree of effectiveness may also be a good indicator of the quality of the model **[30]**. Indeed, given a perfect sequence of observations, the degree of effectiveness is supposed to be equal to 1.0 (see Definition.9).

5 An application to autonomous vehicles

Autonomous vehicles are gaining momentum in the CPS community. Although promising a breakthrough in terms of traffic optimization and regulation, these vehicles will be unable to fulfil their potential without ensuring safety of passengers and surroundings. Besides the physical environment these vehicles operate in and in which unanticipated events may hamper their operation at any time, these vehicles are also prone to cyber attacks, potential communication infrastructure and electronic devices issues [57]. For designers, it is then about handling the behavior of these vehicles from an holistic point of view rather than considering each part of the system separately [58],[59]. Thus, by modeling the expected behavior of the system taken as a whole rather than considering its internals, the proposed approach is coherent with the holistic point of view.

This section aims at providing a possible application of the method developed throughout the paper to the domain of autonomous vehicles. The proposed application consists in providing autonomous vehicles designers with a bench-marking tool used for assessing the effectiveness of the controllers of these vehicles. Without claiming to be exhaustive, the solution can be used in addition to existing approaches for autonomous vehicles security and safety. Two scenarios are then considered. The first one is about considering the speed limitations in force in France. Such limitation rules are complex and depend on several factors depicted in Table.9.

The idea here is to prevent the vehicle going beyond the maximum speed allowed taking into account factors depicted in Table.9. The second scenario consists in considering passengers’ well being. The idea here is to prevent rapid accelerations and decelerations of the vehicle especially when meeting humps, roundabout, ”giveway”, etc.

5.1 Methodology

OpenStreetMap (OSM) is a collaborative map of the world. It is a powerful source of information about all types of infrastructure features such as roads, trails, side-walks, etc. Specifically, given GPS longitude and latitude, one can retrieve from the repository all information about the infrastructure around. Our approach is to use this repository so as to gather infrastructure information from GPS data at each time $t$ as the vehicle moves. Besides OSM, we do use a weather web service (api.weatherbit.io) allowing us to retrieve weather information at the date of interest.

Based on these information, we do first describe the Ev-HMM model corresponding to the first scenario. Table.10 provides a summary of the features provided by OSM regarding the speed limitations and the associated speed limit values further used in the model. Besides speed limitations, the weather web service provides us with precipitation values (in mm) and visibility (in km). Speed limitation values and weather information are used as inputs of the Ev-IOHMM model (multivariate). Instant speed of the vehicle is used as output of the Ev-IOHMM model. All the constraints considered, the model contains 11 states. For the sake of visibility, a partial representation of the model is depicted in Fig.7.

The model for the second scenario is built upon the same approach. The inputs of the associated Ev-IOHMM are gathered from some OSM features of interest, i.e, traffic_calming:hump, traffic_calming:choker, highway:stop and highway:give_way, taking value of ”1.0” if the feature is detected around, ”0.0” otherwise. The output of the Ev-IOHMM corresponds to the constraint on the acceleration and deceleration while being in each state. Instant acceleration and deceleration of the vehicle is computed based on the delta of speed and the delta of the distance travelled between two time steps ($accel=\frac{v_{(t+n)}-v_{(t)}}{n}$). The corresponding Ev-HMM model is depicted in Fig.8.

Note 5**.**

In these scenarios, we do assume the behavioral constraints are defined by the designer of the model. For instance, for the first scenario, the maximum speed limits in force are subject to tolerances inherent to radar systems accuracy (generally, in the 5% range) that cannot be retrieved from learning. For the second scenario, tolerances may be adjusted based on users’ feedback or from their preferences.

5.2 Results

To complete the experimentations, a C# library has been developed based on the Matlab functions developed by Philippe Smets [60]. The library has been further extended with combination rules (CRC and DRC [37]) and normalization rules (Dempster [61], Yager [62], and Dubois-Prade [56]). A mobile phone application is further used for recording GPS data from several drivers. The traces recorded are then post-processed in order to add, based on latitude and longitude information, the OSM infrastructure features of interest near the location of the vehicle along with the weather data, based on timestamp. The post processor aims at generating a dataset that can be replayed from a graphical interface built using Node-Red [63] and in which the Ev-IOHMM models are instantiated for evaluation (see Fig.9 and Fig.10). The degree of effectiveness is then assessed using the Ev-IOHMM models described in Fig.7 and Fig.8.

{comment}

Results are provided in Fig.12. It is worth noting that the proposed approach is not predictive, i.e. the degree of effectiveness is computed based on the last $T$ observations. The length $T$ depends on the sampling rate of the sensors, the Ev-IOHMM model and the physical process whose evolution is observed. By applying a sliding window on the observations, once the window has been filled up with $T$ observations, one can get the computation of the degree of effectiveness performed upon each new observation. Results obtained may help designers of autonomous vehicles to benchmark the controllers for different situations.

6 Conclusion and perspectives

Cyber-Physical Systems (CPS) are computing systems whose purposes are achieved from interactions with the physical world by means of transducers (sensors and actuators). These systems pose new challenges in terms of dependability, the evolution of the physical systems they control being affected by unanticipated physical processes over which they have no control and which may potentially hamper the achievement of their purposes. It is now recognized that designers of such systems can no longer lean, at design time, on comprehensive and reliable models for anticipating and removing faults that may arise once these systems are deployed. Instead, they have to be monitored in vivo and quantitatively evaluated for effectiveness throughout their life cycle.

In this paper, we formally defined and generalized the notion of effectiveness in the context of the measure and viability theories. We further detailed the mathematical properties the measure has to comply with in the context of assessing CPS effectiveness. The measure is further developed in the Transferable Belief Model (TBM) framework, an elaboration of the Dempster-Shafer Theory (DST) of evidence. The proposed approach is intended to have several applications in the context of benchmarking and assessment of Quality of Experience (QoE)[64]:

• –

The measure can be used as a benchmarking tool. For instance, let us consider the case of autonomous driving. One might be interested in comparing algorithms used to control autonomous vehicles according to the highway code. As a future work, we do plan to apply the proposed approach to this use-case based on the UAH-DriveSet [65].

• –

Assuming the expected evolution of the physical system being driven by the CPS is specified by the users (e.g. through end-user programming [66]), the degree of effectiveness might give a direct insight on users’ satisfaction, i.e. QoE as an assessment of the human satisfaction when interacting with technology and business entities in a particular context.

The proposed approach may also provide an added value in self-adaptive systems:

• –

In the context of the Internet of Things (IoT), many physical devices now expose services available to ubiquitous computing systems leveraging them for composing the so-called ambient applications (e.g. smart-home, smart-city, etc.). The question then arises for these systems of how to select the relevant services. Current approaches rely on semantic annotations used to formally describe the services [67][68]. While this approach is relevant, (1) semantic annotations are pure models, agnostic to the target operational environment, (2) the behavior of a composed application cannot be inferred solely given the individual behavior of the services it is composed with. Hence, observing the concrete behavior of these systems and providing them with a feedback through the measure of effectiveness would help them selecting more appropriate services over time.

• –

More generally, self-adaptive systems pose new challenges in term of assurance, i.e. the ability to provide evidence that these systems satisfy their behavioral requirements, irrespective of the adaptations over time [69]. One may envision leveraging the assessment proposed in this paper within a feedback loop providing CPS with self-awareness capability allowing them to react towards any deviation.

However, although promising, the proposed approach suffers from limitations :

• –

The approach is not predictive, the measure is computed based on past events. This could be a problem for safety critical CPS for which an immediate response is required,

• –

As such, the Ev-IOHMM cannot manage temporal constraints which are of importance in the CPS context. Following what has been done on Hidden Semi-Markov Models (HSMM)[70], we do plan to develop the Ev-IOHSMM where temporal constraints are used to specify the maximum amount of time allowed to switch from one state to another one or to specify the maximum time one is allowed to stay in a particular state,

• –

The Ev-IOHMM is memory greedy, it implies $2^{|\Omega|}\times 2^{|\Omega|}$ elements in the state-transition matrix $A$. For instance, for the first scenario described in section 5, the model of the expected behavior contains 11 states leading a 2048$\times$2048 state-transition matrix. The complexity in time of the forward algorithm described in 4.3.3 is $\mathcal{O}(2^{2|\Omega|}T)$ when using commonalities $q^{\Omega}$. By using plausibilities $pl^{\Omega}$, the complexity is $\mathcal{O}(2^{|\Omega|}T)$ [28]. As a recommendation, when $|\Omega|$ is large, it is preferable to avoid matrix calculus and use the binary format as means to encode focal elements [71],

• –

Finally, it is assumed that sensors required for measuring the effectiveness are available. First, it might not be the case and one needs to assess cost/benefits of adding the required sensors. Second, it might be the case but with sensors not as relevant as desired (for instance, a luminosity sensor is made available but not exactly at the desired location). In that case, one can determine some contextual discounting factors [72], but the challenge remains on the determination of the discounting factor values.

7 Acknowledgment

This work has been supported by GFI Informatique, Innovation group, Saint-Ouen, France. The authors wish to thank the reviewers for their valuable suggestions that greatly helped improve the quality of this paper.

The authors are also thankful to Emmanuel Ramasso (emmanuel.ramasso@femto-st.fr) for his help in better understanding some specific points of the TBM theory.