Probabilistic QoS-aware Placement of VNF chains at the Edge

TL;DR

This paper introduces EdgeUsher, a declarative approach that optimally places VNF chains in Cloud-Edge environments by considering probabilistic infrastructure variations to meet diverse deployment and communication requirements.

Contribution

EdgeUsher is the first solution to incorporate probabilistic models for dynamic infrastructure variations in VNF placement at the Edge.

Findings

Successfully determines all eligible placements satisfying requirements.

Models infrastructure variations using probability distributions.

Ensures placement robustness against dynamic changes.

Abstract

Deploying IoT-enabled Virtual Network Function (VNF) chains to Cloud-Edge infrastructures requires determining a placement for each VNF that satisfies all set deployment requirements as well as a software-defined routing of traffic flows between consecutive functions that meets all set communication requirements. In this article, we present a declarative solution, EdgeUsher, to the problem of how to best place VNF chains to Cloud-Edge infrastructures. EdgeUsher can determine all eligible placements for a set of VNF chains to a Cloud-Edge infrastructure so to satisfy all of their hardware, IoT, security, bandwidth, and latency requirements. It exploits probability distributions to model the dynamic variations in the available Cloud-Edge infrastructure, and to assess output eligible placements against those variations.

Equations

$$p(q)=\sum _{\begin{array}{c}W\in \Omega (q)\end{array}}\prod _{f\in W}p(f)$$

Full text

**Keywords. ** Edge computing, IoT, VNF chain placement, NFV, SDN, probabilistic logic programming.

1. 1 Introduction

2. 2 VNF Chain Placement: Problem Statement

3. 3 EdgeUsher Methodology

4. 3.1 Matching a VNF to an Infrastructure Node

5. 3.1.1 Hardware Requirements

6. 3.1.2 IoT Requirements

7. 3.1.3 Security Requirements

8. 3.2 Matching a VNF Chain to Infrastructure Nodes

9. 3.3 Routing Traffic Flows

10. 3.4 (Anti-)Affinity Constraints and Partial Solutions

11. 3.5 Complexity Analysis

12. 4 Probabilistic Modelling

13. 4.1 Background: The ProbLog Language

14. 4.2 Probabilistic EdgeUsher

15. 4.3 Complexity Analysis and Heuristics

16. 5 EdgeUsher at Work

17. 5.1 Motivating Example

18. 5.2 Motivating Example: Experiments

19. 6 Related Work

20. 7 Conclusions and Future Work

21. A Code

22. A.1 EdgeUsher Prototype

23. A.2 Proof of Correctness and Termination of EdgeUsher

1 Introduction

New Edge computing [1] infrastructures aim at supporting Internet of Things (IoT) applications, especially when applications must meet stringent Quality of Service (QoS) requirements (e.g. latency, bandwidth, security) or handle large amounts of data. To achieve this goal, such new distributed infrastructures rely on computing capabilities which are closer to the edge of the Internet and to where data is produced and consumed (e.g. personal devices, access-points, smart network gateways, base stations, switches, and micro-datacentres). The fruitful interplay between Cloud and Edge resources aims at realising a Cloud-IoT continuum [36], also commonly known as Fog computing [45].

Meanwhile, the ongoing evolution of network technologies, namely Software-Defined Networking (SDN) [30] and Network Function Virtualisation (NFV) [27], is targeting a more effective usage of network resources for containing deployment and operational costs while coping with dynamic traffic demands, including requirements for delivering customised IoT-enabled services [4].

On one hand, SDN decouples network control plane functions (e.g., routing decisions) from data plane functions (i.e., actual packet forwarding). The SDN control plane can leverage a logically centralised management view of network resources to make informed control decisions, and the data plane API of network devices to fully program packet forwarding. Thus, SDN permits defining policies for more agile and cost-effective network operations, e.g. on-demand and QoS-aware traffic flow management, and optimised resource sharing/partitioning.

On the other hand, NFV permits realising complex network services as chains of Virtual Network Functions (VNFs), having each function (e.g. firewall, NAT, video optimiser) implemented as a software application that can be installed and run on commodity hardware, virtual machines or containers. Each VNF can be seen as an independently deployable application service, which can be chained (viz. composed) with others so to apply their functionalities over data flows steered through them by means of SDN-enabled programmable forwarding devices.

In this context, to fully release the potential of combining Edge and Cloud computing paradigms with new generation networks, novel models and methodologies should be devised in support of decision making when deploying VNF chains that enable IoT applications [3]. The problem of placing VNF chains consists of jointly determining:

suitable VNF placements over the available infrastructure (viz. mapping from each VNF to the node that will host it at run time), so to guarantee fulfilling all IoT (i.e., sensors and actuators), hardware and security requirements of each virtual service function, and

• -

suitable routing of traffic flows from one function to the other (viz. between the nodes that will host them at run time), so to satisfy end-to-end bandwidth and latency constraints.

Both decisions concern what is known as VNF placement [2], or VNF chaining and embedding [3], and the corresponding problems are NP-hard [40]. In order to solve these problems in Edge computing scenarios, new peculiarities of Edge infrastructures should be considered that were not considered in Cloud-only settings.

First and foremost, the edge of the Internet is characterised by the presence of resource-constrained (sometimes battery-powered) and heterogeneously capable devices, which communicate via wired and wireless communication technologies. This leads to a potentially high uncertainty and to variations in the available infrastructure for what concerns both the availability of resources (e.g., due to node workload variations) and the QoS of the communication links (e.g., due to traffic variations).

Second, as an extension to the Cloud, Edge computing will inherit from it many security threats, while including its peculiar ones [13]. On one hand, the number of security enforcement points will increase by allowing local processing of private data closer to the IoT sources. On the other hand, new infrastructures will have to face brand new threats for what concerns the physical vulnerability of devices. Indeed, VNF deployments to Edge infrastructures will include accessible (Edge or IoT) devices that may be easily hacked, stolen or broken by malicious users [31]. As highlighted by [8], most of the existing work on VNF placement in Cloud scenarios, as well as preliminary work targeting Edge computing, solved the problem only considering static infrastructure conditions. For what concerns security aspects, to the best of our knowledge, only a few works [14, 10, 39] have been proposed that consider them when deciding on VNF placement and SDN-enabled routing in Cloud-Edge scenarios [13].

In this article, we present a simple, yet general, probabilistic declarative methodology and a (heuristic) backtracking strategy to model and solve the VNF placement problem in dynamic Cloud-Edge computing scenarios, while considering hardware, IoT, security, bandwidth and end-to-end latency requirements of the VNF chain to be deployed. The methodology has been open-sourced by means of the probabilistic logic programming language ProbLog [9] in the prototype EdgeUsher 111Available at: https://github.com/di-unipi-socc/EdgeUsher and also allows to easily specifying and considering placement constraints (i.e. affinity and anti-affinity among functions). The main novel contribution of this work, exploiting a probabilistic declarative description of the VNF chain placement problem, is in that EdgeUsher permits determining VNF chain placements that are likely to ensure high QoS guarantees, security and service reliability over dynamic Cloud-Edge infrastructures that will be needed to achieve the ultimate NFV and SDN vision [22].

Since it follows a declarative implementation, EdgeUsher is more concise, easier to understand, modify and maintain with respect to procedural solutions, and it shows a high level of flexibility, being extensible and capable of accommodating the possibly evolving needs of Cloud-IoT scenarios. Besides, EdgeUsher is intrinsically explainable as it derives proofs for input user queries by relying on state-of-the-art resolution engines, and it can be easily extended to justify why a certain management decision was taken at runtime in the spirit of explainable AI .

The rest of this article is organised as follows. After formulating the VNF chain placement problem and highlighting the main features of the proposed solutions (Sect. 2), the Prolog implementation of EdgeUsher is incrementally described by means of a series of examples (Sect. 3). Then, the probabilistic heuristic version of EdgeUsher – exploiting ProbLog capabilities – is described (Sect. 4) and shown at work over a lifelike motivating example (Sect. 5). A discussion of related work (Sect. 6) and some lines for future work (Sect. 7) conclude the article222Appendix A lists the whole EdgeUsher code and sketches a proof of termination and correctness of the proposed solving strategies..

2 VNF Chain Placement: Problem Statement

As aforementioned, the joint adoption of NFV and SDN technologies is considered by many as a promising approach to support next-gen IoT applications in Edge and Fog environments (Massonet et al. 2017; Farris et al. 2019). Indeed, those networking technologies are expected to enable the flexible matching of diverse IoT traffic requirements, ranging from low-latency and deployment costs minimisation (Wang et al. 2018; Leivadeas et al. 2019) to security mechanisms coping with threats in dynamic distributed environments (Farris et al. 2019; Puliafito et al. 2019).

In this context, our work aims at contributing to solve the problem of placing VNF chains into VNF- and SDN-enabled Cloud-Edge infrastructures. Such a problem can be generally stated as follows:

Let $C$ be a VNF chain with a set of deployment requirements $R_{D}$ on its composing VNFs and a set of communication requirements $R_{C}$ to suitably support communication flows between VNFs, and let $I$ be a distributed Cloud-Edge infrastructure made of computational nodes and communication links.

Eligible solutions to the VNF chain placement problem are joint mappings:

(i)

from each VNF of $C$ to some infrastructure node in $I$, meeting all requirements in $R_{D}$, and

(ii)

from each communication flow of $C$ to a valid routing path across the links of $I$, connecting consecutive VNFs mapped as per (i), and meeting all requirements in $R_{C}$.

Eligible solutions can include nodes hosting multiple VNFs and links supporting multiple communication flows, if the involved nodes and link capacities can meet cumulative requirements of the VNFs and flows they have to support as per (i) and (ii), respectively.

This work tackles the described problem following a probabilistic declarative methodology, based on the logic programming paradigm. As we will show throughout this article, the proposed EdgeUsher methodology and prototype input:

• –

a description of one (or more) VNF chain(s) along with its (their) hardware, IoT, and security requirements (i.e. $R_{D}$), and minimum bandwidth, maximum end-to-end latency and security requirements between consecutive VNFs in the specified chain(s) (i.e. $R_{C}$), and

• –

a probabilistic description of the corresponding hardware, IoT, security, bandwidth and latency capabilities offered by the available Cloud-Edge infrastructure (i.e. $I$).

Based on those, EdgeUsher outputs a ranking of all eligible solutions for the input instance of the VNF chain placement problem. Solutions include the mapping of each chain VNF to its deployment nodes and the routing of traffic flows between those deployment nodes. The ranking of the eligible solutions considers how likely is a certain placement to satisfy all chain requirements as the infrastructure state (probabilistically) varies.

3 EdgeUsher Methodology

In this section, we incrementally describe our methodology to solve the VNF chain placement problem, by following the declarative implementation 333Prototype and examples code is also available at: https://github.com/di-unipi-socc/EdgeUsher of our Prolog prototype, EdgeUsher. Working increments of the prototype are assessed against small running examples, excerpted from a lifelike motivating example on a university video-surveillance distributed application, which we fully describe– and use to assess EdgeUsher – in Sect. 5. Such examples refer to a video surveillance application (consisting of various service functions) to be deployed on a campus network connecting various buildings and, more specifically, computing nodes with different hardware, IoT and security capabilities.

3.1 Matching a VNF to an Infrastructure Node

3.1.1 Hardware Requirements

In the first place, we consider a single VNF to deployed to a single infrastructure node by matching only its hardware requirements. In this scenario, a VNF can be simply declared as in

⬇

1service(FId, HWReqs).

where FId is an identifier of the considered function and HWReqs is an integer representing the amount of a generic hardware unit needed by the function to run properly.

Specularly, an infrastructure node to match a VNF with can be easily declared as in

⬇

1node(NodeId, HWCaps).

where NodeId is an identifier of the considered node and HWCaps is an integer representing the availability of hardware units at that node.

In these very simple settings, the matching of a VNF with a node that can support its hardware requirements can be simply achieved as in

⬇

1servicePlacement(FId,NId):- service(FId, HWReqs),

2 node(NId, HWCaps),

3 hwReqsOK(HWReqs, HWCaps).

4hwReqsOK(HWReqs, HWCaps) :- HWCaps >= HWReqs.

**Example. ** Consider a service function feature_extr that analyses video frames streamed through it and extracts features for further analyses and requires $5$ hardware units to run properly. It can be declared as

⬇

1service(feature_extr, 5).

Now, consider an infrastructure made of the following two nodes (named after the building they are installed at):

⬇

1node(studentCenter, 8).

2node(briggsHall, 2).

Naturally, querying the predicate servicePlacement(feature_extr, N) will result in the possibility of deploying feature_extr to the studentCenter node, which features $8$ hardware units, but not to the briggsHall node, which only features $2$ hardware units.

$\square$

3.1.2 IoT Requirements

We can extend the representation of a VNF by including its requirements in terms of IoT devices it needs to sense data from or act upon in response to cyber-physical triggers. By assuming that IoT devices can be uniquely identified by a symbolic name, we can now declare a VNF as in

⬇

1service(FId, HWReqs, TReqs).

where TReqs is a list [IoTId1, ..., IoTIdk] of all the identifiers of the IoT devices – sensors and actuators – that need to be reachable by the deployment node of the function identified by FId.

Analogously, we extend the representation of an infrastructure node as in

⬇

1node(NodeId, HWCaps, TCaps).

where TCaps is the list of the identifiers of the IoT devices – sensors and actuators – that the node reaches out directly.

The servicePlacement/2 predicate defined before can be simply extended with a check on the IoT requirements as in

⬇

1servicePlacement(FId,NId):- service(FId, HWReqs, TReqs),

2 node(NId, HWCaps, TCaps),

3 hwReqsOK(HWReqs, HWCaps),

4 thingsReqsOK(TReqs, TCaps).

5% hwReqsOK/2 definition

6thingsReqsOK(TReqs, TCaps):- subset(TReqs, TCaps).

**Example. ** Consider a service function cctv_driver that requires $1$ hardware unit to run properly, and to directly reach a CCTV system identified by video1. It can be declared as

⬇

1service(cctv_driver, 1, [video1]).

Now, consider an infrastructure made of the following two nodes:

⬇

1node(parkingServices, 2, [video1]).

2node(briggsHall, 2, []).

Naturally, querying the predicate servicePlacement(cctv_driver, N) will result in the possibility of deploying cctv_driver to the parkingServices node, which reaches out the required CCTV system and features enough hardware resources.

$\square$

3.1.3 Security Requirements

The logic-based formulation of the VNF placement problem that we are giving permits to naturally express security policies for a given VNF. To this end, we assume that different infrastructure nodes can feature different security capabilities, expressed in terms of a common vocabulary of edge computing security capabilities, as per the taxonomy444The example taxonomy can be changed or extended so to include new security categories and third-level security capabilities as soon as normative security frameworks will get established in Cloud-Edge scenarios. of Fig. 1 [16].

Such a taxonomy can be used to express security policies for a given VNF as either a list or an AND/OR composition of security properties, over a common dictionary.

**Example. ** As an example, a security policy for a function collecting sensitive data at the edge of the Internet, could be expressed as the list

⬇

1[anti_tampering, access_control]

requiring both anti-tampering and access control capabilities to be available, as the deployment node can be easily accessed, stolen or broken by malicious users.

As another example, a video processing functionality that stores and analyses possibly sensitive data for a long time might require enforcing the following AND/OR policy

⬇

1and(access_control, or(obfuscated_storage, encrypted_storage))

which ensures the presence of an access control mechanism and of at least one data storage protection mechanism.

$\square$

Building on top of this, we can now extend the representation of a VNF by including its security requirements as in

⬇

1service(FId, HWReqs, TReqs, SecReqs).

where SecReqs is the security policy associated to the function identified by FId, represented as a list or an AND/OR composition of atomic security policies.

Analogously, we extend the representation of an infrastructure node as in

⬇

1node(NodeId, HWCaps, TCaps, SecCaps).

where SecCaps is the list of security capabilities featured by the node, expressed in terms of a common dictionary.

Finally, the servicePlacement/2 predicate defined before can be simply extended with a check on the security policies as in

⬇

1servicePlacement(FId,NId):- service(FId, HWReqs, TReqs, SecReqs),

2 node(NId, HWCaps, TCaps, SecCaps),

3 hwReqsOK(HWReqs, HWCaps),

4 thingsReqsOK(TReqs, TCaps),

5 secReqsOK(SecReqs, SecCaps).

6

7% hwReqsOK(HwReqs, HwCaps) definition

8% thingsReqsOK(TReqs, TCaps) definition

9

10secReqsOK([],_).

11secReqsOK([SR|SRs], SecCaps) :- subset([SR|SRs], SecCaps).

12secReqsOK(and(P1,P2), SecCaps) :- secReqsOK(P1, SecCaps),

13 secReqsOK(P2, SecCaps).

14secReqsOK(or(P1,P2), SecCaps) :- secReqsOK(P1, SecCaps);

15 secReqsOK(P2, SecCaps).

16secReqsOK(P, SecCaps) :- atom(P), member(P, SecCaps).

**Example. ** Consider a service function cctv_driver that requires $1$ hardware unit to run properly, to directly reach a CCTV system identified by video1, and the presence of either anti-tampering capabilities or an access control mechanism at the deployment node, or both. It can be declared as

⬇

1service(cctv_driver, 1, [video1], or(anti_tampering, access_control)).

Now, consider an infrastructure made of the following two nodes that reach out the same CCTV system (video1) but feature different security capabilities:

⬇

1node(parkingServices, 2, [video1],

2 [authentication, anti_tampering, wireless_security, obfuscated_storage]).

3node(parkingServices2, 4, [video1],

4 [authentication, wireless_security, obfuscated_storage]).

Querying the predicate servicePlacement(cctv_driver, N) will result in the possibility of deploying cctv_driver to the parkingServices node, as parkingServices2 do not feature anti-tampering nor access control capabilities, despite satisfying all VNF requirements on hardware resources and IoT.

$\square$

3.2 Matching a VNF Chain to Infrastructure Nodes

At this stage, it is possible to easily specify chains of virtual network service functions and their hardware, IoT, network QoS and security requirements. Indeed, a VNF chain can be declared as

⬇

1chain(ChainID, ServiceFunctionIDs).

where ChainID uniquely identifies the chain and ServiceFunctionIDs lists the identifiers of all VNF composing the chain.

In these settings, it is then possible to extend and exploit the servicePlacement predicate so to determine the placement of an entire chain, by placing one-by-one all services that compose it. This new behaviour is achieved by the placement/2 predicate that follows

⬇

1placement(Chain, Placement) :-

2 chain(Chain, Services),

3 servicePlacement(Services, Placement, []).

4

5servicePlacement([], [], _).

6servicePlacement([S|Ss], [on(S,N)|P], AllocatedHW) :-

7 service(S, HWReqs, TReqs, SecReqs),

8 node(N, HWCaps, TCaps, SecCaps),

9 HW_Reqs =< HW_Caps,

10 thingReqsOK(TReqs, TCaps),

11 secReqsOK(SecReqs, SecCaps),

12 hwReqsOK(HWReqs, HWCaps, N, AllocatedHW, NewAllocatedHW),

13 servicePlacement(Ss, P, NewAllocatedHW).

14

15% thingReqsOK(TReqs, TCaps) definition

16% secReqsOk(SecReqs, SecCaps) definition

17

18hwReqsOK(HWReqs, _, N, [], [(N, HWReqs)]).

19hwReqsOK(HWReqs, HWCaps, N, [(N,A)|As], [(N,NewA)|As]) :-

20 HWReqs + A =< HWCaps, NewA is A + HWReqs.

21hwReqsOK(HWReqs, HWCaps, N, [(N1,A1)|As], [(N1,A1)|NewAs]) :-

22 N == N1, hwReqsOK(HWReqs, HWCaps, N, As, NewAs).

The new servicePlacement/2 predicate inputs the list of Services in the chain and returns an eligible Placement of them to the available infrastructure. In doing so, not only it checks that hardware (line 9), IoT (line 10) and security requirements (line 11) of each VNF are satisfied but it also checks that cumulative hardware requirements of VNFs mapped to a same infrastructure node N do not exceed the overall capacity of the node. To this end, the Prolog program relies on an extended version of the hwReqsOK predicate (line 12, lines 18–22) to update the accumulator list AllocatedHW added as a third parameter in the servicePlacement/3 predicate to keep track of the hardware resources as per the placement being built.

**Example. ** As an example, consider a simple chain made of three VNFs that streams video footage from a CCTV system towards a feature extraction service function capable of identifying events of interests such as unauthorised vehicles access or fire and sending them to a lightweight analytics function for more accurate pattern recognition. Such a chain can be declared as in

⬇

1chain(ucdavis_cctv, [cctv_driver, feature_extr, lw_analytics]).

2service(cctv_driver, 1,[video1],

3 or(anti_tampering,access_control)).

4service(feature_extr, 3,[],

5 and(access_control,or(obfuscated_storage,encrypted_storage))).

6service(lw_analytics, 5,[],

7 and(access_control,and(host_IDS,or(obfuscated_storage,encrypted_storage)))).

Then, consider an infrastructure declaration with the following three nodes, featuring heterogeneous hardware, IoT and security capabilities:

⬇

1node(parkingServices, 1, [video1], [authentication, anti_tampering,wireless_security,obfuscated_storage]).

2node(westEntry, 1, [], [authentication, anti_tampering,wireless_security,obfuscated_storage]).

3node(lifeSciences, 4, [video4], [access_logs, authentication, access_control, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]).

4node(firePolice, 8, [video2, alarm1], [access_logs, access_control, authentication, backup,resource_monitoring, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]).

Querying the predicate placement(cctv_driver, P) will output the following placements

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

placement(ucdavis_cctv,[on(cctv_driver,parkingServices),                        on(feature_extr,firePolice),                        on(lw_analytics,firePolice)])placement(ucdavis_cctv,[on(cctv_driver,parkingServices),                        on(feature_extr,lifeSciences),                        on(lw_analytics,firePolice)])

where cctv_driver is always placed on the parkingServices so to reach out the required CCTV system, while feature_extr and lw_analytics can be placed either both on the firePolice node (which satisfies their cumulative hardware requirements of $8$ units), or on the lifeSciences and firePolice nodes, respectively.

$\square$

3.3 Routing Traffic Flows

As a last step to complete the EdgeUsher prototype, we consider QoS requirements related to bandwidth allocation and end-to-end latency along a VNF chain. To this end, we extend the representation of a service with the information on its processing time as in

⬇

1service(FId, TProc, HWReqs, IoTReqs, SecReqs).

where TProc is the average time – expressed in milliseconds – elapsed between the instant an input is received by function FId and the instant the corresponding output is ready to be transmitted to the next function in the chain.

We also extend the representation of a chain so to include the possibility of specifying bandwidth requirements as directed traffic flows between couple of functions, as in

⬇

1flow(FId1, FId2, BwReq).

where FId1 and FId2 are two VNF identifiers and BWReq is the bandwidth to be allocated via SDN directives along the path that connects their deployment nodes.

Then, we permit specifying constraints on maximum tolerated latency for (directed) service paths crossing the functions F1 $\rightarrow$ F2 $\rightarrow\cdots\rightarrow$ FN as

⬇

1maxLatency([F1, F2, ..., FN], LatReq).

where LatReq is the end-to-end latency (in ms) not to be exceeded, summing up network and function processing delays.

Finally, a (point-to-point or end-to-end) link555EdgeUsher also permits specifying asymmetric links, for which upload and download QoS differ (e.g., like xDSL or 3/4G). connecting NodeA to NodeB which is available in the considered infrastructure can be declared as

⬇

1link(NodeA, NodeB, Latency, Bandwidth).

where Latency is the latency experienced over the link (in ms) and Bandwidth is the transmission capacity it offers (in Mbps).

The definition of the placement predicate illustrated before can be now extended so to check whether, for a given a Placement output by the service placement step, it is possible to determine eligible Routes for the chain traffic flows across function services. This is done by the flowPlacement(Placement, ServiceRoutes) predicate as in

⬇

1placement(Chain, Placement, Routes) :-

2 chain(Chain, Services),

3 servicePlacement(Services, Placement, []),

4 flowPlacement(Placement, Routes).

5

6% servicePlacement(S, P, AllHw) definition

7

8flowPlacement(Placement, ServiceRoutes) :-

9 findall(flow(S1, S2, Br), flow(S1, S2, Br), ServiceFlows),

10 flowPlacement(ServiceFlows, Placement, [], ServiceRoutes, [], S2S_latencies),

11 maxLatency(LChain, RequiredLatency),

12 latencyOK(LChain, RequiredLatency, S2S_latencies).

13

14flowPlacement([], _, SRs, SRs, Lats, Lats).

15flowPlacement([flow(S1, S2, _)|SFs], P, SRs, NewSRs, Lats, NewLats) :-

16 subset([on(S1,N), on(S2,N)], P),

17 flowPlacement(SFs, P, SRs, NewSRs, [(S1,S2,0)|Lats], NewLats).

18flowPlacement([flow(S1, S2, Br)|SFs], P, SRs, NewSRs, Lats, NewLats) :-

19 subset([on(S1,N1), on(S2,N2)], P), N1 == N2,

20 path(N1, N2, 2, [], Path, 0, Lat),

21 update(Path, Br, S1, S2, SRs, SR2s),

22 flowPlacement(SFs, P, SR2s, NewSRs, [(S1,S2,Lat)|Lats], NewLats).

23

24path(N1, N2, Radius, Path, [(N1, N2, Bf)|Path], Lat, NewLat) :-

25 Radius > 0, link(N1, N2, Lf, Bf), NewLat is Lat + Lf.

26path(N1, N2, Radius, Path, NewPath, Lat, NewLat) :-

27 Radius > 0, link(N1, N12, Lf, Bf), N12 == N2, + member((N12,,,_), Path),

28 NewRadius is Radius-1, Lat2 is Lat + Lf,

29 path(N12, N2, NewRadius, [(N1, N12, Bf)|Path], NewPath, Lat2, NewLat).

30

31update([],,,_,SRs,SRs).

32update([(N1, N2, Bf)|Path], Br, S1, S2, SRs, NewSRs) :-

33 updateOne((N1, N2, Bf), Br, S1, S2, SRs, SR2s),

34 update(Path, Br, S1, S2, SR2s, NewSRs).

35

36updateOne((N1, N2, Bf), Br, S1, S2, [], [(N1, N2, Br, [(S1,S2)])]) :-

37 Br =< Bf.

38updateOne((N1, N2, Bf), Br, S1, S2, [(N1, N2, Bass, S2Ss)|SR], [(N1, N2, NewBa, [(S1,S2)|S2Ss])|SR]) :-

39 Br =< Bf-Bass, NewBa is Br+Bass.

40updateOne((N1, N2, Bf), Br, S1, S2, [(X, Y, Bass, S2Ss)|SR], [(X, Y, Bass, S2Ss)|NewSR]) :-

41 (N1 == X; N2 == Y),

42 updateOne((N1, N2, Bf), Br, S1, S2, SR, NewSR).

43

44latencyOK(LChain, RequiredLatency, S2S_latencies) :-

45 chainLatency(LChain, S2S_latencies, 0, ChainLatency),

46 ChainLatency =< RequiredLatency.

47

48chainLatency([S], _, Latency, NewLatency) :-

49 service(S, S_Service_Time, _, _, _),

50 NewLatency is Latency + S_Service_Time.

51chainLatency([S1,S2|LChain], S2S_latencies, Latency, NewLatency) :-

52 member((S1,S2,Lf), S2S_latencies),

53 service(S1, S1_Service_Time, _, _, _),

54 Latency2 is Latency+S1_Service_Time+Lf,

55 chainLatency([S2|LChain], S2S_latencies, Latency2, NewLatency).

The program above first checks bandwidth requirements (line 10) and, afterwards, latency requirements (line 11–12). First, a routing satisfying bandwidth constraints is determined by the predicate flowPlacement (line 10) which holds if:

• –

the services S1 and S2 in between which a flow is established have been placed onto the same node N (lines 15–17), or

• –

the services S1 and S2 in between which the flow is established have been placed onto different nodes N1 and N2 and there exists a path in between those nodes that supports the bandwidth requirement of the flow (lines 18–22).

The path(N1, N2, Radius, [], Path, 0, NewLat) predicate determines an acyclic Path of length at most Radius (i.e., maximum hop number) in between N1 and N2, which features latency Lat (line 20). A path is either a direct infrastructure link between N1 and N2 (lines 24–25), or a route of links that connect them (lines 26–29). It is worth noting that even when setting the value of Radius to low values $K$ (i.e., $2-3$) the found routing will actually be able to spread a chain of length $L$ over a path of length $K\times L$, thus extending the chain potential reach. Naturally, it is possible to relax the constraint on the Radius – incurring in longer execution times – by setting Radius to values larger than the default one (viz. $2$) at line $20$.

After a path is found, update checks if the bandwidth requirements of the considered flow can be supported by such path (lines 31–34). Similarly to hardware allocation, a list of elements of the form (N1, N2, Bf) is maintained to keep track of the bandwidth Bf allocated on each link along a certain path and to check whether more flows mapped onto the same link do not exceed its capacity. Particularly, updateOne scans the list of links along a path and checks such requirements by accumulating the bandwidth consumed by all flows routed onto the same link (lines 36–42).

Finally, latencyOK holds if the chain latency – which is computed by summing the functions processing times of the traversed functions with the latency of the chosen path (lines 48–55) – is less than or equal to the one required by the specified maxLatency requirement.

**Example. ** As an example, consider the chain of the previous example, extended with the following requirements on traffic flows and end-to-end latency

⬇

1flow(cctv_driver, feature_extr, 15).

2flow(feature_extr, lw_analytics, 8).

3

4maxLatency([cctv_driver, feature_extr, lw_analytics], 50).

Then, consider the previous infrastructure declaration completed with the following links:

⬇

1link(parkingServices, westEntry, 15, 70).

2link(westEntry, parkingServices, 15, 70).

3

4link(parkingServices, lifeSciences, 15, 70).

5link(lifeSciences, parkingServices, 15, 70).

6

7link(westEntry, firePolice, 15, 70).

8link(firePolice, westEntry, 15, 70).

Querying the predicate placement(cctv_driver, P, R) will output the following placement and associated routing directives

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

placement(ucdavis_cctv,        [on(cctv_driver,parkingServices),         on(feature_extr,firePolice),         on(lw_analytics,firePolice)],        [(westEntry, firePolice, 15, [(cctv_driver, feature_extr)]),         (parkingServices, westEntry, 15, [(cctv_driver, feature_extr)])])

where cctv_driver is placed on the parkingServices and feature_extr and lw_analytics can only be placed on the firePolice node. Indeed, the previously determined placement of feature_extr to the lifeSciences node is not eligible anymore as the path that connects the lifeSciences node to the firePolice node cannot support the end-to-end latency of $50$ ms. It is worth noting that the traffic flow of $15$ Mbps between cctv_driver and feature_extr follows a path passing through westEntry, which connects parkingServices to firePolice. On the other hand, as feature_extr and lw_analytics are mapped onto the same node, no routing is output for the $8$ Mbps traffic flow in between them.

$\square$

3.4 (Anti-)Affinity Constraints and Partial Solutions

It is worth noting that EdgeUsher allows users to easily specify placement constraints in the form of function affinity or anti-affinity requirements among functions. Affinity consists in placing two or more functions in the same physical node, thus reducing latency and communications costs between VNFs, while anti-affinity prevents two or more VNFs from sharing the same resources [33]. The possibility to add affinity and anti-affinity constraints is useful since it allows specifying deployment location requirements needed for performance, economic, resilience, legislative and privacy issues [5].

In the case of affinity constraints, the user can force the mapping of two (or more) functions to the same node, as for instance in the query

⬇

1placement(Chain, [on(F1,N1), on(F2,N2), on(F3,N2)], Routes).

stating that F2 and F3 must be mapped on a same node N2. Analogously, anti-affinity constraints can be specified by queries of the form:

⬇

1placement(Chain, [on(F1,N1), on(F2,N2), on(F3,N3)], Routes), N2 == N3).

imposing that F2 and F3 must be mapped on two different nodes N2 and N3.

Finally, users can specify partial deployments and/or routes, and use EdgeUsher to complete them. This is useful to quickly determine on-demand re-configurations of part of a chain in case this is affected by infrastructure failures or malfunctioning (e.g., crash of a node hosting a function service) without recomputing (and eventually migrating) the whole chain.

3.5 Complexity Analysis

EdgeUsher relies on logic programming backtracking mechanism to determine the eligible VNF placement(s) and traffic routing(s) for a VNF chain to be deployed to an Edge infrastructure.

The worst-case time complexity of servicePlacement is clearly $O(n^{s})$, where $n$ is the number of nodes in the available infrastructure and $s$ is the number of services in the input service chain. The worst-case time complexity of flowPlacement is $O(b^{\tt Radius})$, where $b$ is the average out-degree of nodes in the infrastructure, and in any case it is bounded by $O(n^{\tt Radius})$ (when dealing with fully connected network topologies). These hold both in the case we aim at determining a single eligible placement or routing (and only one exists, and it is the last one found via backtracking), and in the case in which we aim at determining all eligible placements or routings (which naturally requires to fully explore the placement and routing search spaces, independently of the number of existing eligible solutions).

Now, the worst-case case for determining an eligible placement and routing happens when a single eligible solution exists and it corresponds to combining the last found placement with the last found routing. In such a case, the combination of servicePlacement with flowPlacement incurs in a time complexity of $O(n^{s}\times n^{\tt Radius})=O(n^{s+{\tt Radius}})$. As per the above considerations, the worst-case time complexity of the approach described until now is exponential, and so is the exhaustive exploration of the search space.

4 Probabilistic Modelling

In this section, we first recapitulate on the probabilistic logic programming language ProbLog (Sect. 4.1). Then, we illustrate how the probabilistic capabilities of ProbLog permit enhancing the Prolog EdgeUsher prototype so to consider dynamic infrastructure conditions when solving the VNF embedding problem (Sect. 4.2). Besides, we will show how ProbLog meta-reasoning capabilities can be exploited to reduce the exp-time complexity discussed before (Sect. 4.3).

4.1 Background: The ProbLog Language

Probabilistic logic programming extends logic programming by enabling the representation of uncertain information. More specifically, logic programming allows representing relations among entities, while probability theory can model uncertainty over attributes and relations [38]. To implement both the model and the matching strategy we used the ProbLog language [21, 9], a probabilistic extension of Prolog.

Prolog programs are finite sets of rules of the form

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

a :- b1, ... , bn.

stating that a holds when b1 $\wedge\ \cdots\ \wedge$ bn holds, where n$\geq$0 and a, b1, …, bn are atomic literals. Rules with empty conditions (n$=$0) are also called facts.

ProbLog programs are logic programs in which some of the facts are annotated with probabilities. A ProbLog fact, such as

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

p::a.

states that a holds with probability p. Non-annotated facts are assumed to always hold with probability $1$.

Problog also allows to use semicolons to express OR conditions in rules. For instance

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

a :- b1; ... ; bn.

states that a holds when b1 $\vee\ \cdots\ \vee$ bn holds.

Finally, annotated disjunctions, like

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

p1::a1; p2::a2; ...; pK::aK.

state that at most one of the facts a1, …, aK holds with the associated probability666If $T=\sum_{{\tt i}=1}^{\tt K}{\tt pi}<1$, ProbLog assumes the presence of an implicit null choice which states with probability $1-T$ that none of the K options holds..

Each ProbLog program defines a probability distribution over logic programs where a fact p::a. is considered true with probability p and false with probability $1-$p. The ProbLog engine [9] determines the success probability of a query q as the probability that q has a proof, given the distribution over logic programs.

Intuitively, a ProbLog program leverages input probability distributions to analyse all possible Prolog programs (i.e., worlds) that could be generated according to them. Assuming that $\Omega(q)$ is the set of possible worlds $W$ that entail a valid proof for a certain query $q$ (i.e., $\Omega(q)=\{W\mid W\models q\}$), the ProbLog engine computes the probability $p(q)$ that $q$ holds as

[TABLE]

where $f$ are facts within a possible world, and $p(f)$ is the probability they are labelled with.

4.2 Probabilistic EdgeUsher

The usage of ProbLog permits to naturally specify probabilistic profiles of both nodes and links by exploiting annotated disjunctions. Such language constructs permits to capture the intrinsic dynamicity and uncertainty of Edge infrastructures by relying on probability distributions based on historically monitored data.

An infrastructure node can then be declared as in

⬇

1P1::node(NId, HwP1, IoTCapsP1, SecCapsP1);

2P2::node(NId, HWP2, IoTCapsP2, SecCapsP2);

3...;

4Pk::node(NId, HWPk, IoTCapsPk, SecCapsPk).

where $\sum_{i=0}^{\texttt{k}}\texttt{Pi}\leqslant 1$ and Pi is the probability that a particular node configuration (with respect to available hardware, IoT devices, and security capabilities) occurs.

Analogously, links between nodes can be specified as in

⬇

1P1::link(N1, N2, LatP1, BwP1);

2P2::link(N1, N2, LatP2, BwP2);

3...;

4Pk::link(N1, N2, LatPk, BwPk)

where $\sum_{i=0}^{\texttt{k}}\texttt{Pi}\leqslant 1$ and Pi is the probability that a particular link QoS configuration (with respect to end-to-end latency and available bandwidth) occurs.

It is worth noting that three different types of facts (and any mixture of them) can constitute the input infrastructure description in the ProbLog version of EdgeUsher. Indeed, a user can exploit:

• (a)

non-probabilistic facts when using real-time monitoring or averaged monitoring data (i.e., avoiding annotating facts), as shown in Sect. 3,

• (b)

a single-probability facts that are just annotated with an indication of their reliability (e.g., 0.99::(cloudX, 30, [], [firewall]).), or

• (c)

a fully probabilistic facts annotated with complete probability distributions describing the infrastructure dynamics based on aggregate historical monitoring data.

Naturally, running EdgeUsher in ProbLog  with probabilistic input enhances output precision via ranking eligible VNF placements and traffic routings according to how well they are expected to satisfy the chain requirements as the infrastructure state (probabilistically) varies.

**Example. ** As an example, consider the chain of the previous example, to be matched to the following probabilistic infrastructure description

⬇

10.2::node(parkingServices, 2, [video1], [authentication, anti_tampering,wireless_security,obfuscated_storage]);

20.8::node(parkingServices, 1, [video1], [authentication, anti_tampering,wireless_security,obfuscated_storage]).

3

40.2::node(westEntry, 2, [], [authentication, anti_tampering,wireless_security,obfuscated_storage]);

50.8::node(westEntry, 1, [], [authentication, anti_tampering,wireless_security,obfuscated_storage]).

6

70.2::node(lifeSciences, 8, [video4], [access_logs, authentication, access_control, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]);

80.8::node(lifeSciences, 4, [video4], [access_logs, authentication, access_control, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]).

9

100.2::node(firePolice, 16, [video2, alarm1], [access_logs, access_control, authentication, backup, resource_monitoring, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]);

110.8::node(firePolice, 8, [video2, alarm1], [access_logs, access_control, authentication, backup,resource_monitoring, iot_data_encryption, firewall, host_IDS, pki, wireless_security, encrypted_storage]).

12

130.98::link(parkingServices, westEntry, 15, 70).

140.98::link(westEntry, parkingServices, 15, 70).

150.98::link(parkingServices, lifeSciences, 15, 70).

160.98::link(lifeSciences, parkingServices, 15, 70).

170.98::link(westEntry, firePolice, 15, 70).

180.98::link(firePolice, westEntry, 15, 70).

where nodes feature different hardware capabilities777For the sake of the example, we limit the probabilistic variations to hardware capabilities. Of course, EdgeUsher allows to associate to each probability a different node configuration also for what concerns IoT devices, that can temporarily fail or disconnect, and security countermeasures, that could be activated/deactivated depending on contextual factors, e.g. to save residual battery power. according to a probability distribution and links are assumed to be wireless links with an associated reliability of $98\%$.

Querying the predicate placement(cctv_driver, P, R) will output the following placements, the associated routing directives and a probability value ranking them as per how well they can satisfy all chain requirements:

\KV@do

,fontfamily=courier, fontsize=, frame=single, framesep=1mm, framerule=0.1pt, rulecolor=,,

placement(ucdavis_cctv, [on(cctv_driver,parkingServices),  on(feature_extr,firePolice),  on(lw_analytics,firePolice)],  [(westEntry, firePolice, 15, [(cctv_driver, feature_extr)]),   (parkingServices, westEntry, 15, [(cctv_driver, feature_extr)])]): 0.9604placement(ucdavis_cctv, [on(cctv_driver,parkingServices),  on(feature_extr,lifeSciences),  on(lw_analytics,lifeSciences)],  [(parkingServices, lifeSciences, 15, [(cctv_driver, feature_extr)])]): 0.196

where cctv_driver is placed on the parkingServices and feature_extr and lw_analy- tics can be placed both on the firePolice node or on the lifeSciences node, that now has a $20\%$ probability of featuring enough resources to host them. The best choice for the service chain deployer is still represented by the first output VNF chain placement and routing. It is worth noting that additional outputs with lower probability values could be kept as a backup deployments, when their associated probabilities exceed a desired threshold.

$\square$

4.3 Complexity Analysis and Heuristics

As per the considerations we have made in Sect. 3.5, the algorithmic time complexity of the approach described until now is exponential, i.e. $O(n^{s+{\tt Radius}})$. Besides, the probabilistic reasoning on disjoint clauses requires running EdgeUsher over an exponential number of possible worlds with a worst-case time complexity of $O(k^{n+m})$ where $k$ is the average number of disjoint clauses per each of the $n+m$ facts denoting the $n$ infrastructure nodes and the $m$ infrastructure links, respectively. For instance, in the case $k=2$, the overall complexity increases to $O(2^{n+m}\times n^{s+{\tt Radius}})$. Naturally, such time complexity becomes unbearable for very large infrastructures and for long service chains. Hence, we extended the prototype with a heuristic based on the probabilistic modelling we gave for the infrastructure capabilities.

The heuristic version of EdgeUsher allows users to specify two threshold values that are used to prune the search space whenever the probability of satisfying the chain hardware or bandwidth requirements, respectively, falls below them. Such pruning is implemented via the ProbLog subquerying system which is used to evaluate the probabilities of the servicePlacement (i.e. PHw) and the path (i.e. PQoS) goals during the search for eligible placements, and to check them against the user-specified thresholds (i.e. THw and THw). As soon as a candidate solution being built by the ProbLog engine is associated to a probability lower than the user-set thresholds (and, thus, will not suitably satisfy the VNF chain requirements), the heuristic version of EdgeUsher stops searching along the corresponding path, hence reducing search times.

Particularly, the illustrated meta-reasoning behaviour is implemented by simple extensions of the placement and flowPlacement predicates, as in

⬇

1placement(Chain, Placement, ServiceRoutes, THw, TQoS) :-

2 chain(Chain, Services),

3 subquery(servicePlacement(Services, Placement), PHw),

4 PHw >= THw, % meta-reasoning on servicePlacement/2

5 flowPlacement(Placement, ServiceRoutes, TQoS).

6

7flowPlacement(Placement, ServiceRoutes, TQoS) :-

8 findall(flow(S1, S2, Br), flow(S1, S2, Br), ServiceFlows),

9 flowPlacement(ServiceFlows, Placement, [], ServiceRoutes, [], S2S_latencies, TQoS),

10 maxLatency(LChain, RequiredLatency),

11 latencyOK(LChain, RequiredLatency, S2S_latencies).

12

13flowPlacement([],_,SRs,SRs,Lats,Lats,TQoS).

14flowPlacement([flow(S1, S2, _)|SFs],P,SRs,NewSRs,Lats,NewLats,TQoS) :-

15 subset([on(S1,N), on(S2,N)], P),

16 flowPlacement(SFs,P,SRs,NewSRs,[(S1,S2,0)|Lats],NewLats,TQoS).

17flowPlacement([flow(S1, S2, Br)|SFs],P,SRs,NewSRs,Lats,NewLats,TQoS) :-

18 subset([on(S1,N1), on(S2,N2)], P),

19 N1 == N2,

20 subquery(path(N1, N2, 2, [], Path, 0, Lat), PQoS),

21 PQoS >= TQoS, % meta-reasoning on path/7

22 update(Path, Br, S1, S2, SRs, SR2s),

23 flowPlacement(SFs,P,SR2s,NewSRs,[(S1,S2,Lat)|Lats],NewLats,TQoS).

The new placement predicate exploits ProbLog built-in subquery/2 to check whether historical variations in the behaviour of deployment nodes might lead to insufficient (hardware, IoT, security) resource availability for a certain placement of the services composing the VNF chain (lines 3–4). Similarly, the heuristic version of flowPlacement checks whether the historical variations in the behaviour of communication links might intolerably affect the routing of traffic flows along a certain path (lines 20–21).

**Example. ** Running the heuristic prototype over the previous example and requiring that both deployment and communication requirements of output placement are met in $90\%$ of the cases, i.e. setting ${\tt THw}={\tt TQoS}=0.9$, we only obtain as a result the first output placement which has an associated probability of $0.9604$.

$\square$

5 EdgeUsher at Work

In this section, we illustrate a lifelike motivating example (Sect. 5.1), and we discuss the EdgeUsher prototype performance and the effectiveness of the proposed heuristics over such motivating example (Sect. 5.2).

5.1 Motivating Example

Hereinafter, we describe a lifelike example to better introduce the VNF placement problem and to highlight some of the related challenges. The example extends those that have been used throughout the paper to illustrate the proposed approach. We consider a portion of the topology of the Edge computing infrastructure deployed at UC Davis, inspired from [32], and sketched in Fig. 2. Such infrastructure is a Wireless-Optical Broadband Access Network (WOBAN) and consists of $10$ heterogeneously capable edge nodes.

We assume that available edge nodes feature either $2$, $4$, $8$ or $16$ hardware units888For the sake of readability, we only consider generic hardware units. Extensions to account for different resource types (e.g. RAM, CPU, storage) are straightforward. and that they are subject to workload variations as per the distributions reported in Fig. 3. For instance, nodes with 2 hardware units are totally free in $20\%$ of the cases, whilst they only have $1$ free hardware unit in the remaining $80\%$. We also assume that different node types feature different security capabilities as reported in Fig. 3, expressed in terms of a common vocabulary of edge computing security capabilities, as per the taxonomy of Fig. 1. Last, but not least, the nodes featuring $16$ GB of memory (viz., the Fire & Police and the Student Centre devices) connect the edge network to a Cloud data centre through the same ISP Node (not shown in the figure).

Analogously, we assume that network links have the bandwidth and latency profiles listed in Fig. 4. For instance, on-campus wireless connections may be not available in $2\%$ of the cases, and feature 70 Mbps bandwidth and 15 ms latency in the remaining $98\%$.

We suppose that a new smart CCTV system has been installed at the Transportation & Parking Services building, and that it continuously captures video footage and streams it to a CCTV System Driver deployed to the edge node which is in physical proximity.

A VNF chain (Fig. 5) must be deployed to support the video surveillance IoT system with a running application. The chain application, when suitably deployed, permits detecting events of interest (e.g., unauthorised access, fire, anomalous behaviour) by analysing video streams and by promptly notifying an alarm system installed at the Fire & Police station on campus. Such a VNF chain includes:

• •

a Feature Extraction service function that applies image processing techniques to isolate potentially interesting video portions, and

• •

a Lightweight Analytics service function that further processes such video portions by performing object recognition, by detecting anomalies or potentially dangerous situations, and by sending appropriate notifications to the Alarm Driver deployed at the Police Station.

To work as expected, the end-to-end latency from the CCTV System Driver to the Alarm Driver must not exceed $150$ ms latency, as shown in Fig. 5. Additionally, for each link between two VNFs a minimum bandwidth requirement is specified, as shown in the figure. The traffic originated by the CCTV system is also collected by a WAN Optimizer service function that improves video data delivery efficiency (e.g., compression) and forwards video data to a Storage service. Complex video analytics are then performed with more relaxed latency constraints by a Video Analytics service function which updates, when needed, the model used by the system to recognise potentially dangerous events. Figure 6 lists the requirements for the deployment of each VNF in terms of hardware units, connection to IoT devices (sensors or actuators) and security policies, along with the expected processing time of each chain function. As a further (soft) requirement, VNF chain deployers at UC Davis would prefer the Video Analytics and Storage service functions be placed on the same node (affinity) to reduce communication costs.

Overall, both the available Cloud-Edge infrastructure and the VNF chain to be deployed on campus can be naturally declared in the input format exploited by EdgeUsher. In fact, deploying the described chain to the infrastructure available at UC Davis implies solving the VNF placement problem, i.e. deciding how to map a VNF graph on top of an infrastructure substrate made of heterogeneous Edge and Cloud nodes and communication links, so that hardware, IoT and end-to-end network QoS requirements are all satisfied.

Furthermore, the infrastructure is a dynamic environment and we assume it being subject to node workload variations and changing network conditions as per the probability distributions (possibly obtained from historical monitoring data [18]) we described in this section. Such changes can indeed affect deployment performance and turn momentarily optimal solutions into bad or unfeasible ones, potentially leading to unsatisfactory application QoS, and application downtime or unavailability.

As we will show in the next section, EdgeUsher methodology permits determining VNF placement (i.e., function mappings and flow routes) and evaluating their performance against probabilistic infrastructure variations in this scenario. In the next section, after discussing solutions to this first VNF placement, we will illustrate how the deployers at UC Davis can exploit our methodology to determine a further placement for the dashed part of the chain in Fig. 5, handling a videostream for a second CCTV system deployed at the Mann Lab, and joining the first chain at the WAN Optimiser service function.

5.2 Motivating Example: Experiments

We started by looking for eligible placements of the VNF chain supporting the CCTV system installed at the Parking Services building. For the purpose of the experiments999The experiments were run on a commodity laptop provided with an Intel Core i5-6200U CPU (2.30GHz) and 8GB of RAM, running Ubuntu 18.04.2 LTS, ProbLog 2.1.0.36 and Python 3.6., we first run the non-heuristic EdgeUsher over three different inputs (of the types (a)–(c) illustrated in Sect. 4.2) for the infrastructure description:

• (a)

a non-probabilistic description that only considers the most probable values of each probability distribution without indicating the associated probabilities (i.e., for both node hardware and link QoS profiles),

• (b)

a single-probability description of the infrastructure that accounts only for the highest probability value of each distribution (i.e., one probability value per each node or link), and

• (c)

a complete fully-probabilistic description of the infrastructure that includes all probability distributions available for nodes and links.

Figure 7 shows an example of the same link fact in the non-probabilistic, single-probability and probabilistic infrastructure descriptions 101010The three input files are available at: https://github.com/di-unipi-socc/EdgeUsher/tree/master/infra.

Fig. 8 shows the obtained results in terms of number of generated eligible placements and computation time111111Timings obtained by averaging results over $15$ executions for each case. needed to obtain those. Fig. 9 shows one of the best placements obtained, which features $98\%$ probability of complying to all hardware, IoT, security, bandwidth and end-to-end latency requirements of the input chain.

It is worth noting that the prototype runs fairly fast on the non-probabilistic121212The non-heuristic prototype can be run also in traditional Prolog environments to determine eligible placements in non-probabilistic infrastructure conditions. In this case, a first placement solution is returned istantaneously. and on the single-probability infrastructure, which do not suffer from the additional combinatorial complexity that ProbLog incurs in, when evaluating (the probability distributions expressed as) annotated disjunctions in the fully-probabilistic infrastructure description. We will use the results obtained by the non-heuristic prototype over the fully-probabilistic description of the Edge infrastructure at UC Davis as a baseline to evaluate the performance of the heuristic prototype.

Thus, we run the heuristic EdgeUsher over the fully-probabilistic description of the edge infrastructure at UC Davis. For the sake of simplicity, we set both thresholds (i.e., the one on node requirements THW and the one on network QoS TQoS) to a value $T$ that was varied during the experiments in the range $[0.1,0.8]$ with a step of $0.1$. Figure 10 shows the obtained results in terms of number of generated eligible placements and execution times10 needed to obtain those.

The results show that the employed heuristics considerably reduces the search space and, thus, the execution time needed to determine eligible VNF placements and routing. Particularly, the fully probabilistic description of the UC Davis infrastructure can be handled in a time which shows a speed-up between $15$ and $214$ with respect to the exhaustive prototype, and still returns a subset of the optimal results. Besides, with thresholds set to $0.8$, EdgeUsher determines $6$ eligible placements for the VNF chain supporting the CCTV system. Four of such placement solutions have a probability of meeting all set requirements of $96\%$, whilst the remaining two of $98\%$. All output solutions fall within the best solutions generated also by the non-heuristic prototype, when it is run over the complete infrastructure description.

We then included an affinity constraint between the Storage and the Video Analytics service functions, and we run the prototype again over the fully-probabilistic input with $T=0.8$. As a result, execution time halved with respect to the execution without such constraint, reaching around 24 seconds. The placement of Fig. 9 is still output as one of the best possible when forcing the affinity constraint. In this case, both the Storage and the Video Analytics are deployed to the available Cloud node. The highlighted links – along with their labels – show the routing path associated to the placement and the bandwidth to be allocated to traffic flows mapped on each infrastructure link. Such piece of information could be actually used to instruct the network (e.g., via SDN controllers) so to allocate a suitable amount of bandwidth to each flow.

Afterwards, assuming the first chain was deployed as in Fig. 9, EdgeUsher was exploited to check whether it was possible to extend the deployment by placing anew the dashed part of the chain for a second CCTV system installed at the Mann Lab. By querying again the heuristic prototype, $7$ new possible VNF placements were obtained in around $22$ s. All output solutions featured a $96\%$ probability of meeting all chain constraints. Four of such solutions placed services as sketched in Fig. 11 (a), the remaining three as in Fig. 11 (b), other routings – which are not shown – were possible. It is worth noting that the deployers might consider using one of the output solutions and keep some of the others as possible backups to guarantee chain functioning in case of device failures or overloading, or in case of network congestion.

6 Related Work

SDN and NFV technologies are gaining increasing interest for their potential benefits in hybrid Cloud-Edge environments [29] and in the IoT [28]. Indeed, the concept of Service Function Chaining (i.e., the ordered interconnection of service functions implemented as VNFs) is expected to enable the offer of added-value services, like virtual reality or tactile Internet applications, over next-generation telecommunication networks [8] and in Multi-access Edge Computing (MEC) scenarios [41, 12]. Hereinafter, we discuss main related work in the area of SDN and NFV technologies applied in the IoT.

An SDN and NFV architecture for IoT network and application management is proposed in [34]. [28] propose an architecture and a prototype implementation of an NFV/SDN framework enabling automated and dynamic network service chaining across Edge (i.e., IoT gateway) and Cloud (i.e., central data center) domains. SDN and NFV are jointly used in [37] to assure service continuity of a video monitoring application deployed over a flying ad-hoc network (FANET) built on a fleet of drones over rural areas. Drones are used as Point of Presence that can host Virtual Network or Application Functions.

The problem of placing VNFs on a physical substrate for realising service chains in a hybrid edge/cloud infrastructure to support IoT applications has only recently emerged. Previous work has focused on network service placement in VNF infrastructure, considering intra and/or interDC networks [35, 25]. A survey on resource allocation strategies for the network services deployment in VNF infrastructures has been provided by [19].

Although converged approaches are emerging for managing NFV, edge and fog computing services [42], traditional VNF placement approaches do not tackle challenges brought by Fog and Edge computing for IoT applications. These challenges include heterogeneity of computing nodes, dynamic changes of network and node conditions that may turn optimal or quasi-optimal solutions into unfeasible ones, and security threats, just to mention the main ones. Recent work in the area of application placement in the Fog have partially begun to tackle these aspects, but open research problems still exist, such as placement approaches accounting for security aspects and dynamic infrastructure variations, as discussed in the review by [6].

Only few works have addressed the problem of placing VNFs in a hybrid environment made of edge and cloud computing nodes. [23] model the problem of SFC placement in hybrid MEC and cloud environment considering location requirements posed by VNFs and targeting minimisation of deployment costs and delays. They propose a Mixed Integer Programming (MIP) formulation of the problem and a sub-optimal approach based on the Tabu Search meta-heuristic.

SFC placement in IoT scenarios, which demand for low-latency response, high-throughput processing and cost effective resource usage, is tackled in [43]. The work proposes a linear programming model and an approximation optimization algorithm to achieve deadline and packet rate guarantees while avoiding resource idleness. However, SFC orchestration is done within the cloud domain and the availability of computing resources at the edge is not considered.

[44] propose a VNF placement algorithm that optimises access latency and service availability in a mixed edge and cloud environment for ultra-Reliable Low-Latency Communications (uRLLC) services. The multi-objective optimization problem is solved by exploiting a Genetic Algorithm metaheuristic, whose achieved performance is compared against an exact algorithm implemented in CPLEX. Although a network service is defined as a set of VNFs, chaining constraints are not considered.

[29] tackle application component placement in NFV-based hybrid Cloud-Edge systems and propose an ILP formulation that represents applications as non-deterministic VNF Forwarding Graphs. Graphs can be built using sequence, parallel, selection and loop substructures and probabilities are used to model selection and loop iterations.

Although all the above mentioned works [23, 43, 44, 29] consider latency requirements (either as minimisation objective or as constraint), none of them accounts for dynamic variations of network status, which instead can influence the extent to which QoS requirements are satisfied in the long run. Neither security aspects are taken into account.

Typically, VNF placement approaches that consider dynamic network conditions either recompute placement [8], enforce scaling and/or migration actions [11, 20] or try to find a solution that is robust against network status variations [7]. [8] formulate the problem of edge VNF placement as an ILP to derive latency-optimal deployments of VNFs. They also define a dynamic scheduler that recomputes placement to account for latency variations on links. This scheduling problem, which consists in selecting the time for placement recalculation so that unnecessary VNF migrations are prevented and latency violations are bound, is solved using optimal stopping theory. While [8] deals with placement of single VNFs and infrastructure dinamicity is modelled only in terms of network latency variation, EdgeUsher handles chains of VNFs and accounts for probabilistic distributions of latency and available bandwidth of links as well as of resource node capacity. In [7] network dynamics are taken into account to find temporal robust placement solutions. The SFC placement is formulated as a Stochastic Resource Allocation problem that exploits both currently observed network information and future variation. However, the work does not tackle latency-aware placement and the network model does not represent variations of neither network latency nor node resources, as our work does. [46] formulate a stochastic programming problem that minimizes the placement cost and aims at achieving high availability application deployments. The problem formulation thus accounts for probabilities of VM, host and link failures, but does not consider latency constraints.

As analysed in [13], IoT environments introduce challenging security threats, ranging from attacks to IoT devices, attacks in IoT-oriented clouds and networks to threats in the application layer, such as vulnerabilities in software, data leakage and phishing. Risks exist in executing VNFs over third-party infrastructures and security and trust criteria have to inform placement decisions [13]. Indeed, the need to consider security issues in virtual network embedding (VNE) and VNF placement problems is gaining increasing interest. A classification of security requirements into node, link and topological requirements to be considered in virtual network embedding problems is provided in [14]. In [10] the problem of virtual network embedding is formulated so to account for standard protection provided by substrate nodes and links (quantitatively referred to as security level). If the level of security is lower than the security demand, the VNE algorithm tries to place security VNFs (e.g., firewall, deep packet inspection, and intrusion detection) to improve the offered security level. Optimal placement of security SFCs is tackled in [39], where the placement problem is formulated including deployment constraints derived from network security patterns.

Figure 12 provides a comparative overview of the discussed related work and highlights how, to the best of our knowledge, this is the first work aiming at addressing VNF chain placement in a hybrid edge/cloud network with latency constraints while accounting for network status variations and security requirements. In addition, while most related works rely on linear programming formulations, we adopt a probabilistic declarative approach. Indeed, declarative approaches have been successfully applied to modelling and reasoning on problems related to distributed systems other than VNF embedding – as for instance in [24] and [26].

Last but not least, our prototype is released as open source software and the experiment data is also made publicly available.

7 Conclusions and Future Work

In this article, we have proposed a logic programming approach for solving the problem of placing VNF chains onto Cloud-Edge infrastructures. To achieve this objective, we followed three main steps:

we gave a concise logic programming formulation (hence, a declarative solution) of the considered VNF chain placement problem, 2. 2.

we extended it with a suitable probabilistic representation of variations in the available infrastructure capabilities to assess the quality of eligible solutions against those, and 3. 3.

we devised a heuristic to ensure scalability of our prototype and suitably high quality of output solutions, by immediately pruning out low quality solutions based on user-specified thresholds for hardware and QoS requirements.

The obtained prototype, EdgeUsher, returns the eligible deployments of VNF chains to a hybrid Cloud-Edge infrastructure that guarantee the fulfilment of a set of placement requirements, namely: hardware, IoT reachability, bandwidth, latency and security policies. Thanks to the declarative approach, additional constraints, such as affinity, anti-affinity or placement into a specific node can be easily expressed. Moreover, EdgeUsher implementation has been fully provided in this paper (around 70 source lines of code) together with some example of usage. Declarative programming also makes EdgeUsher  more flexible and extensible than procedural solutions, what makes it better suited to accommodate the ever-changing needs of Cloud-Edge scenarios.

By leveraging ProbLog, EdgeUsher permits specifying and solving the VNF chain placement problem while considering infrastructure variations, by assuming that probabilistic distributions describing the behaviour of computing nodes and infrastructure links are available. Since probabilistic logic programming is a natural extension of plain logic programming, it was straightforward to also consider dynamic infrastructure conditions by suitably extending (the input of) the Prolog version of EdgeUsher. Such an extension to account for dynamic settings would have required significantly more effort, if implemented by means of other paradigms. It is worth noting that EdgeUsher could also be used to quickly evaluate VNF placements computed by alternative placement algorithms with respect to the infrastructure variability, by calculating their probability of satisfying hardware and QoS requirements. In this work, we discussed and showcased the use of our prototype over a lifelike reference scenario, which we also used to assess and epitomise the performance of EdgeUsher.

Naturally, being the considered problem NP-hard, the worst-case time complexity of our approach is exponential in the size of the input infrastructure. For larger scenarios, we thus envision a hierarchical architecture of clusters of edge nodes (partitioned, for instance, according to administration, application or geographical criteria) where orchestration features are run by one head node, and connected to a few Cloud nodes. We intend to elaborate further on this vision by running EdgeUsher over a domain made by a few clusters of edge nodes and their associated Cloud nodes. However, the potential advantage of the probabilistic approach relies in the provisioning of solutions that are resilient to infrastructure variations over time. Our effort goes towards the direction of determining placements that are more likely to ensure high QoS-guarantees, security, and service reliability against dynamic infrastructure conditions, thus allowing amortising the cost of reasoning over an increased chain life time.

On direction for future work, we plan to comparatively evaluate our approach with state-of-the-art solutions that react to infrastructure changes by computing and executing costly VNF migrations or scaling actions (e.g., [11, 20]), using simulation as well as testbed environments. The setting up of a small-scale testbed is undergoing in our campus network and it will be used to perform tests using probabilistic distributions derived from real monitoring data [17].

EdgeUsher also allows specifying security requirements in terms of logical expressions over security properties. It is worth noting that some security properties can be provided exclusively as hardware capabilities (e.g., anti-tampering) while other ones could be implemented also as software and deployed as VNFs (e.g., firewall). This option opens up to the possibility of adaptively inserting required security VNFs when needed, which we also plan investigating in the near future. Besides, we envision enhancing EdgeUsher with continuous reasoning capabilities [15] to further tame exp-time complexity at runtime, automatic techniques to perform parameter tuning of the heuristic thresholds (e.g. via machine learning), and a user-friendly GUI to ease user interactions with the prototype.

Acknowledgements

This work has been partly supported by the project “DECLWARE: Declarative methodologies of application design and deployment”(PRA_2018_66), funded by University of Pisa, Italy, and by the project “GIÒ: a Fog computing testbed for research & education”, funded by the Department of Computer Science of the University of Pisa, Italy.

Appendix A Code

A.1 EdgeUsher Prototype

The complete code (72 sloc) of EdgeUsher presented in Sect. 3 follows.

⬇

1placement(Chain, Placement, ServiceRoutes) :-

2 chain(Chain, Services),

3 servicePlacement(Services, Placement, []),

4 flowPlacement(Placement, ServiceRoutes).

5

6servicePlacement([], [], _).

7servicePlacement([S|Ss], [on(S,N)|P], AllocatedHW) :-

8 service(S, _, HW_Reqs, Thing_Reqs, Sec_Reqs),

9 node(N, HW_Caps, Thing_Caps, Sec_Caps),

10 HW_Reqs =< HW_Caps,

11 thingReqsOK(Thing_Reqs, Thing_Caps),

12 secReqsOK(Sec_Reqs, Sec_Caps),

13 hwReqsOK(HW_Reqs, HW_Caps, N, AllocatedHW, NewAllocatedHW),

14 servicePlacement(Ss, P, NewAllocatedHW).

15

16thingReqsOK(T_Reqs, T_Caps) :- subset(T_Reqs, T_Caps).

17

18secReqsOK([],_).

19secReqsOK([SR|SRs], Sec_Caps) :- subset([SR|SRs], Sec_Caps).

20secReqsOK(and(P1,P2), Sec_Caps) :- secReqsOK(P1, Sec_Caps), secReqsOK(P2, Sec_Caps).

21secReqsOK(or(P1,P2), Sec_Caps) :- secReqsOK(P1, Sec_Caps); secReqsOK(P2, Sec_Caps).

22secReqsOK(P, Sec_Caps) :- atom(P), member(P, Sec_Caps).

23

24hwReqsOK(HW_Reqs, _, N, [], [(N,HW_Reqs)]).

25hwReqsOK(HW_Reqs, HW_Caps, N, [(N,A)|As], [(N,NewA)|As]) :-

26 HW_Reqs + A =< HW_Caps, NewA is A + HW_Reqs.

27hwReqsOK(HW_Reqs, HW_Caps, N, [(N1,A1)|As], [(N1,A1)|NewAs]) :-

28 N == N1, hwReqsOK(HW_Reqs, HW_Caps, N, As, NewAs).

29

30flowPlacement(Placement, ServiceRoutes) :-

31 findall(flow(S1, S2, Br), flow(S1, S2, Br), ServiceFlows),

32 flowPlacement(ServiceFlows, Placement, [], ServiceRoutes, [], S2S_latencies),

33 maxLatency(LChain, RequiredLatency), %hp: only one maxLatency def

34 latencyOK(LChain, RequiredLatency, S2S_latencies).

35

36flowPlacement([], _, SRs, SRs, Lats, Lats).

37flowPlacement([flow(S1, S2, _)|SFs], P, SRs, NewSRs, Lats, NewLats) :-

38 subset([on(S1,N), on(S2,N)], P),

39 flowPlacement(SFs, P, SRs, NewSRs, [(S1,S2,0)|Lats], NewLats).

40flowPlacement([flow(S1, S2, Br)|SFs], P, SRs, NewSRs, Lats, NewLats) :-

41 subset([on(S1,N1), on(S2,N2)], P),

42 N1 == N2,

43 path(N1, N2, 2, [], Path, 0, Lat),

44 update(Path, Br, S1, S2, SRs, SR2s),

45 flowPlacement(SFs, P, SR2s, NewSRs, [(S1,S2,Lat)|Lats], NewLats).

46

47path(N1, N2, Radius, Path, [(N1, N2, Bf)|Path], Lat, NewLat) :-

48 Radius > 0,

49 link(N1, N2, Lf, Bf),

50 NewLat is Lat + Lf.

51

52path(N1, N2, Radius, Path, NewPath, Lat, NewLat) :-

53 Radius > 0,

54 link(N1, N12, Lf, Bf), N12 == N2, + member((N12,,,_), Path),

55 NewRadius is Radius-1,

56 Lat2 is Lat + Lf,

57 path(N12, N2, NewRadius, [(N1, N12, Bf)|Path], NewPath, Lat2, NewLat).

58

59update([],,,_,SRs,SRs).

60update([(N1, N2, Bf)|Path], Br, S1, S2, SRs, NewSRs) :-

61 updateOne((N1, N2, Bf), Br, S1, S2, SRs, SR2s),

62 update(Path, Br, S1, S2, SR2s, NewSRs).

63

64updateOne((N1, N2, Bf), Br, S1, S2, [], [(N1, N2, Br, [(S1,S2)])]) :-

65 Br =< Bf.

66updateOne((N1, N2, Bf), Br, S1, S2, [(N1, N2, Bass, S2Ss)|SR], [(N1, N2, NewBa, [(S1,S2)|S2Ss])|SR]) :-

67 Br =< Bf-Bass, NewBa is Br+Bass.

68updateOne((N1, N2, Bf), Br, S1, S2, [(X, Y, Bass, S2Ss)|SR], [(X, Y, Bass, S2Ss)|NewSR]) :-

69 (N1 == X; N2 == Y),

70 updateOne((N1, N2, Bf), Br, S1, S2, SR, NewSR).

71

72latencyOK(LChain, RequiredLatency, S2S_latencies) :-

73 chainLatency(LChain, S2S_latencies, 0, ChainLatency),

74 ChainLatency =< RequiredLatency.

75

76chainLatency([S], _, Latency, NewLatency) :-

77 service(S, S_Service_Time, _, _, _),

78 NewLatency is Latency + S_Service_Time.

79chainLatency([S1,S2|LChain], S2S_latencies, Latency, NewLatency) :-

80 member((S1,S2,Lf), S2S_latencies),

81 service(S1, S1_Service_Time, _, _, _),

82 Latency2 is Latency+S1_Service_Time+Lf,

83 chainLatency([S2|LChain], S2S_latencies, Latency2, NewLatency).

A.2 Proof of Correctness and Termination of EdgeUsher

We include here a sketch of the proofs of termination and correctness of EdgeUsher.

Proposition 1. The query $placement(Chain,Placement,ServiceRoutes)$ always terminates.

Proof. It is easy to prove that the query $placement(Chain,Placement,ServiceRoutes)$ always terminates since it:

• •

calls $chain$/2, which is matched against a set of facts and terminates immediately,

• •

calls $servicePlacement$/2 and $flowPlacement$/2, which both terminate.

The call $servicePlacement(Services,Placement)$ terminates since:

• •

predicate $servicePlacement$/2 just calls $servicePlacement$/3,

• •

$servicePlacement$/3 performs tail-recursion by reducing the size of its first term (a list), so that if the size of the first term in the first call to $servicePlacement$/3 is $n$ then $servicePlacement$/3 performs $n$ tail-recursive calls and terminates,

• •

before tail-recurring, $servicePlacement$/3

• –

calls $service$/5 and $node$/4, which are both matched against a set of facts and terminate immediately,

• –

calls $thingReqsOK$/2, which scans $m$ times its second term (a list), where $m$ is the size of its first term (a list, too), and terminates,

• –

calls $secReqsOK$/2, which terminates

either after scanning $m$ times its second term (a list), where $m$ is the size of its first term (if it is a list)

• *

or after recurring $m$ times by reducing the size of its first term (if it is an and-or term of depth $m$) and after scanning $m$ times its second term (a list),

• –

calls $hwReqsOK$/5, which performs tail-recursion by reducing the size of its fourth term (a list), and terminates.

The call $flowPlacement(Placement,ServiceRoutes)$ terminates since it:

• •

calls $findall$/3, whose inner goal is matched against a set of facts and terminates,

• •

calls $flowPlacement$/6, which performs tail-recursion by reducing the size of its first term (a list), and terminates; before tail-recurring, $flowPlacement$/6

• –

calls $subset$/2, which scans twice its second term (a list),

• –

calls $path$/7, which performs tail-recursion by reducing the size of its third term (a natural number), and terminates,

• –

calls $update$/6, which performs tail-recursion by reducing the size of its first term (a list), and terminates,

• –

before tail-recurring, $update$/6 calls $updateOne$/6, which performs tail-recursion by reducing the size of its fifth term (a list), and terminates

• •

calls $maxLatency$/2, which is matched against a set of facts and terminates immediately,

• •

calls $latencyOK$/3, which just calls $chainLatency$/4

• –

$chainLatency$/4 performs tail-recursion by reducing the size of its first term (a list), and terminates

• –

before tail-recurring, $chainLatency$/4 calls $service$/5 (which is matched against a set of facts and terminates immediately) and scans once its second term (a list). $\diamond$

Proposition 2. If $servicePlacement([s_{1},\ldots,s_{k}],P)$ is proved with computed answer substitution $P=[on(s_{1},n_{1}),\ldots,on(s_{k},n_{h})]$, then the service placement defined by $P$ satisfies all the IoT, security and hardware requirements of $[s_{1},\ldots,s_{k}]$.

Proof. We first prove —by induction on the size of the first term of $servicePlacement([s_{1},\ldots,s_{k}],P)$ — that:

[TABLE]

by inductive hypothesis and by lines 24—26 of the code in Appendix A.

We now prove that if $servicePlacement([s_{1},\dots,s_{k}],P)\rightarrow^{*}true$ with computed answer substitution $P=[on(s_{1},n_{1}),\dots,on(s_{k},n_{h})]$ then the service placement defined by $P$ satisfies all the IoT, security and hardware requirements of $[s_{1},\ldots,s_{k}]$.

The proof is by induction on the size of the first term of $servicePlacement([s_{1},\ldots,s_{k}],P)$.

(Base case) If $servicePlacement([s_{1}],P)\rightarrow^{*}true$ with computed answer substitution $P=[on(s_{1},n_{1})]$ then $P$ satisfies the IoT, security and hardware requirements of $[s_{1}]$ by lines 16 and 19—22 of the code in Appendix A, and by (*) respectively.

(Inductive case) If $servicePlacement([s_{1},\dots,s_{k}],P)\rightarrow^{*}true$ with computed answer substitution $P=[on(s_{1},n_{1}),\dots,on(s_{k},n_{h})]$ then $P$ satisfies all the IoT, security and hardware requirements of $[s_{1},\ldots,s_{k}]$ by inductive hypothesis and by lines 16 and 19—22 of the code in Appendix A and by (*), respectively. $\diamond$