A Survey on the Use of Preferences for Virtual Machine Placement in Cloud Data Centers

TL;DR

This survey reviews how preferences influence virtual machine placement in cloud data centers, highlighting challenges and opportunities for integrating stakeholder preferences into placement strategies.

Contribution

It provides a comprehensive overview of recent literature on preference-based VM placement and discusses future research directions.

Findings

Preferences significantly impact VM placement decisions.

Current methods often overlook stakeholder preferences.

Identified key challenges and research opportunities.

Abstract

With the rapid development of virtualization techniques, cloud data centers allow for cost effective, flexible, and customizable deployments of applications on virtualized infrastructure. Virtual machine (VM) placement aims to assign each virtual machine to a server in the cloud environment. VM Placement is of paramount importance to the design of cloud data centers. Typically, VM placement involves complex relations and multiple design factors as well as local policies that govern the assignment decisions. It also involves different constituents including cloud administrators and customers that might have disparate preferences while opting for a placement solution. Thus, it is often valuable to not only return an optimized solution to the VM placement problem but also a solution that reflects the given preferences of the constituents. In this paper, we provide a detailed review on the role of preferences in the recent literature on VM placement. We further discuss key challenges and identify possible research opportunities to better incorporate preferences within the context of VM placement.

Tables

Table 1. Table 1 : Existing application scenarios for preferences

Preference	References	Usage example
Resources	[18, 20, 25, 29, 89, 35, 34, 47, 62, 65, 68, 69, 72, 119, 118, 126, 135, 136, 14]	• A VM ranks PMs based on available resources or resources utilization ratio. • A PM ranks VMs based on required resource/s. • A VM prefers a PM with faster CPU.
Location	[74, 92, 113, 117]	• A customer prefers a geographical location over another for security or reliability concerns. • A cloud administrator prefers to place a group of VMs in a specific location for organizational purposes.
Inter-VM dependency	[89, 76, 75, 112, 136, 99]	• Preference information is derived from expected/measured traffic between VMs. • VMs collaborating in performing specific tasks or services.
Colocation	[26, 41, 42, 112, 117, 122, 43]	• VMs belonging to the same customer are preferred to be placed close to each other. • Flexible split preferences i.e., allow group VMs to be placed on different clouds or in the same cloud.
Anti-colocation	[26, 42, 47]	• Place VMs disjointly for security, reliability, or administrative purposes.
Sharability	[79, 115]	• A customer prefers to not share a PM with others.
Migration cost/overhead	[65, 72, 134, 142]	• VMs with lower migration overhead are more preferred to migrate. • VMs with migration cost (e.g., susceptible to SLA violation due to migration) are less preferred to migrate.
Availability/ Reliability	[142, 17]	• Include the degree of PM wear-and-tear (e.g., due to frequent PM ON/OFF cycles) in weighted function. • Replicate a VM to improve its availability.
Cost	[20, 17, 56]	• A customer may prefer a lower cost PM. • A cloud admin may prefer to place VMs based on their payment scale and the cost of the infrastructure. • Typically, the cost computed as utility function composing multiple attributes or just monetary value.
Ordering	[13, 52, 54]	• Preference order of preferred PM to host a VM or a set of VM.
Distance	[29, 66, 134]	• An application prefers computing and storage PMs pair with smaller distance. • An application prefers to be placed near a specific PM.
Objectives	[12, 22, 39, 77, 103, 125, 54]	• The extent to which an objective is prioritized over other objectives.
SLA	[108, 142]	• Soft SLA tolerates some violations.
Security	[20, 141]	• Security preferences can be modeled as a function of the system diversity. • Place VMs according to their security preferences and PMs capabilities.
Other/ Unspecified	[98, 120, 121, 130, 87, 74, 106, 44]	• a generic model in which preferences information are customized/defined by the user/DM. • Place a VM in a specific PM [74, 106].

Table 2. Table 2 : Existing Approaches that handle preferences in VMP

Method	References
Metaheuristics	Simulated annealing [120, 121, 12], NSGA-II [22, 13], Improved grouping genetic algorithm [39], Memetic Algorithm [54, 77] Ant-colony [76, 75], heuristics [79] Hybrid algorithm [103]
Matching	Stable matching [69, 41, 135, 66, 35, 34, 126, 68, 72, 56], Matching game [20, 99], Three-sided stable matching [29], Markov approximation & stable matching [89], College admission problem [65, 136], Resource Balancing Placement matchmaking [74] Uncertainty principle of game theory [92], Egalitarian stable Matching [134], Game theory [141]
MCDM	TOPSIS [119, 118, 18, 44], PROMETHEE [62], General MCDM [98], Fuzzy MCDM [17], AHP [111]
Heuristic	First-Fit [108], Next-Fit and First-Fit Decreasing [42], Opportunistic-Fit [47], Multiple Heuristics [115]
Other	Fair proportional scheduler [25] Bin packing [26, 52], ILP repacking algorithm [106], Distributed bartering algorithm [112] Random search (biased statistical sampling) [117] Random search (biased statistical sampling) [122] Brute-force [125], Constraints-based model [130], NA [142], A framework [87, 82], cloud broker [14]

Table 3. Table 3 : Detailed overview of the existing literature (Metaheuristics)

Reference	Perspective	Problem Type	Objectives	Algorithm	Preferences	Comments
Addya et al., [12]	cloud provider	placement	Energy – Revenue	Simulated annealing	Objectives	A bi-objective optimization model to maximize revenue and minimize energy consumption. A trade-off between the two objectives is maintained based on DM’s preference.
Alashaikh and Alanazi, [13]	cloud provider	placement	Traffic – Energy – Resource wastage	MOEA	Ordering	Placement decision preferences are given a priori as a total order.
Blagodurov et al., [22]	cloud provider	placement	Traffic – Energy – Resource consumption	NSGA-II	Objectives	Posteriori preferences on objective space.
Deng et al., [39]	cloud provider	consolidation	Energy – Resource consumption – Availability/Reliability	MOEA - improved grouping genetic algorithm	Objectives	Defines a weighted utility functions as a weighted combination of SLA satisfaction, reliability and energy impacts, where the weights on each metric represents the DM preferences.
Ihara et al., 2015b [54]	cloud provider	placement migration	Traffic Energy – Revenue – No. of migrations – Migration cost	Metaheuristics Memetic Algorithm	Ordering	Set of Pareto solutions with preferences applied on objective space. Multiple selection strategies such as distance to origin and Lexicographic Order (Each objective function is given in an order of evaluation)
Liu et al., [76]	cloud provider	consolidation	Energy – No. of migrations	Ant-colony	Inter-VM dependency	Obtains Pareto-based solutions and then the trade-off between the two objectives is adjusted based on the DM’s preferences.
Liu et al., [75]	cloud provider	consolidation	Resource consumption – min. active PMs	Ant-colony	Inter-VM dependency	The ant-colony algorithm deposits the pheromone between VMs rather than between a VM and a PM and represents the preference between VM and PM. The preference is based on the historical experience of packing the VM together with those VMs that have already been placed on the PM.
López-Pires and Barán, [77]	cloud provider	placement	Traffic – Energy – load Balancing – Revenue – QoS priority	Metaheuristics Memetic Algorithm	Objectives	Preference knowledge is utilized to reduce search space by imposing upper and lower bounds on objectives.
Mahalingam and Sengottaiyan, [79]	user – multi-cloud	placement	satisfaction	Simulated Annealing-based	Sharing	The model allows customers to specify their preferences (along with other QoS requirements) as hints based on whether their VMs can be located with other users? VMs or alone. Further, it considers soft constraints that only contribute to the satisfaction score and not enforced as the hard ones. (Uses Nefeli [121])
Saber et al., [103]	multi-cloud	re-placement	Energy – Cost – Migration cost – Availability	Metaheuristics	Objectives	Hybrid cloud managed by multiple entities, each of which has different requirements and preferences in maintaining its virtual cloud. Preferences are incorporated as weights on the objectives. Three algorithms are utilized greedy algorithm to find initial solutions, NSGA to diversify, and a local search to obtain more non-dominated solutions.
Tsakalozos et al., [120]	user	migration	other/unspecified	Simulated annealing-based & 2-phase heuristics	other/unspecified	Adopts hints to represents user preferences, internal physical cloud infrastructure, and high-level management goals that the DM seek. Fine-grained constraints refer to a) all specifics regarding the hosting capacity including features and resource availability in PMs, b) user-provided deployment hints and c) cloud administrative goals. A ranking algorithm finds different sets (Cohorts) of PMs that satisfy VM requests in terms of resources. Simulated Annealing algorithm to select a single cohort that satisfy preferences.
Tsakalozos et al., [121]	user – cloud provider	placement	other/unspecified	Simulated annealing	other/unspecified	proposes Nefeli, a hint-based VM placement algorithm. In Nefeli, a predefined set of preferences are mapped onto a utility function with some weight (a score) on each preference criterion. The model also utilizes soft and hard constraints for both consumer and provider.

Table 4. Table 4 : Detailed overview of the existing literature (Matching)

Reference	Perspective	Problem Type	Objectives	Algorithm	Preferences	Comments
Azougaghe et al., [20]	cloud provider	migration	Energy – Resource consumption	Matching game	Resource – Cost – Security	VMs preferences and PMs are represented as utility functions that take into considerations multiple attributes such as resources and security level for VMs and risk, energy, and revenue for PMs.
Chu et al., [29]	cloud provider	placement	Cost	Three-sided Stable matching	Resource – Distance	PM nodes are decoupled as computing and storage nodes. three-sided matching among computing, storage resources and applications, each with its preference list
Pham et al., [89]	cloud provider	placement	Traffic Energy	Markov approx. and Stable matching	Resource – Inter-VM dependency	vNFs of the same service chain maintain a preference list on PMs, while each PM maintains a preference list on vNF instances.
Cui and Tso, [35]	cloud provider	migration consolidation	Resource consumption – Policies	Stable matching	Resource	preference knowledge is the cost of paths and links for VMs traffic.
Cui et al., [34]	cloud provider	migration consolidation	Resource consumption – Policies	Stable matching	Resource	same as above.
Dhillon et al., [41]	cloud provider	placement	Satisfaction	Stable matching	Colocation	Every VM acts as selfish agent and has a preference (strict total order) over other VMs to partner with. The system tries to minimize the degradation of VMs placement problem and find a solution that minimizes the global performance degradation.
Ranjana et al., [99]	cloud provider	placement	Energy –comm. cost – min. active PMs	Stable matching	VM dependency	Affinity-based algorithm.
Jain et al., [56]	multi-cloud	placement		Stable matching	Cost	multiple cloud providers, intermediate brokers and end users
Kella and Belalem, [65]	multi-cloud	migration	Energy	Stable matching /college admission problem	Resource – Migration cost	VMs preference list contain PMs put in an ascending order according to the transfer time. PMs preference list contain VMs put in an ascending order according to their impact to cpu utilization.
Kim and Lee, [66]	cloud provider	migration		Stable matching	Distance	the model allows ties in the preference list ordering (i.e., preferences are not given in a strict order)
Korasidis et al., [68]	user – cloud provider	placement	other/unspecified	Stable matching	other/unspecified	Addresses algorithm scalability.
Korupolu et al., [69]	cloud provider	placement	Cost	Stable matching	Resource	One of the first proposals that applied stable matching to the VM placement problem. It considers compute and storage PMs, and applications of different computation and data needs. Requirements and other applications preferences can be integrated into a weighted utility function.
Li et al., [74]	user	consolidation	min. active PMs – Resource wastage	Stable matching (resource balancing and matchmaking)	Location – Specific PM	Defined normal and categorized types of preferences and flexible and inflexible preferences.
Li et al., [72]	cloud provider	consolidation	Energy	Stable matching	Migration cost – Resource	Modeled as a matching problem on a weighted bipartite graph, with VM migration cost used as a weight on edge between VM and PM. Each VM candidate for migration has a preference list on a subset of PMs chosen to accommodate migrated VMs. The list is ordered based on weights.
Pillai and Rao, [92]	cloud provider	placement	Revenue	Stable matching and uncertainty principle of game theory	Location	The proposal considers forming coalition between PMs to satisfy requirements that are beyond single machine capabilities. The PM has preference over which coalition of machines to go with based on expected payoff.
Wang et al., [126]	cloud provider	migration	Energy	stable matching	Resource	PMs and VMs have mutual objective of freeing overloaded PMs and utilizing underutilized PMs in order to reduce overall energy consumption. The proposed algorithm tries to achieve a reasonable trade-off between power consumption and SLA violation. Preference list is computed accordingly.
Xu and Li, [134]	cloud provider	migration	other/unspecified	Egalitarian stable Matching	other/unspecified	A VM or a server ranks the agents on the other side based on some criteria, which is its preference list. An algorithm then computes a matching that cannot be improved by any pair of VM and server. VMs rank servers in an ascending order of the transmission cost. Servers rank VMs in an ascending order of the migration overhead.
Xu and Li, [135]	user cloud provider	placement	Resource consumption – other/unspecified	Stable matching	other/unspecified	A VM preference list ranks servers according to their resources. PMs may prefer packing larger or smaller VMs recognizing DM’s preference of consolidation or load balancing, respectively. Proposed a revised DA algorithm that finds weakly stable matching which can be an effective approach for handling large-scale problems.
Xu et al., [136]	user cloud provider	placement	Resource consumption	Stable matching/ College admission problem	Resource – Inter-VM dependency	The stable matching theory is applied to generate an optimal mapping from containers to physical servers.
Zhang et al., [141]	cloud provider	migration	Cost	game theory	Security	Users security preferences are modeled as a linear function of the system diversity (using weighted function of the total number possible VM-placements). The cost accounts for migration time and number of migrations.

Table 5. Table 5 : Detailed overview of the existing literature (MCDM)

Reference	Perspective	Problem Type	Objectives	Algorithm	Preferences	Comments
Araujo et al., [17]	user	placement	Availability – Cost	Stochastic model & MCDM	Objectives	The authors use stochastic models to assess and evaluate different clouds from user’s perspective. Assessment results are then used to rank solutions, taking into account the user’s preference of each criterion (set as a weight).
Arianyan et al., [18]	cloud provider	consolidation	other/unspecified	TOPSIS	other/unspecified	Preferences are expressed as weights for each criteria. The weights are computed based the average utilization of all system resources in the data center (i.e., functional). The utmost objective is to remove hotspots.
Feng et al., [44]	cloud provider	consolidation	Cost	TOPSIS/AHP	Resources	The algorithm is mainly concerned about the migration decision.
Kabir et al., [62]	multi-cloud	placement	Resource consumption	PROMETHEE	other/unspecified	a preference function is used to denote the preference of solution over another for each criterion. Then, each solution is evaluated by the number of alternatives outrank it and the number of alternatives it outranks based on the preference index. The solution with the highest net outranking flow is considered as the most preferred.
Rane and Srivastava, [98]	user	re-placement	Satisfaction	MCDM	other/unspecified	Defines a constraint significance factor that signifies the preference of customer corresponding to each constraint.
Son and Huh, [111]	user – cloud provider	placement/ migration	Resource utilization – QoS – no. of migrations – Energy	MCDM/ AHP	Objectives	Proposes to use Analytical Hierarchy process (AHP) coupled with a fuzzy interferer system to solve the VMP problem. By monitoring resources and analyzing service requirements, a placement decision can be obtained from the fuzzy system. Then through AHP algorithm, a placement action is performed. The AHP is managed to provide analysis of the weights for each metrics involved in the decision.
Tarighi et al., 2010b [119]	cloud provider	migration	load Balancing Migration cost	Fuzzy TOPSIS	Resource	deterministic, linguistic, and fuzzy preference information.
Tarighi et al., 2010a [118]	cloud provider	migration	load Balancing Migration cost	TOPSIS	Resource	same as other one

Table 6. Table 6 : Detailed overview of existing literature (Heuristics & others)

Reference	Perspective	Problem Type	Objectives	Algorithm	Preferences	Comments
Chen et al., [25]	cloud provider	placement	Resource consumption	Heuristics	Resource	VMs are classified into domains based on their CPU-affinity.
Chen et al., [26]	cloud provider	placement	min. active PMs	Bin packing	Colocation – Anti-colocation	Affinity-aware placement – hard constraint
Dow, [42]	cloud provider	consolidation	min. active PMs	Heuristics (pin-packing using next-fit and ffd)	Colocation – Anti-colocation	decomposed into two phases; resources computation then colocation.
Georgiou et al., [47]	cloud provider	placement	other/unspecified	Heuristics (Opportunistic-fit greedy algorithm)	Resource – Anti-colocation	IaaS requests. Used the notion of hints to allow users specify their bandwidth needs for specific pairs of VMs and anti-collocation constraints for pairs of VMs. aim to faster solution time. Flexibility of constraints.
He et al., [52]	user – cloud provider	migration	Resource consumption – Cost	Heuristics	Ordering	Each PM machine has a set of properties associated with its resources and attributes (e.g., RAM, type of OS, etc). If multiple PMs are suitable for hosting a VM (i.e., satisfy the requirements), then the system looks for user’s preferences.
Ooi et al., [87]	cloud provider	placement – replication	Availability	Team formation	other/unspecified	A framework for resource evaluation to identify preferable resources and prepare for service placement.
Sedaghat et al., [106]	cloud provider	repacking	Resource utilization	ILP	other/unspecified	Repacking by changing the configuration of VMs (either number or sizes) that reside in a server, as the load changes. The goal is minimizes resource utilization cost accounting for both price and performance issues. A decision of repacking or not can be based on the calculated cost and other factors e.g., expected time to next change, interruption time, etc.
Shi and Hong, [108]	cloud provider	placement	Energy – Revenue	First-fit Heuristic	SLA	Hard SLA and soft SLA.
Sonnek et al., [112]	cloud provider	migration	Traffic	Heuristics	Inter-VM dependency – Colocation	affinity; satisfy it as much as possible
Sotiriadis et al., [113]	user – multi-cloud	migration	other/unspecified	Migration tool	location – other/unspecified	migration of VMs in FIWARE systems
Sunil Rao and Santhi Thilagam, [115]		migration	min. active PMs – Resource wastage	Heuristics	Sharing	not implemented/discussion only.
Tantawi, [117]	user – cloud provider	placement	Load Balancing	Heuristics/random search (biased statistical sampling)	Location – Colocation	location constraints can be hard or soft.
Unuvar et al., [122]	user – multi-cloud	placement	Traffic load Balancing Cost	Heuristics - Biased Importance Sampling	Resource – Cost – other/unspecified	Introduced a hybrid cloud placement algorithm that incorporates two policy parameter: reservation and cost weight and addresses the trade-off between deployment cost and request rejection rate.
Viswanathan et al., [125]	cloud provider	placement	Energy – Completion time	Brute-force + set partitioning algorithm	Objectives	Addresses HPC applications placement in DC.
Wright et al., [130]	user	placement	other/unspecified	Heuristics	other/unspecified	user perspective. measured/evaluated in cost
Zhou et al., [142]	cloud provider	migration	Cost – Migration cost	Heuristics	other/unspecified	Incorporates user experience (IO penalty), cluster management (migration time) and device usage (degree of wear). Utilizes a weighted cost function in which the weights on each metric reflects different design preferences.

Full text

1 Introduction

Today, virtualization technology has facilitated the hosting of millions of services and applications on the cloud and simplified their deployments and accesses. Many small, medium, and large organizations have been moving their applications or even their entire data centers to the cloud. This is mainly motivated to their economical advantage. in terms of real-estate, equipment costs and manpower compared to private data centers. Cloud data centers also maintain high degree of flexibility where services and applications can be deployed, terminated, migrated, and replicated on-demand [91, 114]. In addition, cloud data centers offer a wide range of customizable settings and add-on features to suit the requirements and non-functional needs of different customers.

Essentially, virtualization enables dynamic sharing of physical resources such as processors, memory, and storage of the cloud infrastructure. An application is installed on a logically isolated Virtual Machine (VM) that has limited and pre-configured access to the shared resources of Physical Machines (PM) and networking devices. The process of mapping the VMs to the existing PMs is commonly known as the Virtual Machine Placement (VMP) problem. The VMP problem is of paramount importance to the design and operation of cloud data centers. Throughout the last decade, there have been many works addressing the VMP from different perspectives. The majority of these works are concerned about the profitability of data centers and the energy bill as well as other operational aspects. While many proposals share similar objectives, they may have taken different approaches, yielding a wide range of frameworks, models, and algorithms.

A pre-requisite to the success of existing approaches is handling customer and administrator preferences when looking for a placement solution. This is due to the fact that cloud administrators and customers are often not satisfied with any placement solution. Rather, they are interested in the best, or at least good, solution that meets their preferences. As a motivating example, consider Bob who wants to place a VM that requires one unit of CPU and two units of RAM. Alice, on the other hand, has a VM that requires two units of CPU and one unit of RAM. The cloud data center, which is administrated by Charlie, has two physical machines $P_{1}$ with 3 CPU units and 4 RAM units and $P_{2}$ with 4 CPU units and 3 RAM units. Assume the objective is to minimize the total wastage of RAM and CPU units. Obviously, the four possible placements $(P_{1},P_{1})$,$(P_{2},P_{2})$,$(P_{1},P_{2})$, and $(P_{2},P_{1})$ are all Pareto optimal with total wastage of 8.

Now, assume both Alice and Bob prefer placing their VMs in a PM with faster CPU. In such case,the solution $(P_{2},P_{2})$ is preferred to all other solutions. Furthermore, in case this solution is unattainable (*e.g., *violating a constraint), $(P_{1},P_{2})$ and $(P_{2},P_{1})$ are both preferred to $(P_{1},P_{2})$. These placements represent the most preferred feasible options given both Bob and Alice preferences. Finally, Charlie prefers Bob’s VM be placed on $P_{1}$ to better meet Bob’s service level agreement (SLA) and therefore, $(P_{1},P_{2})$ is the optimal placement given the three preferences. This example demonstrates that cloud customers and administrators may have preferences that need to be considered while consolidating customer VMs (*e.g., *customers prefer servers with faster CPUs) and solutions are no longer indifferent but some are preferred to others.

There has been a number of attempts trying to capture preferences during the VMP process. For instance, statements of the form “I prefer $X$ and $Y$ to be placed on a different geographic location", “I prefer $X$ not to be co-located with other VMs", or "I prefer $X$ and $Y$ to be placed on the same PM" are particularly popular in the literature as customer preferences [138, 120, 121, 43, 40, 42, 13].

Furthermore, the cloud administrator is usually overwhelmed with a large set of potential placements and has several preferences on the deployment process. While any solution in the set is optimal to the underlying constraints and objectives, some solutions are expected to be preferred (by the administrator or customer) compared to others [61, 59, 127, 123, 121]. Neglecting the preferences while finding a solution would discourage the customers from subscribing to the cloud and administrators from using the proposed approach. Hence, proper handling of preferences is a critical issue to address in the design of a cloud.

The relevant literature is rich with survey papers on the VMP problem. There are multiple VMP survey papers that interested readers may refer to, such as [128, 80, 93, 91, 114, 81, 110, 85, 19, 139, 116]. Most of these surveys provide multiple classifications and reviews of approaches to handle the VMP problem including modeling, the objectives and constraints representation and formulation, the techniques used to solve the problems, the scope and the context in which the placement is taking place and more profoundly to its variants. The focus on this paper is different as it aims to look deeper into the notion of preferences in the VMP context. We begin with a clarification of what is meant by preferences in the context of an optimization problem. Figure 1 shows a conceptual difference between the three types of knowledge. Basically, the objectives and constraints of the problem instantiate its optimality and feasibility regions, whereas satisfaction is commonly attributed to preferences [112]. Indeed, one can promote a preference to an objective or constraint. Many attempts in the literature adopt this methodology and add preferences as an objective to the problem. We believe such transformation albeit perhaps efficient, has its own drawbacks as it is not always possible to construct objectives that reflect the preferences without losing some information. Especially when the customer or administrator pose complex preference statements over different aspects of the network.

In this paper, we provide a detailed survey of VMP proposals that incorporate soft constraints and preferences highlighting their types, implications, and the algorithms used to handle them. We also discuss some challenges and identify possible research opportunities to better incorporate preferences in the context of VMP. A classification of the proposed approaches is given in Figure 2. Some of the important aspects that need to be addressed when proposing an approach include:

Source

identifies who initiates the preferences, the customer who owns the VMs, the cloud administrator who owns the PMs, or both.

Type

depicts how the preferences are represented. (*i.e., *via utility function, binary relation, etc.)

Approach

classifies the algorithm family used to develop an algorithm. In a broad sense, they can be classified to heuristic, metaheuristic, and exact algorithms.

Problem

describes the VMP problem context as there are different variations of the problem, such as migration and re-placement.

It is important to realize that the different considerations of these aspects impose some limitations on how the preferences are represented and processed. More details about these aspects and their implications will be discussed throughout the paper.

The rest of the paper is organized as follows: Section 2 presents background regarding the VMP problem and preferences. We then review the recent literature on the VMP problem with preferences in Section 3. Section 4 provides our insights on the potential benefits for the use of preferences in the context of the VMP problem, followed by a discussion over current limitations and future research opportunities. Section 5 concludes this work.

2 Background

2.1 VMP design problem

VMP is an essential process for the design and operation of cloud data centers. VMP is concerned with the assignment of virtual machines to physical machines (servers) in the data center. The decision of such placement highly impacts the performance of the data center in terms of utilization levels, traffic congestion level, and energy consumption, all of which the Decision Makers (DMs) continuously try to optimize.

A wide range for data center network topologies have been studied extensively in the literature. Basically, the architecture tells us how routers, switches, and servers are connected, and what is the number of ports required on these devices to implement the network. The choice of a topology has a great impact on the cost, scalability, resiliency (in terms of alternative paths), complexity in terms of number wires, and bandwidth and traffic distribution within the data center. As an example, consider the tree topology shown in Figure 3. It is comprised of three layers; namely, Core, Aggregation, and Edge. The core layer is the data center gateway to the public Internet, whereas the switches in the edge layer interconnect the servers (called physical machines) to each other and to the gateway through the middle layer switches. This architecture is a classic one that is commonly used as a reference or baseline for evaluating other proposals. It is widely adopted due to its simplicity. Despite that, it raises a number of technical concerns regarding its scalability and resiliency. Hence, modern data centers have evolved to utilize more sophisticated architectures such as Clos tree, recursive and random topologies. More about alternative architectures and their feature can be found in [94, 137, 33, 109].

Many virtualization tool developers have realized the importance of the customizability of the virtual services and also enabled VMP based on multiple attributes that also consider non-technical ones, *e.g., *Profile-Driven Storage in vSphere 5.0 [9, 11], HP Moab [6], Server affinity policies in Oracle VM server [8], and many others [15].

In realization to consumers varying needs, cloud service providers have designed multiple options for VMs owners to choose from, in which an owner can choose from a set of pre-defined VM types (*i.e., *a combination of disk space, allocated memory, bandwidth, etc.) [3, 1]. On top of that, many providers went to offer added-value or policy-based options, such as colocation, geo-redundancy, availability, etc (see for instance, placement policies and proximity placement group of Microsoft Azure [5, 2], phoenixNAP colocation and security [4], IBM High availability [7] , and most recent mobile core network virtualization on AWS [10]).

A basic VMP problem can be defined as follows: Given $\mathcal{PM}=\{P_{1},P_{2},...,P_{m}\}$, a set of $m$ physical machines ($P_{k}$) and $\mathcal{VM}=\{V_{1},V_{2},...,V_{n}\}$, a set of $n$ virtual machines ($V_{i}$), how to place each virtual machine into a physical machine while optimizing a set of objectives and satisfying a set of constraints?

2.1.1 Objectives

Basically, the utmost goal when solving a VMP problem is to optimize some metrics of interest. These metrics are identified as objectives. From a cloud administrator’s perspective, the main objectives considered in the VMP are highly related to maintaining profitable operation of the cloud. On top of the list of these objectives is energy consumption. Typically, energy consumption is modeled in terms of CPU utilization, and a possible minimization approach is to distribute the load among existing PMs to reduce CPU usage on each PM. Alternatively, turning off underutilized PMs and move their VMs to other PMs appeared to be more a effective strategy. However, this approach brings the issue of wear-and-tear to the surface as PMs maintaining higher resource utilization are expected to fail sooner than others. In addition to energy consumption, efficient packing of VMs with low defragmentation (resource wastage index) increases the ability to accommodate more VMs in the cloud which is an important feature from the profitability perspective. Moreover, VMs may need to communicate with each other to perform a certain task. If two communicating VMs are hosted on the same PM, then no traffic will go through the switches. However, if they reside on two different machines, traffic has to go through intermediate switches. This kind of traffic is called intra-data center traffic as opposed to external traffic from/to the Internet. Excessive intra-data center traffic may disrupt traffic between VMs and Internet as intermediate switches may become congested. In order to prevent this from happening, the traffic between peer VMs needs to be localized as much as possible. This can be achieved by placing VMs that are highly interdependent close to each other. However, this is mainly constrained by the capacities of the PMs.

The objectives may also involve other metrics that contribute to the profitability index. For example, traffic minimization to avoid unpredicted/undesired congestion that may degrade performance, QoS satisfaction, infrastructure reliability, VM availability, energy consumption of switches, routers, and auxiliary equipment (*e.g., *cooling systems), etc. For more examples on alternative objectives the reader is referred to [93, 80, 19, 110, 85, 139].

Generally, a single objective or multiple objectives can be involved in making the placement decision. However, due to the nature of the VMP problem, involving multiple dimensions, factors, stakeholders, and influential metrics, researchers usually consider multiple objectives. As discussed above, these objectives are highly interdependent and likely conflicting with each other.

2.1.2 Constraints

Mostly, any VMP problem can be seen as a pin packing problem. Hence, the constraints are more or less similar to the pin packing problem: the assignment constraint ensures that each VM is assigned to one PM, and one PM can host multiple VMs, and the capacity constraint ensures the availability of enough resources on the PM to utilize and fulfill VM requirements. Typically, each VM requires resources described in terms of CPU cycles, RAM space, disk space, network bandwidth, etc. While a server $P_{k}$, leveraging virtualization techniques, can host more than a single VM, the resource utilization (*i.e., *load) should not exceed its resource capacities. In addition, other design considerations can be imposed as constraints. For example, a cloud administrator may be interested in setting a limit on the number of VMs migrations to avoid instability and interruption of the cloud operation, or impose a constraint on security levels for some VMs.

2.1.3 Variants

There are multiple variants of the basic VMP problem that appeared in the literature. The basic VMP problem may assume that there are some VMs hosted in a DC and new VMs are to be placed on PMs based on the current utilization and operational status of the DC. On the long-run, this might lead to defragmentation of the resources. Therefore, rearrangement of the VM-PM mapping is desired in which multiple VMs are consolidated into a PM to enhance the mapping efficiency [115]. Migration is the technique used to move a running VM from its current PM and re-assign it to a new PM. It is a powerful technique that enables the cloud operator to reallocate resources and reconfigure assignment based on current status of the machines [139]. However, there are critical considerations when making the migration decisions, particularly, related to the timing, frequency, and the cost of migrations [124]. Hence, these factors exacerbate the complexity of problem in which adopting soft rules or preferences to handle them and introduce a meaningful trade-off is advisable. In our study, we refer to these VMP problem variants when classifying the research efforts that appeared in the literature.

VMP can also be classified based on the number of the clouds involved [80]. A single-cloud (private or public) VMP problem is concerned with the placement of VMs into PMs within the same data center. Multiple clouds VMP, typically, entails solving two assignment problems in which the system has to distribute incoming VM requests over the different clouds and then optimize each cloud individually. There are multiple architectures and models for multi-cloud VMP. For example, hybrid cloud [122], multiple cloud with multiple administrators [103], and others [82, 49, 58, 56]. Another related but different problem is the cloud administrators selection problem where it is assumed that customers express preferences and intermediate brokers negotiate for the best offer [14, 65, 79, 83].

2.2 Preferences

Preferences are ubiquitous. Whether we want to choose a restaurant for lunch, watch a movie or purchase an airline ticket, we often have preferences over the set of available alternatives. Thus, preferences play a key role in the decision making process and have been an active topic of research in many fields such as artificial intelligence, economics, and operations research [64]. Preferences can be broadly classified based on their representation as quantitative and qualitative. In quantitative preferences, users express preferences with numerical values representing the utility of having one alternative. On the other hand, qualitative preferences are expressed based on comparative assessments of alternatives. Figure 4 shows a general classification of preferences. The graphical models refer to a class of models that have been introduced lately in the literature to represent the preferences in a compact way. They rely on the notion of (conditional) preferential independency among attributes and can be described in terms of vertices and edges of a graph [16]. In what follows, we review the basic methods used to represent preferences (interested readers can refer to [45] for a comprehensive introduction to the field). Informally, preferences are any additional knowledge that is not part of the problem requirements and objectives, but represent the subjective judgments of the users in terms of liking or disliking an alternative.

In decision theory, preferences provide a way to evaluate alternatives based on subjective judgments. It is typically assumed that every alternative $\bf x$ is a vector over some $k$ attributes ${\bf x}=(x_{1},x_{2},\dots,x_{k})$ that represent the world. For instance, in the context of VMP, an alternative can be represented by a possible placement of the VMs to PMs and the set of alternatives represent the placements that are possible given the problem constraints and objectives. Given a set of alternatives $S$, there are two important questions that need to be answered by the preference model:

• •

(optimality) What is the best alternative given the preferences?

• •

(dominance) Given two possible alternatives $\bf x$ and $\bf y$, which one is more preferred?

One of the simplest models for preferences is the weighted sum model obtained by assigning an importance weight $w_{i}$ to every attribute $i$. Then alternatives are assessed based on their value i.e., $w({\bf x})=\sum_{i=1}^{k}w_{i}x_{i}$ where $w_{i}\geq 0$ and $\sum_{i=1}^{k}w_{i}=1$. For instance, consider three possible alternatives $x_{1}=(10,20,30)$, $x_{2}=(30,10,20)$, and $x_{3}=(20,5,25)$ defined on a set of three attributes $a_{1},a_{2},a_{3}$ with importance weights $0.6$ for $a_{1}$ and $0.2$ for both $a_{2}$ and $a_{3}$. Using the weighted sum model, $w(x_{1})=0.6\times 10+0.2\times 20+0.2\times 30=16$, similarly $w(x_{2})=24$ and $w(x_{3})=18$. Given these values, one can conclude that $x_{2}$ is the best alternative, as it has the highest value among other alternatives and also $x_{3}$ is preferred $x_{1}$ since its value is higher.

Another representation is the value (or utility) function where alternatives are mapped to real numbers $\mu:S\rightarrow\mathbb{R}$. Here, every alternative $\bf x$ is mapped to a real number $\mu({\bf x})$ that represents the desirability of the user for such alternative. Considering the above example in the weighted sum model, we can have $S=\{x_{1},x_{2},x_{3}\}$ and the alternative utility is simply its sum, i.e., $\mu(x_{1})=16$, $\mu(x_{2})=24$, and $\mu(x_{3})=18$. In general, an arbitrary numerical value can be associated with every possible alternative expressing its preference to the decision maker.

Generally, a preference over a set of alternatives $S$, can be defined as a binary relation on $S$ [64]. Several properties arise to different relations. We start by general definition of preference relations.

Definition 1

A preference relation $\succeq$ is a preorder if $\succeq$ is reflexive and transitive.

A preorder is called a partial order if it is also antisymmetric. A statement of the form $x\succeq x^{\prime}$ is interpreted as $x$ is at least as good as $x^{\prime}$. Given two alternatives $x,x^{\prime}$, one can derive some useful conclusions between the two as follows

$x$ is strictly preferred to $x^{\prime}$ iff $x\succeq x^{\prime}$ and $x^{\prime}\not\succeq x$. 2. 2.

$x$ is indifferent from $x^{\prime}$ iff $x\succeq x^{\prime}$ and $x^{\prime}\succeq x$. 3. 3.

$x$ is incomparable with $x^{\prime}$ iff $x\not\succeq x^{\prime}$ and $x^{\prime}\not\succeq x$.

Incomparability captures the scenario where there is no possible information in $\succeq$ to assess the two alternatives. However, being indifferent means the user actually prefers both alternatives to the same level. One important property that is usually assumed in practice is the completeness of the partial order.

Definition 2

A partial order $\succeq$ is said to be complete iff $x\succeq x^{\prime}$ or $x^{\prime}\succeq x$ for any $x,x^{\prime}\in S$.

A complete partial order is also known as a linear order or a ranking over a set $S$. In such case, any two alternatives are comparable and one is preferred to the other. Lexicographic preference relations are also common in practice where an importance order over attributes are assumed.

Another model of preferences that has gained much attention in recent years is the soft constraint model. In such model, constraints are associated with numerical values representing the cost, importance, or preference and unlike hard constraints, alternatives are not required to satisfy all of them. A formal representation for soft constraints is Valued Constraint Satisfaction Problem (VCSP) [105]. Lastly, the concept of reference point is widely used in the evolutionary algorithms to capture users’ idealized vision of the solution [37] and define a distance to relatively evaluate the closeness of a given alternative to the reference point.

2.3 Applications of preferences in communications problems

Given the rise of multiple aspects involved in the design of communications system that are of interest to decision makers (*e.g., *delay, availability, utilization level, hop-count etc.) as well as concepts or design features that are desired to achieve (*e.g., *load balancing, fairness, human-centric design, etc.), many communications problems have been regarded today as multi-dimensional or multi-criteria problems. Hence, incorporating the preference or soft constraint concept into the problem can be useful to many applications in communications beyond the VMP problem.

There are many problems in the communications and networking field where it is advisable to consider such preferences and/or soft constraints. Such problems include multiple decision makers in which each entity has its own requirements (solid constraints) and preferences, or a problem where multiple metrics are involved in the design with some sort of priority or importance (*e.g., *delay is more important than reliability) and require insightful handling of preferences in the design [30].

Consider for example, a multi-criteria routing problem. Typically, one metric is regarded as an objective while others are given as constraints, or alternatively the multiple metrics are aggregated into a single objective function with some defined weights. In a multi-criteria approach, the problem is approached from a different perspective where metrics to be optimized are handled as objectives. This promotes optimization in all dimensions of the problem which results in multiple solutions scattered in hybrid-planes. Using preferences (*e.g., *bounds, sequences, etc.), solution space can be partitioned into smaller regions each has a subset of the solutions with different desirability level [31].

Similarly, a telecom network planning with multiple cost types (*e.g., *investment, maintenance, management, extension, etc.) and different design criteria (*e.g., *capacities, availabilities, diameter, edge degree, etc.) is another example where the adoption of preferences can help achieve more desirable outcome [38]. This also can be extended to multilayer network design to perform mapping between different layers or design overlay networks [100], traffic grooming [102], or resource provisioning considering the cost of each layer independently [133].

Another example in communication networks is the design of wireless sensor networks which entails multiple design sub-problems such as sensor placement, data aggregation, power assignment, etc. Typically, each considers a set of objectives to be optimized (*e.g., *detection probability, latency, deployment cost), and a proper definition of objective preferences or layout preferences would improve our perception to the design [67, 97, 86].

Furthermore, the rapid evolution in many technological fields in communications as well as the complexity associated with design decisions stems from the nature of these problems, being multi-criteria, multi-attribute, multi-stakeholders, make the adoption of the preference concept a more viable and promising option. There are many problems where the application of the preference concept can be seen advantageous in modeling and solving the problem. For example, in the multiobjective SDN controllers placement problem, trade-offs between objective metrics are needed [104] [57]. Another example is the resource scheduling problem in mobile networks where multiple users of different requirements and classes of service are being served with limited radio resources and changing conditions.

3 Literature Survey

Pietri and Sakellariou, [91] surveyed the literature addressing the VMP problem, and provided a set of common optimization techniques used to achieve the desired objectives. In this section, we survey existing techniques that incorporate the notion of preference and its variants in the context of the VMP problem.

In the context of the VMP problem, preferences have been tackled by means of hints, soft constraints, soft SLAs, relaxation bounds, weights on the objectives, weights on the constraints, and affinity. One natural classification is to distinguish between preferences that are issued directly by the user (*i.e., *cloud administrator or customer) and preferences that are provided implicitly from a certain measurement (*e.g., *ranking VMs based on current communication cost). The later are related to measurable system attributes. For example, CPU usage index, memory usage index, and inter-VM traffic are operational factors that can be translated into preference relations or rankings. A commonly proposed approach of this type is the affinity-based VMP problem. It adopts the notion of affinity to evaluate which VM is preferred to be hosted on a PM, and which PM a VM prefers [25, 112].

The former preferences are derived from people interacting with the system including cloud administrators and customers. For example, a cloud admin may prefer to assign some VMs to newer PMs due to cost considerations, or a customer prefers to have his VMs located on the same PM or without sharing other VMs [121, 98, 13]. Figure 5 shows the distribution of the approaches from functional and non-functional perspectives.

Table 1 lists some examples of preference uses reported in the recent literature. Clearly, the literature shows a wide variety of applications. For instance, preferences have been utilized to maintain higher degree of non-functional aspects such as SLA satisfaction and availability. Also, they have utilized to represent the desire to assign sub-systems to servers with higher resources, or to servers in certain geographic locations. One notable application is ranking PMs based on their available resources and ranking VMs based on their required resources. Such ranking is often used in the matching algorithms.

In the remainder of this section, we review the literature based on the proposed approaches. We classify existing works into five broad classes of algorithms: Metaheuristics, Matching, MCDM, Heuristics, and others. Table 2 shows an overview of the existing works based on this classification. At the end of this section, we compararatively discuss the advantages and shortcomings of the five classes.

3.1 Metaheuristic-based Approaches

3.1.1 Genetic & Evolutionary Algorithms

Evolutionary Algorithms (EAs) [32] have proven to be a successful approach in solving many optimization problems including the VMP problem. EAs search for solutions by producing a population of candidates in every run. The population evolves by applying crossover and mutation operators. In the past decade, several preference-based EAs have been proposed (see for example [96] for a survey on preference-based EAs) where the goal is to incorporate users’ preferences and desires when solving the problem. Thus, returning solutions that are not only optimized but also preferred by the user.

For VMP, preferences have been expressed as a utility function of the administrator for different combinations of SLA satisfaction, reliability and energy [39] and as tie breaking mechanism for final solutions of the MOP [76]. Also, preferences have been used to correlate two VMs to be placed on the same PM based on historical experience; thus, minimizing number of active PMs [75]. Indeed, one of the most well-suited roles of preferences in EA is to reduce the final Pareto set. Thus, the work of Ihara et al., 2015b [54], Blagodurov et al., [22] and López-Pires and Barán, [77] considered administrator preferences to reduce the final solutions of the VMP problem. A decentralized data center over different local departments have been considered in Saber et al., [103] where it assumes every department administrator has a (possibly conflicting) preference. Alashaikh and Alanazi, [13] considered the case where some VMs have a ranking over which PM to be placed and used a variant of NSGA-II to return preferred solutions based on the notion of ceteris paribus preferences.

3.1.2 Simulated Annealing

Another common technique to solve the VMP problem is Simulated Annealing, a single-based metaheuristic technique to solve optimization problems [60]. A number of proposals have developed simulated annealing-based heuristics to solve the VMP problem. Tsakalozos et al., [120] developed a two-phase VM-to-PM placement scheme in which a group of PMs, termed cohort, are first identified as potential host based on some properties, then the final constrained mapping is resolved taking into account customers’ special requirements and preferences. The goal of selecting a subset of all available PMs in the first phase is to reduce the number of constraints and the search space during the second phase. In addition to resource and cloud specifications constraints, the scheme considers customers’ deployment hints such as colocation of multiple VMs in a single PM and reserving a whole PM for a single VM. The authors proposed two heuristics for filtering cohort PMs and placement based on simulated annealing.

Moreover, Tsakalozos et al., [121] developed a cloud gateway, Nefeli, that takes customers’ deployment hints when mapping incoming VMs to PMs. The hints realize several preferences related to VMs deployment such as colocation, availability, proximity and other information.

Their objective is to optimize cloud energy efficiency and load balancing while meeting cloud admin constraints. At the same time, Nefeli tries to satisfy customers’ preferences but it may only satisfy parts of them and ignore the rest based on the available resources. For evaluating multiple placement solutions, Nefeli adopts a weighted score function on constraints, where a weight derived from customers’ hints is associated with each constraint. The set of constraints is further decoupled into hard and soft constraints. While the first is strict and has to be met, the second only contributes to the score function. Eventually, the optimal solution is defined as the one with the highest score. The algorithm adopts a simulated annealing-based algorithm to solve the placement problem and a search space reduction method was used for large scale problems.

Mahalingam and Sengottaiyan, [79] adopted Nefeli to develop a cloud service selection mechanism that takes into account VMs’ preferences for price and QoS and search for acceptable offers. Lastly, Addya et al., [12] developed a bi-objective optimization model to maximize revenue and minimize energy consumption and solve it using a simulated annealing technique. The proposed model attempts to maximize the revenue and minimize the power budget. A trade-off between the two objectives is maintained based on administrator’s preference.

Table 3 provides a summary to the works presented here.

3.2 Matching

VMP can be naturally described as a matching problem. The set of participants in this matching are often the available VMs and PMs where every participant expresses their preferences over who to partner with. Such preference relations are due to some metrics such as transmission cost or processing capabilities and requirements which can be measured or estimated. The result of the matching is that every VM is placed on exactly one PM. Stability is an important property where a matching is considered stable when no machine (VM or PM) would prefer another machine to its current partner [90].

The literature shows different roles of preferences when matching VMs to PMs. VMs could rank PMs based on some properties such as transmission cost [134], migration time and cost [65] , processing capabilities [136] , and security [20] while PMs rank VMs based on their migration overhead [65] , CPU and resource utilization [136, 29], and risk [89, 20]. Kim and Lee, [66] also considered the case where preferences are not necessarily a ranking and ties are allowed. The goal is often to minimize the overall dissatisfaction score which is defined as sum of ranks. This score can be intuitively seen as the average happiness of VMs and PMs.

Preferences have been utilized to place high throughput applications near computation nodes to minimize the cost of communication [69]. Dhillon et al., [41] addressed consolidating multiple VMs on the same PM while minimizing degradation performance. It assumed that every VM acts as a selfish agent and has a preference over which VMs to partner with on the same PM. The preference is based on some hidden cost function related to the VM. Kella and Belalem, [65] considered the case of multiple cloud providers with intermediate brokers where customers express preferences. The live migration problem has been formulated as a matching problem with the goal of reducing the overall energy consumption without degrading QoS. Cui and Tso, [35] considered networks where the policies, which reside on middle boxes such as firewalls and load balancers, etc., are migrated first then a matching between PMs and VMs based on the communication cost and resource usage is applied. In [74], a descriptive language based on XML for the preferences of data center managers was proposed. The goal is to automate the matching procedure while classifying preferences into two classes; namely, flexible and inflexible. Wang et al., [126] addressed the problem when VMs and PMs cooperate on a mutual objective (reducing the overall energy consumption). Chu et al., [29] added storage PMs as a third set into the matching where storage PMs prefer smaller VMs. Pillai and Rao, [92] considered the case where IaaS request is beyond the capability of a single PM and therefore developed a mechanism to form coalition with the goal of minimizing the total execution time based on uncertainty principle in game theory. The work in [141] addressed the problem of migrating VMs for security preferences to make it harder for attackers to locate targeted VMs. Lastly, Korasidis et al., [68] enhanced the scalability of the stable matching technique for the VMP problem by considering only the top-k alternatives in the preference list. Table 4 summarizes the presented works of this method.

3.3 MCDM

Multi Criteria Decision Making (MCDM) [64] is a field of study to find and analyze preferred options in complex problems under conflicting criteria. Two well-known techniques in MCDM are AHP and TOPSIS [128]. Araujo et al., [17] presented an MCDM method to rank the best cloud infrastructure based on a customized dependability metric and the cost. The customized metric takes customer service constraints such as reliability, downtime, and cost into consideration. The model allows users to express their preferences on cost versus availability, through a planning and analysis tool, to obtain the best configuration for them. Tarighi et al., 2010b [119], Tarighi et al., 2010a [118] presented a method for VM migration between cluster nodes (a group of PMs) using TOPSIS algorithm The aim of their algorithm is to achieve load balancing among PMs by moving load from the most overloaded PMs to the least-overloaded PMs and attempting to minimize migration cost defined in terms of amount of data copied and transferred. The TOPSIS method is a multi criteria decision making technique. It stands for order preference by similarity to ideal solution. It is based on selecting an alternative solution, from a finite set of alternatives, that is closest to the ideal solution and farthest from the negative ideal solution. The ideal solution is the solution that maximizes the benefits and minimizes the cost, whereas the negative ideal solution does the opposite. Typically, each alternative is evaluated with respect to some attributes (decision criteria), and each attribute is assigned a weight based on its importance (*i.e., *given by the user).

The alternatives are ranked according to their relative closeness to the ideal solution. The closer the alternative, the better.

PM nodes are sorted from the highest overloaded to the idle ones based on some criteria such as CPU, RAM, NET, etc. Each attribute is given a weight. Then after finding the most saturated PM, the algorithm tries to find the best VM candidate for migration based on similar criteria w.r.t. VMs. The proposed TOPSIS algorithm can receive three types of information including deterministic, linguistic, and fuzzy information.

Arianyan et al., [18] proposed a TOPSIS algorithm aiming to optimize energy, SLA, and number of migrations in a cloud data center. The algorithm selects VMs to be migrated when overloaded PMs are predicted. It considers a decision matrix of the resource utilizations and capacities of VMs. Each VM is assigned a score for each resource criterion based on the ideal points concept. The total score of a VM is the weighted combination of the scores of each criterion. The weights for each VM are set independently. Similarly, Feng et al., [44] developed a TOPSIS algorithm that incorporates AHP to find weights of coefficients of the resources decision matrix. Rane and Srivastava, [98] introduced a cloud broker architecture to help application customers select their cloud provider considering both functional and non-functional requirements on the service. While functional requirements must be satisfied, non-functional requirements are associated with weights to specify their priority on these requirements. Further, an MCDM constraint significance factor realizing the preference of each constraint is utilized. Then, the method tries to maximize customers’ satisfaction. Kabir et al., [62] proposed a two phase VMP algorithm that first selects a cluster of PMs and then a PM inside the cluster to place a VM. The proposed algorithm is based on the MCDM method PROMETHEE. Overall, the method is based on outranking alternatives. Preferences on each criterion are quantified as real numbers and aggregated onto a weighted preference index. Then, each alternative is evaluated by the number of alternatives that outrank it and the number of alternatives it outranks based on the preference index. The alternative with the largest net outranking is the most preferred one.

Table 5 summarizes the works presented here.

3.4 Heuristics

Shi and Hong, [108] proposed a First-Fit algorithm aiming to maximize the profit without violating the SLA and the power budget. The algorithm considers two kinds of SLAs: the hard SLA and the soft SLA. A hard SLA is not tolerable and violating any of its terms yields no revenue. For a soft SLA, application customers can tolerate performance degradation to some extent and provider revenue suffers certain loss accordingly. The more elastic the VMs are, the more VMs can be consolidated onto a PM. Thus, this elastic model allows providers to manage revenue based on SLA. Moreover, Dow, [42] used modified Next-Fit and First-Fit heuristics to solve the VMP pin packing problem. They proposed to decompose the VM consolidation into two phases; namely, an abstract packing phase that satisfies constraints, and a concrete assignment phase based on preferential colocation and anti-colocation. Alternatively, Georgiou et al., [47] developed an Opportunistic-Fit algorithm addressing the problem of placing VMs with an Infrasturctue-as-a-service (IaaS) request, in which a group of VMs are jointly consolidated in the cloud. Considering DC PortLand architecture, they proposed two VM placement algorithms that exploit users’ hints about the desired features of the requested IaaS. Users specify their requirements on the VMs and their colocation preferences through an XML document. The hints considered here are concerned with the bandwidth needs for specific pairs of VMs and the anti-colocation constraints for pairs of VMs. In the first algorithm, a greedy algorithm places requested VMs as close as possible to each other while satisfying colocation requirements as much as possible; otherwise, a path with sufficient bandwidth is established between them. The second algorithm tries to minimize the time expended for the placement decision by identifying promising neighborhoods of PMs for deploying the IaaS first before applying the greedy placement algorithm.

3.5 Other Methods

In this part, we review several works that adopt exact methods or general customized heuristics.

Viswanathan et al., [125] addressed the placement of High-Performance Computing (HPC) applications in clouds. They proposed a VM consolidation strategy that aims at optimizing resource utilization and energy efficiency while satisfying application’s QoS. A brute-force search algorithm is used to optimize energy efficiency and performance with a parameter utilized to adjust the possible trade-off between them. He et al., [52] proposed a matching algorithm for VMs migration addressing IaaS management in clouds. For incoming IaaS request, the algorithm finds a suitable PM based on available resources. If there are multiple suitable PMs, the system gives preferential attention to more important customers. A PM is recognized by the different properties it has, and a customer may express different preferences over these properties. The preferences are given in values indicating their importance to the customer. Then, the algorithm chooses the most suitable PM that satisfies the preference order the most. Wright et al., [130] developed a two phase resource selection model that can handle a combination of hard and soft constraints. The model intends to enable users to match their application requirements to infrastructure resources. A direct search algorithm is proposed to solve the problem.

Chen et al., [25] considered the problem of mapping virtual CPUs of VMs to physical ones. A proportional scheduler (PS) is proposed to provide fair allocation of CPU resources among virtual CPUs. It assigns a weight to every VM based on its computational requirements. CPU-affinity is defined as a property of a virtual CPU that describes which physical CPU it can run on. In their study, they illustrated that the inappropriate assignment of weights may produce unfair proportional sharing of CPU. Sonnek et al., [112] presented a decentralized affinity-aware migration technique that dynamically migrates VMs based on network affinity between pairs of VMs in order to minimize intra-cloud traffic. Affinity here is defined as VMs interdependency (*i.e., *volume of traffic monitored between pairs of VMs). The authors developed a distributed bartering algorithm that strives to place VMs on a PM that is closer to the data they need. VMs have preference ordering on which PM they migrate to based on the monitored intra-traffic. The VM is responsible for negotiating the placement with PMs which improves the scalability of the algorithm and minimizes the overhead. In a similar vein, Ranjana et al., [99] classify PMs based on CPU and communication bandwidth and then apply affinity-based matching to minimize energy consumption, number of active PMs, and communication cost. Zhou et al., [142] developed three live VM storage migration mechanisms. The mechanisms introduce a trade-off between the efficiency and cost of VM migration. They also consider user experience (IO penalty), cluster management (migration time) and device usage (degree of wear) in the migration decision by utilizing a migration cost (utility function) in which weights reflect different design preferences.

Tantawi, [117] proposed a biased random search method based on biased statistical sampling. It maps VMs and other workloads into PMs and other physical entities so that an objective function is optimized. The algorithm takes into account both administrators’ and users’ requirements and allows users to specify their preferences over the constraints (hard/soft constraints) over some criteria such as location. The biasing pushes the solution towards satisfying constraints and preferences. Unuvar et al., [122] adopted the same method to address VMP in hybrid clouds. The placement decision takes into account whether private or public cloud is preferred for a customer and a customer preference on splitting VMs over private and public clouds. Sotiriadis et al., [113] presented VMs migration within OpenStack cloud platforms considering the FIWARE platform 222FIWARE is developed within the FI-STAR FP7 project that utilizes cloud technologies and IoT to support healthcare services and applications, http://www.fi-star.eu.. The migration tool is developed following the FI-WARE conceptual model and offers a set of software modules, called Specific Enablers (SEs), that allow flexible development of healthcare applications. Within the deployment process of VMs, the system stores and retrieves users’ preferences including geographic location preference. This is done utilizing a graphical interface to configure customer preferences. Chen et al., [26] proposed a Joint Affinity Aware Grouping and Bin Packing (JAGBP) method that aims to minimize application performance cost and maximize resource utilization. The affinity relationships of VMs define colocation and disperse placement preference based on VMs interdependency. They define three affinity types and affinity relations; namely, colocation placement affinity/relation (co-affinity/relation), no-colocation placement affinity/relation (no co-affinity/relation) and fixed placement affinity/relation (fixed-affinity/relation). JAGBP was empirically found to improve the above objectives compared to non-affinity aware placement.

Table 6 summarizes the works presented in sections 3.4 and 3.5.

3.6 Discussion

It is clear from the five classes of approaches mentioned above that there exists different ways to incorporate preferences within the problem with different goals as some methods try to reduce the search space (e.g., bounds on objectives [53], preference order on the assignment [111]) while others generate a single solution or an ordered list of solutions (e.g., most of matching algorithms). Metaheuristics are viable option in tackling complex domains with conflicting objectives. For instance, when reducing the energy while maximizing the revenue during virtualization. However, they are prone to returning suboptimal solutions and require good tradeoff between exploration and exploitation that, in practice, is hard to construct efficiently. In case the preferences are user-centric, *i.e., *they represent provider or user, there is a need for interactive metaheuristics approaches that consult the user during the search process [77].

Indeed, the heuristic approaches require domain knowledge to construct an effective heuristic function with which promising search space is discovered first. Such domain knowledge is sometimes prohibitive when multiple stakeholders’ preferences are involved. One needs to assume preferences follow certain structure and often construct a heuristic function for a simplified version of the original problem. Hence, it is obvious that the proposed heuristic approaches are general and usually domain independent [12].

Most of the matching methods consider resource as the attribute to base their strategy on. This is a reasonable consideration as physical machines have limited resources and utilizing them efficiently when looking for a stable matching between VMs and PMs is of great importance. Stability of the match guarantee the average happiness among all participants, *i.e., *that there exists no pair $(X,Y)$ of virtual machine and physical machine that prefer to be with each other than their current partner (*e.g., *the physical machine hosting $X$, and the VMs placed on $Y$).

MCDM approaches such as AHP and TOPSIS are known to require large number of pairwise comparisons and do not show attribute interdependencies. They are also time consuming and have scalability issues when the number of attributes is large. In such case, preferential dependencies among attributes can lower the time needed to assess preference between alternatives [63].

4 Benefits, Research Challenges, and Trends

4.1 Potential Benefits

After the detailed review of preference applications, here we draw insights regarding the benefits of incorporating preferences into the VMP problem.

User Satisfaction

Indeed one of the main benefits of considering preferences is to find solutions that are not only optimized but also preferred by the user or decision maker. Apart from setting firm and solid requirements, a customer or a cloud administrator can express additional soft requirements through preferences. Considering such preferences would increase the satisfaction of both parties and usually a graphical interface is utilized to specify the preferences [113, 121, 120].

Reducing the Complexity

VMP is a complex problem given all the possible conflicting objectives, constraints from multiple entities, functional and non-functional properties, random nature of workload changes, and managerial and geographical policies. It is remarkably noticed in the literature that considering preferences not only handles users’ and provider’s fine details but also introduces some sort of flexibility to the problem. In addition, the problem of optimizing VMP is known to be NP-complete and even sub-optimal solutions needs to be obtained in a timely manner. Preferences can effectively not only provide but also be the tool to control the margin of flexibility considering performance, efficiency, solution time [101], and violation rate [28].

For example, Ye et al., [138] reported in their study that the number of PMs required to host VMs with colocation constraints decreases when the constraints are softened. In addition, the gain of incorporating preferences improves the scalability to large-scale problems as in [68].

Moreover, preference information can be utilized to set criteria for managing complex processes. As an example, consider the dynamic PM consolidation process. Typically, there exists a trade-off between the goodness of the process and completion time due the complexity of the process [85]. Preferences can be utilized to resolve this trade-off in an efficient manner. In addition, cloud providers typically offer predefined instances of virtual resources with different capabilities as well as value-added features and services (*e.g., *choice of OS, traffic on a private VLAN, etc). The customer choices impact the pricing of such leasing. Wright et al., [130] defined application soft and hard constraints to reduce the complexity of choosing suitable deployment.

Besides, there are cases that inspire to use predefined soft constraints. Hao et al., [51] considered bounded constraint violation to maintain and fix the feasibility of the LP problem under study. The selection of constraints and the extent to which they are softened is critical to the quality of the solution and should be set based on administrator’s preferences rather than arbitrarily.

Reducing Search Space

Multiobjective problems typically obtain multiple Pareto solutions, and the number of these solutions grows significantly as the number of objectives grows. Thus, preferences have been utilized to help mitigate this burden [39, 54, 75, 103, 13, 77]. The same can be said about other algorithms (*e.g., *alternative based algorithms). For example, the work in [120] considered a two phase approach and applied non-functional preferences in the second phase to reduce the number of candidate PMs.

Another example, there can be the case where two or more solutions are available (*e.g., *ties in preference list [90]), and favoring such an option might have some advantages that the DM was not aware of.

Adaptation of Specialized Services

Preferences and soft constraints can be utilized within VMP schemes to enable the hosting and QoS guarantees of specialized services (*e.g., *mapping special services to a PM or a set of PMs). Security concerns [141], NFV chaining [89], HPC placement [125], mobile applications [140], smart TV services [66], and Web services [82] are examples of VMP for specialized services that incorporate the notion of preferences in their approaches.

**Improving Performance **

Preferences can be used as a guideline for optimizing the performance of a system. Such preferences are usually inspired from meta-analysis of the problem and identified as preferred settings or configurations to the machine. The work in [124] for instance, provides a detailed experimental study and identifies the parameters that influence the cost of migrating a VM and provides some recommended rules to improve VM migration efficiency. We expect that these rules (*e.g., *migrate applications with small active memory first, migrate low priority applications first) can suitably be identified as preferences rather than strict constraints or objectives.

Furthermore, constraints and objectives are considered known, hard and strict. preferences are soft, and can be given on top of the problem in a way that does not impact constraints and objectives. However, preferences can be known in advance, known a posteriori, or learned throughout the system’s life. The later can be useful in the VMP context as the system preferences can be learned and adapted to the operational status of the cloud [21]. This gives a wider perspective to administrators on getting to know how different unknown options arise from the problem and to what extent they can change the output [21, 87]

4.2 Trends in the cloud

Software-defined Networking (SDN) and Network Function Virtualization (NFV) are two of the key technologies in the cloud and 5G networks. The SDN architecture provides flexible and programmable network control by decoupling the control plane from the data plane. This enables the use of simple network devices (*i.e., *switches) to create and manage networks for different purposes on the fly via a logically centralized SDN controller.

NFV enables deploying network functions (*e.g., *routing, firewalling, load balancing, etc.) as software applications (*i.e., *VM) stored and run on generic servers (*i.e., *PM). It also introduces flexibility in terms of location, in which functions can be placed anywhere in the network satisfying users and provider requirements. However for a specific task that requires more than function, there is an order sequence that must be followed [71].

The advent of SDN/NFV technologies has facilitated the cloudification of many communication systems [50, 84, 36, 71]. Here we shed light on a number of trends that leverage virtualization and cloud computing technologies in order to improve overall system performance.

Internet of Spectrum Devices

aims to expand the radio spectrum of wireless mobile networks in which devices/elements of the network (*e.g., *base stations) have no restriction on spectrum use and can utilize multiple shared resources of the spectrum dynamically. A cloud is responsible for gathering spectrum status/info. from monitoring devices and distributing them to spectrum utilizing devices. There is a virtual DM for each entity in the spectrum cloud that is responsible for making spectrum decisions. Various cloud computing operations can be applied based on DM request and users’ preferences [132].

Cloud-Radio Access Networks (C-RANs)

offer cloud-based architecture in which all control algorithms are placed and executed by data centers [131].

Specifically, in a C-RAN, traditional base stations functionalities are decoupled into radio functions that are stored in the remote radio unit (RRU) and digital and higher layer functions that are stored in the baseband unit (BBU). The RRU remains at the base station site, while the BBU is moved to the cloud. C-RANs extend the mobile radio access capacity and coverage by pooling BBU resources from multiple cells. These cells can be heterogenous and may belong to multiple collaborating and competing carriers. Following NFV standards, each BBU functionality is represented by one or more VMs which can be enabled or disabled for each base station. Further, these VMs have to be placed in PMs that satisfy their requirements and should be dynamically migrated to where they are needed. The sequence and order of execution of network functions as well as the colocation requirements have to be taken into account when making a placement decision.

Content delivery and Caching

In modern ISP networks, content (videos and cached pages) can be stored in VMs, and the CDN is configured adaptively by SDN controllers to account for traffic loads on the ISP network [129].

Small Cells

One of the challenges to small cell technologies (*e.g., *WiFi and Femtocells) is high-level of interference between cells due to the lack of channel coordination. Cloud-based control is a promising method to manage channel assignment between different, multi-vendor, small cells. There are many other technologies that rely on cloud computing concepts such as mobile cloud, edge Computing, cognitive IoT, etc. In general, cloud-based architectures can provide more flexibility and centralized control and management of the participating entities/devices, in which, coordination and performance can be optimized. Furthermore, proper understanding and translation of the requirements of each VM (*e.g., *application) will lead to desired placement decisions.

VMP with Preferences on the Edge

The problem of solving VMP with preferences on the edge brings several challenges on top of the VMP challenges. One needs compact models to represent and reason about preferences. Such models need to be compact and succinct in their preference language to handle the possibly large number of attributes and limited resources on the edge. Even in the case of successfully stored, preferences on the edge may be insecure for the genuine user’s preferences and selection criteria and, therefore, privacy-preserving techniques for storing and dealing with preferences on the edge are required. Recently, Ouyang et al., [88] proposed an adaptive algorithm for service placement on the edge, which jointly optimizes latency and migration cost, weighted by the on-line learned user’s preferences. Qian et al., [95] developed an privacy-aware algorithm for service placement in edge cloud, and adopting a user preference model to learn about user preferences toward services. In addition, Chiti et al., [27] tackled the problem of virtual function placement in fog-cloud by applying matching game algorithm to fog nodes and virtual functions.

Automation of Cloud Operations

The scale and the frequent changes of cloud data center networks and configurations necessitate faster handling of cloud operations (*i.e., *service management, deployment, and configuration). This includes the provisioning of resources and instantiation, configuration, and delivery of services at acceptable service level and without human intervention. Thus, orchestration and automation of cloud services have been of high interest to both academia and industry which gave rise to new concepts addressing these concerns. For instance, Zero-Touch networking is an emerging concept that leverages SDN and NFV capabilities and aims to automate cloud-based/virtual networks management and operation. A related emerging concept is intent-driven networking. It is an end-to-end network configuration automation solution that aims to optimize performance and user experience. It leverages big data and AI technologies to achieve such a goal. Preferences and non-functional parameters can be a helpful tool in the design of automated control algorithms. [106]

4.3 Limitations and Research Opportunities

Compactness and Cognitive Burden

As the number of alternatives becomes large, direct assessment of preferences becomes problematic. VMP indeed has huge space of possible alternatives and directly evaluating them would be a demanding cognitive task. For example, consider a customer expressing her preferences for the placement over 10 PMs. She needs to express her preferences in ordinal way through pairwise comparisons of every pair of PMs. This is indeed problematic as the users would lose focus let alone being cognitively able to accurately specify their preferences.

The vast majority of VMP approaches assume the existence of a utility function. Accurately specifying utility functions by end-users has been shown to be a difficult and tedious task [46]. In many situations, users cannot afford to specify more than qualitative statements of the form “$x$ is better than $y$". The literature lacks concrete attempts that address user’s policies and preferences qualitatively. Furthermore, it is usually assumed that a utility function of one attribute is independent from other attributes. This assumption is too strong to be met in practice as the utility of one attribute may depend on other attributes’ values. Generalized Additive Independence Networks (GAI-net) [48] weaken this assumption by allowing the utility function of one attribute to depend on other attributes. On the qualitative side, Conditional Preference Networks (CP-nets) [23] are a prominent graphical model to represent conditional preferences.

Such compact models have been proposed in the last decade to avoid the cognitive burden of specifying preferences in multiattribute domains. They require reasonable effort from the user and adopt the conditional preferential independency notion. This allows for the decomposition of the preferences into smaller but compact components while being expressive enough to capture many scenarios in practice [64, 16].

Uncertainty, Vagueness, and Risk

Recently, there have been several works on dynamic and uncertain VMP [78, 24, 53, 80] where workloads and operational statuses may change frequently over time. However, it is important to tackle the problem from the decision maker perspective and consider preferences in uncertain situations. Current approaches usually assume preferences occur with certainty where, in practice, preferences occur with uncertainty over the set of possible alternatives. This is evident in VMP as customers are unaware of possible complexities on the cloud. In such cases, preferences are no longer deterministic choices but rather a lottery over the alternatives. A lottery is a probability distribution over alternatives [64]. Preferences are then represented over lotteries and known concepts such as expected utility can be used to rank the alternatives.

Learning and Revision

Machine learning has been recognized as an enabler for building and designing cloud data centers [73, 70, 107]. The VMP problem is dynamic in nature and preferences often change over time. This urges for approaches to not only handle uncertain preferences but also learn to revise them in light of new information. Instead of insisting on having the preferences upfront, it is better to elicit them during the solving process. Such elicitation holds the promise of proposing interactive learning strategies that consider humans in the loop when solving the VMP; thus, lowering the cognitive burden in specifying preferences. Another approach is to learn customers’ and administrators’ preferences from historical data (*i.e., *learning preferences passively from past deployments).

Group Decision Making

The VMP problem is inherently a group decision making problem. A group of customers and cloud administrators issue their preferences on different aspects of the problem. It is likely that the issued preferences are conflicting with each other and further aggregation and reconciliation methods are required. For instance, consider the scenario of Ilkhechi et al., [55] where each group of VMs relies heavily on some PMs (named sinks). These sinks (*e.g., *supercomputers) are a subset of the PMs and it is likely that customers’ preferences would be conflicting on how to utilize them. Thus, the aggregation of preferences needs to ensure certain properties like fairness of the deployment among customers.

5 Conclusions

Virtualization technology has enabled cloud data centers to become cost effective, flexible and customizable to host diverse enterprise services and applications at different scales. The VMP is a problem of significant importance to the cloud design for sustaining its profit and popularity. The literature shows a wide variety of approaches to solve the problem and its variants. The problem is also full of preferences of the involved parties including cloud administrators and customers. Therefore, the proper handling of preferences is crucial to the success of proposed approaches.

In this paper, we discussed the current approaches for handling preferences in the context of VMP. We highlighted certain limitations and identified research opportunities. Analyzing the VMP problem from a decision making point of view gives rise to several interesting research challenges. Adopting compact models of preferences such as CP-nets and GAI-net is particularly promising given the large number of attributes involved in typical VMP problems. This has the potential of reducing the cognitive overhead of the problem; thus, increase its applicability. Another important research to be explored is interactively learning the constraints and preferences of the cloud admin and/or customers as well as passively learning the soft constraints among VMs and PMs from historical data. Lastly, proposals based on group decision making techniques hold the promise of solving the problem taking everyone’s preferences in a fair and transparent manner. Addressing such issues may lead to the development of user-centric VMP that is widely adopted by the stakeholders.