TECHNICAL FIELD

The invention relates to a method for allocating cloud computing resources, and a scheduler server, a scheduler system, a computer program and a computer program product therefore.

BACKGROUND

Cloud computing refers to a way of managing hardware equipment, making it easy to segment hosts, storage areas and network links. All the segmentations available for each of those technologies are not directly dependent on the cloud but they are heavily used in cloud environments in order to get the maximum utilization rate. Increased resource-usage efficiency and reusability of available hardware makes cloud computing a very attractive solution for many companies nowadays. Therefore, cloud computing can be thought of as a way of managing hardware equipment to aggregate capacity and segment it in order to give service to multiple users.

There are three main resources required to achieve cloud computing:

compute resources, networking resources and storage resources.

The compute resources represent the computing power available in the cloud. The compute resources give the user the ability to define the number of resources needed to fulfill his/her query in terms of Central Processing Unit (CPU) and Random-Access Memory (RAM).

The networking resources represent the networking side in a deployment. The networking resources are currently developing rapidly and it allows cloud administrators to configure not only the virtual networking

deployment but also the physical networking deployment. Software-Defined Networking (SDN) may be used for networking deployment.

The storage resources represent the storage part in the running resources, i.e. the virtual machines. The storage part in the running resources correspond to the disk and/ or memory blocks used by the virtual machines or any kind of managed object using the persistent storage in the cloud. The storage resources can usually be configured to perform redundancy or distributed storage across different hosts.

Cloud computing components usually make use of different drivers, such as hypervisors and virtual switches, installed on hosts. A hypervisor or virtual machine monitor (VMM) is a piece of computer software, firmware or hardware that creates and runs Virtual Machines (VMs). A virtual switch is a logical switching fabric built into a VM infrastructure so that the Virtual Machines (VMs) can be networked wherever you need them. Even though the majority of cloud deployments use virtualization of resources, virtualization is not strictly necessary and hence, cloud computing and virtualization can be de-coupled. Cloud computing can run in bare-metal in some occasions in order to avoid virtualization overheads and achieve better performance using other kinds of isolation systems like containers. Virtualization can also be used in other contexts than cloud computing.

Both mechanisms, containers and hypervisors, are different methods for achieving segmentation and better resource usage efficiency in the cloud infrastructure. Both containers and hypervisors are mainly used to achieve segmentation in order to divide the resources and achieve much higher usage-efficiency of the available physical resources, and also to achieve isolation of compute, networking and storage resources so these resources can be seen as treated as completely independent machines even though they are sharing the physical resources underneath.

Resource scheduling is a known problem in cloud computing. Resource schedulers are a part in the infrastructure that is aware of the available resources and responsible for deciding where to assign new processes or virtual services or VMs upon request.

The patent application PCT/EP2014/079143 discloses a method of allocating cloud computing resources in a cloud computing environment to a user. The method comprises determining a prediction accuracy score for each user indicative of their user predictability, and allocating cloud computing resources to each user dependent on their user predictability,

SUMMARY

It is an object of the invention to enable improved resource scheduling in a cloud computing environment, for efficient utilization of available computing resources.

According to a first aspect, it is presented a method for allocating cloud computing resources in a cloud computing environment, which method is performed by a scheduler server. The method comprises the steps of: causing determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization; and causing allocation of a cloud computing resource in dependence on the determined weight. A weight representing used cloud computing resources for a host is thus set in dependence on both current cloud computing resource utilization and future predicted cloud computing resource utilization for this host.

The method may further comprise the steps of: causing prediction of a resource utilization for each of a plurality of cloud computing recourses; and causing aggregation of the predicted resource utilizations for each of the plurality of hosts controlling the plurality of cloud computing resources. The aggregation may comprise, at each of the plurality of hosts, summing the predicted resource utilization for each cloud computing resource of the plurality of cloud computing resources.

The weight for each of the plurality of hosts maybe calculated by adding a predicted resource utilization to a current resource utilization, wherein the predicted resource utilization has a reduced significance compared to the current resource utilization. The weight may also comprises a plurality of predicted resource utilizations for a plurality, i.e. at least two, of time instances, and a prediction further into the future has a reduced significance compared to a prediction nearer into the future. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may further be set in dependence on accuracy of the predictions. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may also be dynamically set by machine learning by building a model and extracting each predicted resource utilization of the plurality of predicted resource utilizations through ensemble learning.

The cloud computing resources may comprise one or more of the following types: core of a central processing unit (CPU core), memory, disc usage, incoming network traffic, outgoing network traffic, and stability of a user's utilization of the cloud computing resources.

The method may further comprise the steps of: causing prediction of a resource utilization for each of a plurality of a first type of cloud computing recourses; causing aggregation of the predicted resource utilizations of the first type for each of the plurality of hosts controlling the plurality of cloud computing resources; causing determination of a first weight for each of the plurality of hosts, in dependence on current resource utilization of the first type and predicted resource utilization of the first type; causing prediction of a resource utilization for each of a plurality of a second type of cloud computing recourses; causing aggregation of the predicted resource utilizations of the second type for each of the plurality of hosts controlling the plurality of cloud computing resources; causing determination of a second weight for each of the plurality of hosts, in dependence on current resource utilization of the second type and predicted resource utilization of the second type; and causing determination of a total weight for each of the plurality of hosts, in dependence of the first weight and the second weight.

The method may yet further comprise the steps of: causing collection of historical data for a resource utilization for each of a plurality of cloud computing recourses; and causing training of a prediction model in dependence on the historical data. The prediction of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may be caused to be performed distributed in one of at least one virtual machine of the host of the plurality of hosts. The prediction of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may also be caused to be performed centrally in the scheduler server.

According to a second aspect, it is presented a scheduler server configured to allocate cloud computing resources in a cloud computing environment. The scheduler server comprises: a processor; and a computer program product storing instructions that, when executed by the processor, causes the scheduler server to: cause determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization; and cause allocation of a cloud computing resource in dependence on the determined weight. The instructions may further cause the scheduler server to: cause prediction of a resource utilization for each of a plurality of cloud computing recourses; and cause aggregation of the predicted resource utilizations for each of the plurality of hosts controlling the plurality of cloud computing resources. The aggregation may also comprise, at each of the plurality of hosts, summing the predicted resource utilization for each cloud computing resource.

The weight for each of the plurality of hosts maybe calculated by adding a predicted resource utilization to a current resource utilization, wherein the predicted resource utilization has a reduced significance compared to the current resource utilization. The weight may also comprise a plurality of predicted resource utilizations for a plurality of time instances, and a prediction further into the future has a reduced significance compared to a prediction nearer into the future. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may be set in dependence on accuracy of the predictions. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may be dynamically set by machine learning by building a model and extracting each predicted resource utilization of the plurality of predicted resource

utilizations through ensemble learning.

The cloud computing resources may comprise one or more of the following types: core of a central processing unit (CPU core), memory, disc usage, incoming network traffic, outgoing network traffic, and stability of a user.

The instructions may further cause the scheduler server to: cause prediction of a resource utilization for each of a plurality of a first type of cloud computing recourses; cause aggregation of the predicted resource utilizations of the first type for each of the plurality of hosts controlling the plurality of cloud computing resources; cause determination of a first weight for each of the plurality of hosts, in dependence on current resource utilization of the first type and predicted resource utilization of the first type; cause prediction of a resource utilization for each of a plurality of a second type of cloud computing recourses; cause aggregation of the predicted resource utilizations of the second type for each of the plurality of hosts controlling the plurality of cloud computing resources; cause determination of a second weight for each of the plurality of hosts, in dependence on current resource utilization of the second type and predicted resource utilization of the second type; and cause determination of a total weight for each of the plurality of hosts, in

dependence of the first weight and the second weight.

The instructions may further cause the scheduler server to: cause collection of historical data for a resource utilization for each of a plurality of cloud computing recourses; and cause training of a prediction model in dependence on the historical data. The prediction of a resource utilization, for each of a plurality of cloud computing resources, for a host of the plurality of hosts may be caused to be performed distributed in one of at least one virtual machine. The prediction of a resource utilization, for each of a plurality of cloud computing resources, for a host of the plurality of hosts may be caused to be performed centrally in the scheduler server. According to a third aspect, it is presented a scheduler system configured to allocate cloud computing resources in a cloud computing environment. The scheduler system comprises at least two nodes interacting with each other, and each node comprising a processor and a computer program product. The scheduler system stores instructions that, when executed by the processors, causes the scheduler system to: cause determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization; and cause allocation of a cloud computing resource in dependence on the determined weight. According to a fourth aspect, it is presented a scheduler server for allocating cloud computing resources in a cloud computing environment. The scheduler server comprises: a determination manager for causing determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization; and an allocation manager for causing allocation of a cloud computing resources in dependence on the determined weight.

The scheduler server may further comprise: a prediction manager for causing prediction of a resource utilization for each of a plurality of cloud computing recourses; and an aggregation manager for causing aggregation of the predicted resource utilizations for each of the plurality of hosts controlling the plurality of cloud computing resources.

According to a fifth aspect, it is presented a computer program for allocating cloud computing resources in a cloud computing environment. The computer program comprising computer program code which, when run on a scheduler server or scheduler system, causes the scheduler server or scheduler system to: cause determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource

utilization; and cause allocation of a cloud computing resources in

dependence on the determined weight. According to a sixth aspect, it is present a computer program product comprising a computer program and a computer readable storage means on which the computer program is stored.

Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is now described, by way of example, with reference to the accompanying drawings, in which:

Fig. l is a schematic diagram illustrating a method for allocating cloud computing resources;

Fig. 2 is a schematic diagram illustrating an environment where

embodiments presented herein can be applied;

Figs. 3A and 3B are schematic diagrams illustrating embodiments presented herein; Figs. 4A-4C are flow charts illustrating methods for embodiments presented herein;

Fig. 5 is a schematic diagram illustrating a method for an embodiment presented herein;

Fig. 6 is a schematic diagram illustrating weight calculation according to an embodiment presented herein;

Fig. 7 is a schematic diagram illustrating users sorted based on predictability of their behavior; Fig. 8 is a schematic diagram illustrating some components of a scheduler server; and

Fig. 9 is a schematic diagram showing functional modules of a scheduler server. DETAILED DESCRIPTION

The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.

Making good decisions in allocation of cloud computing resources is important for maintaining a proper load on a cloud infrastructure and to prevent uneven distribution of resources in managed hosts (e.g. a cluster of hosts/ a server farm) forming the cloud infrastructure. In a normal cloud cluster the aggregated number of Central Processing Units, CPU, cores or disk capacity can reach very high levels and achieving fairness at the CPU level and a shared load is critical for good performance and a good return on performance. A host is a physical machine/host.

An allocation decision is usually made taken into account two main factors, available resources and special requirements.

Available resources: this takes into account the amount of resources currently available on the hosts and the amount of resources needed to fulfill the incoming query. Common resources are CPU, RAM, network bandwidth and disk capacity.

Special Requirements: it is common to divide a cloud cluster into different areas, either because of the type of hardware, different bandwidth availability or even geographical distribution of the infrastructure. These special requirements may e.g. be given in order to ensure certain quality. A need to maintain a maximum delay can e.g. force the cloud to allocate resources close to a request source. These special requirements may also be given to ensure that a virtual machine (VM) is located in a specific host (i.e. explicit host selection), e.g. to allocate a VM in an area near an office, or to ensure that certain disks have replication, when some disks in the cloud may not have replication.

As the cloud was designed to be hardware agnostic, a cluster can run with different kind of hardware which creates a need to evaluate what the capacity is for connected hardware. Moreover, the load on each host (hardware) can be different as a VM running on top may have different traffic, CPU or disks loads. Finally different applications can have loads that conflict with each other, and collocating two such applications on the same host can lead to degraded performance.

This situation puts lot of requirements to achieve an efficient scheduling system in a cloud. A scheduler needs to be aware at any time what the situation is for the cloud computing resources to be able to efficiently decide where to allocate new virtual resources. In an Openstack scheduler, as illustrated in Fig. 1, the scheduler follows a weight mechanism in order to decide where to allocate newly incoming requests. It creates a weight for each of the hosts connected to the cluster and takes into account the currently consumed resources by the virtual resources allocated on it. In this way these resources are ordered in a prioritized list of hosts. The scheduler can then allocate the requested resources and comply with the available ones on a given host.

Six different hosts, 1-6, are in this example available from a pool of hosts, for the scheduler. A function cost is used to calculate currently consumed (i.e. used) resources at each host, which then is used to calculate a weight for each host. Each calculated weight is a sum of the costs for a host, and thus represents total consumed computing resources for a host. In this case, host 1 receives a weight of 12, host 2 receives a weight of 87, etc. The scheduler thereafter orders/ranks the hosts in dependence of weight in a sorted list of hosts. Host 4 with the lowest weight of 10, i.e. least consumed resources, is ranked highest. The scheduler can then allocate newly requested resources primarily to host 4, to provide even distribution of load between the hosts 1- 6.

Moreover, an Openstack scheduler has a mechanism to handle special requirements, as explained above, and it is done through filters. Current schedulers are trying to optimize the load and maintain even resource utilization all over the cluster. For either a centralized solution or a distributed solution, the idea behind is the same: take the currently available resources and calculate the best place to allocate new queries.

However, this principle is still not optimal in many cases due to the fact that loads are not always the same and, that it is common to oversubscribe resources in order to deliver on the resource-usage efficiency.

Oversubscription is a common problem in virtualized environments. As a VM will not always be using 100% of its resources, it is common to allocate more virtual resources than available physical resources. Oversubscription on the network at 100:1 on bandwidth is common. As a result if all VMs start to increase resource usage, it may cause misbehaviors or malfunctioning at some point.

The problem of oversubscription is a side effect of trying to exploit to the maximum available cloud resources, and to increase overall usage. The solution presented herein considerably enables reduction of the risk of oversubscription misbehaviors appearance over time, in addition to improving resource utilization.

Current scheduling schemes do not take in account whether a host is being underused or not at a specific instant "t" and whether there is a high likelihood or not that it will be heavily loaded in e.g. "t + 5" minutes. This is not possible as there is no historical record information of the resource allocator decisions. As a consequence, even if a decision is right at that instant "t", that decision can be a bad one at "t + 5" minutes or in the longer run. Further, usage patterns for users of a cloud can be used to classify users into different groups. Examples of groups are a first group associated with very dynamic users and a second group associated with stable users.

Very dynamic users are users that create, delete, start, and/ or stop resources in the cloud infrastructure very often. This usage pattern is not very predictable.

Stable users do usually not change their running setup very often, i.e. runs with very predictable pattern regarding cloud computing resource usage.

A classification can be done by looking at the predictability for each user. This means that a prediction score is calculated for each user that describes how accurate this user's behavior can be predicted in the cloud.

The prediction scores for all users are then sorted from high to low. This is illustrated in Fig. 7, which contains 18 users (each bar in the histogram) and their prediction score. Based on this sorting a number of user classes can be defined. This may for instance be "high", "med" and "low". The number of classes and the range for each class is configurable. A high predictability class may be used define a stable user. A low predictability class may be used to define a very dynamic user. A med predictability class maybe used to define users having a predictability being between stable users and very dynamic users. Details for implementation of classification of predictability of users can be found in patent application PCT/EP2014/ 079143.

The user classes may then be used as input to a policy framework that decides how new resources can be distributed, or how existing resources can be redistributed. The input configuration may be: number of clusters and ranges that each cluster should represent. Examples of policies can be to ensure a certain mix of different user groups utilizing resources on the same physical hardware with the aim of ensuring good resource utilization, or to separate highly dynamic users and put their resource utilization on an own hardware so that they do not interfere with resource utilization from more stable users/ running applications.

Monitoring of the cloud infrastructure is about understanding the current status of the infrastructure as such as well as the applications running on top. The metrics of interest are usually divided into four different categories;

compute, memory, network and storage. Each category can relate to metrics either obtained at the hypervisor or VM level.

In the following, metrics are exemplified in each individual category. Note that each metric can either be measured for the whole cloud computing system, for the hypervisor or for each individual VM executing in the cloud computing system. The compute metric category is mainly related to the CPU or Graphics Processor Unit, GPU, of the system. Specific metrics include: total CPU utilization, CPU utilization for user level processes, CPU utilization for system level processes, CPU waiting time for I/O operations to finish, and affinity per CPU and core. The Memory metric category includes the following examples: total installed memory in system, free memory, total swap memory, free swap memory, and memory utilization.

For the network metric category, metrics are mainly exemplified that can be measured within a hypervisor or VM. The metrics include: total number of packets transmitted, total number of packets received, total number of bytes transmitted, and total number of bytes received.

The storage metric category includes metrics such as: available disk space, storage I/O operations per time unit, number of disk reads per time units, number of disk writes per time units, and disk idle time. Applying analytics and machine learning to the existing allocation solutions makes it possible to detect and determine trends in the physical resource usage. These trends can then be used to estimate the resource usage at a certain point in the future. Therefore, methods for allocating cloud computing resources presented herein combine the current situation analysis with a predicted resource utilization analysis. Combining the two models will provide much more accurate and appropriate decisions.

The solution presented herein substantially amplifies the information available to the algorithm used in order to decide and achieve an optimal resource allocation in the cloud infrastructure. By having knowledge and models on how the infrastructure is behaving now, and a trend/prediction with a high level of confidence of how the infrastructure will behave in the future can drastically change the decisions taken by a scheduler. These scheduler decisions may thus achieve a reduced risk of

oversubscription, a better load distribution, and an ability to classify cloud users.

Reduced risk of oversubscription: this problem may not be completely solved with this solution but it will at least reduce the occurrence of it. The solution presented herein will further ensure that even though there is an

oversubscription on a resource, the load patterns of virtualized applications do not collide and misbehave.

Better load distribution: load values should be compared in the long term. A resource scheduler presented herein will try to impose even load in the machines but by taking into account current and future load values. As a result, current load peaks will not interfere that much with an allocation decision taken and will much better manage the situation of allocation of resources in the long term. The ability to classify cloud users: provides further information about the quality of the prediction by the stability of cloud user usage, but may also provide some characteristics of the associated load.

An overview of a cloud computing system according to an embodiment is illustrated in Fig. 3A. The figure shows the following key components.

Virtual Machines (VMs) from where current resource utilization is collected.

Resource predictor VM's running on each host predicting resource utilization for a period X in the future. Even though one prediction VM per host is illustrated, there maybe e.g. one prediction VM for every n Virtual Machines running on the host.

Hosts 23a-23c, each including a hypervisor hosting the virtual machines.

Current resource utilization for each host is collected centrally in a scheduler server and is aggregated for total current resource utilization.

Predicted resource utilization for each host is collected centrally in the scheduler server and is aggregated for total predicted resource utilization from each resource predictor as well as classification of usage.

The scheduler in the scheduler server takes total current resource utilization as well as total predicted resource utilization as input to make scheduling decisions. Application Programming Interface, API, in the scheduler server may be used to for instance to instantiate, start, stop, and delete virtual machines.

Fig. 3A thus illustrates how the cloud computing system can be deployed in a distributed manner regarding resource prediction. It is however also possible to deploy the cloud computing system in a more centralized manner, with a separate system responsible for doing predictions. This is illustrated in Fig. 3B. The main difference with the distributed deployment illustrated in Fig. l6

3A is that all data is sent from each host to a centralized system having a resource predictor that will do the predictions for each host.

Examples of cloud computing resource, which are monitored and used by the scheduler server, are CPU (each core), memory, disc usage, incoming network traffic and outgoing network traffic.

Predicted resource utilizations can be used to improve scheduling decisions. CPU utilization will be used as an example on how this works. The method is however also applicable to other metrics such as memory, disc usage, disc IO, network bandwidth, incoming network traffic and outgoing network traffic. The main process according to an embodiment is illustrated in Fig. 5 for a single metric (CPU).

Data Collection

Data needed for the predictions of future cloud computing usage are collected. This includes the metrics mentioned earlier (CPU, memory, network traffic, etc.). The amount of data needed depends on how far in the future predict are to be made. The longer into the future that predictions are to be made, the more historical data is needed.

This data is first of all used to train a prediction model that is specific for the particular virtual machine and its usage pattern, and secondly to get predictions once the model is ready. The training may be performed by machine learning, e.g. supervised, unsupervised or a mix of supervised and unsupervised machine learning such as RandomForest, neural networks or deep learning. The training extracts statistical information or patterns to build up the prediction model. There will be a prediction model per VM and per resource type (CPU, network traffic, memory, etc.).

Fig. 2 is a schematic diagram illustrating a cloud computing environment 21 where embodiments presented herein can be applied. The cloud comprises multiple cloud computing resources 22a and 22b, illustrated as resources A and B. The cloud further comprises a scheduler server 20, and users A and B each using the cloud through user equipment or communication devices. The equipment or devices of users A and B will be allocated cloud computing resource A or B dynamically, controlled by the scheduler server 20.

Monitoring of resources (both virtualized and physical) in cloud computing environments is a relatively developed area, with production-grade

environments operating in large public and private clouds for several years. Amazon CloudWatch and VMware vCenter Hyperic are representative toolsets for the public and private cloud monitoring, respectively. For a general better understanding of the cloud computing environment,

CloudWatch, Hyperic and Openstack Telemetry are shortly described below.

CloudWatch monitors resources at Amazon, currently the largest provider of public cloud infrastructure. CloudWatch is made available in an as-a-service manner. It includes a large number of pre-defmed metrics at both application and infrastructure level. The users have a limited capability to define their own metrics, and billing depends on the frequency of the monitoring interval.

Hyperic is part of the VMware cloud management suite. It provides performance monitoring for physical and virtual infrastructure, middleware and several enterprise applications using a combination of agentless and classical agent-based monitoring. Significant capabilities are the auto- discovery of key properties for newly-created VMs and automatic

configuration of monitoring functionality. Hyperic also has the capability to copy and re-use monitoring configurations and alert policies, speeding up the deployment of monitoring capabilities. Hyperic is part of the vCenter

Operations Enterprise framework, which allows determining dynamic thresholds based on hourly-observed behavior of a performance metric in correlation with other metrics.

The Openstack Telemetry component, also known as Ceilometer, implements infrastructure level monitoring in cloud environments based on the

OpenStack platform. It collects monitoring information from compute, network and storage resources in Openstack-managed environments. It expects the resource managers to publish monitoring information through an l8

Oslo messaging bus, but push and pull agents that communicate directly with the resource managers are also supported. Ceilometer has no capabilities of monitoring virtual network functions, which are regarded as applications from its perspective. Ceilometer offers a REST API to access monitoring information once it is stored in the database. Ceilometer is integrated with Heat, the orchestration component of Openstack, and provides input data for performance-triggered auto-scaling rules.

Prediction

The resource predictor is based on an earlier trained model to predict the future resource usage based on the current load. With the understanding that the bandwidth usage may be interpreted as a time-dependent function, neural networks may e.g. be used to predict bandwidth usage. The output for the prediction will be predicted resource usage for each resource metric collected. The time period that is possible to predict is dependent on how long the virtual machine has been running, which means that possible prediction time gradually can be increased/decreased while more data is collected and modeling of the resource usage is improved. The time period considered in order to achieve good predictions is adjusted to the particular resource and VM over time.

Aggregation

The aggregation part will take the predicted resource metrics and sum them up for each physical resource:

Host 1: Predicted CPU = SUM(predicted CPU VMi + predicted CPU VM ₂ +...+predicted CPU VM _n )

Host 2: Predicted CPU = SUM(predicted CPU VMi + predicted CPU VM ₂ +...+predicted CPU VM _n ) The aggregation part needs to take into account the physical setup and how virtual machines are allocated to the hosts. This is handled dynamically.

Scheduling

The scheduler takes the current resource utilization, in this case CPU utilization as well as the predicted CPU utilization as input when making scheduling decisions.

A weight W for each host is then calculated according to W = a*X + bi*Yi + b2*Y2 + ... + bn*Yn, where: a = weight for current resource utilization, X = current resource utilization, bn = weight for predicted resource utilization for each time instance that is included. This means that higher values on n predict further into the future, and

Yn = predicted resource utilization. Higher value on n means predictions further into the future.

The weights a and bn are dynamically set based on how good/accurately the prediction model captures the resource utilization. Intuitively this means that the predicted resource utilization Yn will get a higher weight bn when the predictions are working well. This can for instance be when we have stable and long-running services that have a certain regularity in its resource utilization data pattern.

The predicted resource utilization Yn will get a low (or o) weight bn in cases where services/ applications are behaving very unpredictable with sudden peaks etc. The process to determine a and bn can be done in multiple ways: i) The accuracy of the prediction model may be used to set a and bn, i.e. high accuracy means higher weight on the predicted values, and vice versa, low accuracy means lower weight on the predicted value. The accuracy can also determine how far into the future we want to include in the weight function. Some models may be good at predicting X minutes into the future and terrible at doing long-term predictions, and vice versa.

2) Machine Learning may be used to dynamically decide the parameters a and bn. This means that the weights in the weight function are unknown and that they are determined by using for instance ensemble learning.

The processes l) and 2) above are used to determine the importance of future samples, i.e. how much importance to be given to the predicted samples (parameters bn that are multiplied by the predictions Yn and give the weight

W).

The machine learning takes into account how stable a host is so it can give more or less importance to the future predicted samples. However, if the host is unstable, the machine learning algorithm will give very little importance to the future samples as they might not be reliable. The difference between the two processes above is that in the first one, the weights are static (for example the a and bn coefficients may be a negative exponential function), while in the second one, the coefficients are dynamic and depend on the accuracy of the predictions done before.

In addition, classification of predictability of users provide an additional input that helps to train the prediction model by taking in account the probable usage of this load, and the expected quality of the usage prediction. Users may be separated into stable users and very dynamic users. A stable user resource usage and main resource allocation is thus known, and contributes to weighting the proper resource. The classification can also be used to know if the load will be running long-term or short-term. As an example, wherein the predicted usage is very short-term, then prediction based on predictability of users may not be necessary.

The process to calculate a total weight for each host is illustrated in Fig. 6. The scheduler can then, based on these total weights, schedule new requests to the host with the lowest weight. In comparison with the weight calculation illustrated in Fig. l, the method presented herein adds a predicted utilization to the current utilization, and determines a weight function to utilization, before applying a weight calculation based on costs. In other words, current schedulers base their decisions on the compute load that is being used at the specific point in time when a request is received to do some operation, whereas the solution presented herein additionally takes into account historical information about the load on the system and use it in order to decide where to allocate new virtual resources.

A method is thus presented that based on the previous load metrics collected from the cloud infrastructure is capable of predicting what the usage will likely be in the near future and take it into consideration in order to select a better allocation in the cloud infrastructure.

The correct placement of virtual resources is critical for a good functioning of the cloud infrastructure as it can significantly improve the performance and resource usage efficiency. Furthermore, it will play a crucial role in a telecommunication cloud, or telco cloud, as this type of infrastructure will have to deal with different application/services running on top, diverging from the Information Technology, IT, cloud environments that currently exists. The telco cloud will in addition to resources such as computation, storage and network also provide base stations, the evolved packet core and datacenters of various sizes.

Traditional telecommunication services such as the mobile base station, the mobile core and Internet Media Services (IMS) will run as virtualized services on top of the Telco cloud. This concept is often referred to as network function virtualization (NFV) and is being defined in standard groups such as ETSI GS NFV. The group has defined and described use cases for various telecommunication functions, how to virtualize the functions and how they can coexist with the non-virtualized world. The main differences between a standard cloud and the Telco cloud can be summarized in higher demands on Service Level Agreements (SLAs) related to reliability and performance, a more diverse execution environment and regulatory requirements. This involves continuous monitoring of the relevant Key Performance Indicators (KPIs) relating to the specific SLA for the service, analyzing the data for finding abnormal trends and anomalies and triggering the suitable cloud orchestration actions in case of any violations.

A method, according to an embodiment, for allocating cloud computing resources, is presented with reference to Figs. 4A-C, which method is performed by a scheduler server 20. The method comprises, which is illustrated in Fig. 4A, the steps of causing determination 42 of a weight for each of a plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization and predicted resource utilization; and causing allocation 43 of a cloud computing resource 22a, 22b in dependence on the determined weight.

The method may further, which is illustrated in Fig. 4B, comprise the steps of: causing prediction 40 of a resource utilization for each of a plurality of cloud computing recourses; and causing aggregation 41 of the predicted resource utilizations for each of the plurality of hosts controlling the plurality of cloud computing resources. The aggregation may comprise, at each of the plurality of hosts, summing the predicted resource utilization for each cloud computing resource.

The weight for each of the plurality of hosts maybe calculated by adding a predicted resource utilization to a current resource utilization, wherein the predicted resource utilization has a reduced significance compared to the current resource utilization. The weight may comprises a plurality of predicted resource utilizations for a plurality of time instances, and a prediction further into the future has a reduced significance compared to a prediction nearer into the future. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may be set in dependence on accuracy of the predictions. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may be dynamically set by machine learning by building a model and extracting each predicted resource utilization of the plurality of predicted resource

utilizations through ensemble learning.

The cloud computing resources may comprise one or more of the following types: core of a central processing unit (CPU core), memory, disc usage, incoming network traffic, outgoing network traffic, and stability of a user' s utilization of the cloud computing resources.

The method may further, which is illustrated in Fig. 4C, comprise the steps of: causing prediction 40 of a resource utilization for each of a plurality of a first type of cloud computing recourses 22a, 22b; causing aggregation 41 of the predicted resource utilizations of the first type for each of the plurality of hosts 23a, 23b, 23c controlling the plurality of cloud computing resources 22a, 22b; causing determination 42 of a first weight for each of the plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization of the first type and predicted resource utilization of the first type; causing prediction 40of a resource utilization for each of a plurality of a second type of cloud computing recourses 22b, 22a; causing aggregation 41 of the predicted resource utilizations of the second type for each of the plurality of hosts 23a, 23b, 23c controlling the plurality of cloud computing resources 22b, 22a; causing determination 42 of a second weight for each of the plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization of the second type and predicted resource utilization of the second type; and causing determination 45 of a total weight for each of the plurality of hosts 23a, 23b, 23c, in dependence of the first weight and the second weight. The method may further comprise the steps of: causing collection of historical data for a resource utilization for each of a plurality of cloud computing recourses; and causing training of a prediction model in dependence on the historical data.

The prediction 40 of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may also be caused to be performed distributed in one of at least one virtual machine of the host of the plurality of hosts. This may be made for each of the plurality of hosts 23a, 23b, 23c. The prediction 40 of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may further be caused to be performed centrally in the scheduler server 20. This may be made for each of the plurality of hosts 23a, 23b, 23c.

The different steps of causing refer to that the scheduler server 20 may perform an action with own computing resources, or cloud based with other computing resources in the cloud. The scheduler server 20 maybe a dedicated server collecting current and historical resource utilization data for a plurality of hosts, for determining future predicted resource utilization and hence a weight for each of the plurality of hosts. The scheduler server 20 may alternatively utilize one or more server applications in a distributed manner collecting current resource utilization data for a plurality of hosts and collecting future predicted resource utilization predicted locally at each host. Further, the scheduler server 20 may for some hosts act as a dedicated server and for other hosts act in a distributed manner. The predictions may thus be done by doing logical groupings where each predictor will take care of predicting resources of certain parts of the infrastructure or it may also be centralized where a single predictor makes the prediction of the whole infrastructure, or a combination thereof.

A scheduler server 20, according to an embodiment, is presented with reference to Fig. 8, which scheduler server is configured to allocate cloud computing resources 22a, 22b in a cloud computing environment 2i.The scheduler server 20 comprises: a processor 60; and a computer program product 62 storing a computer program 64 with instructions that, when executed by the processor 60, causes the scheduler server 20 to: cause determination 42 of a weight for each of a plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization and predicted resource

utilization; and cause allocation 43 of a cloud computing resource 22a, 22b in dependence on the determined weight. Fig. 8 is a schematic diagram showing some components of the scheduler server 20. The processor 60 maybe provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit etc., capable of executing software instructions of a computer program 64 stored in a memory. The memory can thus be considered to be or form part of the computer program product 62. The processor 60 may be configured to execute methods described herein with reference to Figs. 4A-4C.

The memory may be any combination of read and write memory (RAM) and read only memory (ROM). The memory may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

A second computer program product 63 in the form of a data memory may also be provided, e.g. for reading and/or storing data during execution of software instructions in the processor 60. The data memory 63 can be any combination of read and write memory (RAM) and read only memory (ROM) and may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The data memory 63 may e.g. hold other software instructions 65, to improve functionality for the scheduler server 20.

The scheduler server 20 may further comprise an I/O interface 61 including e.g. a user interface. Other components of the network device are omitted in order not to obscure the concepts presented herein. The scheduler server 20 may comprise further instructions causing the scheduler server to: cause prediction 40 of a resource utilization for each of a plurality of cloud computing recourses; and cause aggregation 41 of the predicted resource utilizations for each of the plurality of hosts 23a, 23b, 23c controlling the plurality of cloud computing resources.

The aggregation may comprise, at each of the plurality of hosts, summing the predicted resource utilization for each cloud computing resource.

The weight for each of the plurality of hosts maybe calculated by adding a predicted resource utilization to a current resource utilization, wherein the predicted resource utilization has a reduced significance compared to the current resource utilization. The weight may also comprise a plurality of predicted resource utilizations for a plurality of time instances, and a prediction further into the future has a reduced significance compared to a prediction nearer into the future. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may be set in dependence on accuracy of the predictions. The significance of each predicted resource utilization of the plurality of predicted resource utilizations may also be dynamically set by machine learning by building a model and extracting each predicted resource utilization of the plurality of predicted resource utilizations through ensemble learning.

The cloud computing resources may comprise one or more of the following types: core of a central processing unit (CPU core), memory, disc usage, incoming network traffic, outgoing network traffic, and stability of a user' s utilization of the cloud computing resources.

The instructions may further cause the scheduler server 20 to: cause prediction 40 of a resource utilization for each of a plurality of a first type of cloud computing recourses 22a, 22b; cause aggregation 41 of the predicted resource utilizations of the first type for each of the plurality of hosts 23a, 23b, 23c controlling the plurality of cloud computing resources 22a, 22b; cause determination 42 of a first weight for each of the plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization of the first type and predicted resource utilization of the first type; cause prediction 40 of a resource utilization for each of a plurality of a second type of cloud computing recourses 22b, 22a; cause aggregation 41 of the predicted resource

utilizations of the second type for each of the plurality of hosts 23a, 23b, 23c controlling the plurality of cloud computing resources 22b, 22a; cause determination 42 of a second weight for each of the plurality of hosts 23a, 23b, 23c, in dependence on current resource utilization of the second type and predicted resource utilization of the second type; and cause

determination 45 of a total weight for each of the plurality of hosts 23a, 23b, 23c, in dependence of the first weight and the second weight.

The instructions may further cause the scheduler server to: cause collection of historical data for a resource utilization for each of a plurality of cloud computing recourses; and cause training of a prediction model in dependence on the historical data.

The prediction 40 of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may be caused to be performed distributed in one of at least one virtual machine of the host of the plurality of hosts. This may be done for each of the plurality of hosts 23a, 23b, 23c. The prediction 40 of a resource utilization, for each of a plurality of cloud computing recourses, for a host of the plurality of hosts may also be caused to be performed centrally in the scheduler server 20. This may be done for each of the plurality of hosts 23a, 23b, 23c. A scheduler system, according to an embodiment, is presented, which scheduler server is configured to allocate cloud computing resources 22a, 22b in a cloud computing environment 21. The scheduler system comprises at least two nodes interacting with each other, and each node comprising a processor and a computer program product. Each node may be implemented by a scheduler server 20 as described above, according to an embodiment, with reference to Fig. 8. The scheduler system stores instructions that, when executed by the processors, causes the scheduler system to: cause

determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization; and cause allocation of a cloud computing resource in dependence on the determined weight. The two nodes may e.g. be a centralized scheduler node and a distributed prediction node.

Fig. 9 is a schematic diagram showing functional blocks of the scheduler server 20. The modules maybe implemented as only software instructions such as a computer program executing in the cache server or only hardware, such as application specific integrated circuits, field programmable gate arrays, discrete logical components, transceivers, etc. or as a combination thereof. In an alternative embodiment, some of the functional blocks may be implemented by software and other by hardware. The modules correspond to the steps in the methods illustrated in Figs. 4A-4C, comprising a prediction manager/ prediction manger unit 100, an aggregation manager/ aggregation manager unit 101, a determination manager/determination manager unit 102, and an allocation manager/allocation manager unit 103. In the embodiments where one or more of the modules are implemented by a computer program, then it shall be understood that these modules do not have to correspond to programming modules, but can be written as instructions according to the programming language in which they would be implemented, since some programming languages do not typically contain programming modules.

The prediction manager 100 is for causing prediction of a resource utilization for each of a plurality of cloud computing recourses. This module

corresponds to the predict resource utilization step 40 of Figs. 4B-4C. This module can e.g. be implemented by the processor 60 of Fig. 8, when running the computer program.

The aggregation manager 101 is for causing aggregation of the predicted resource utilizations for each of the plurality of hosts controlling the plurality of cloud computing resources. This module corresponds to the aggregate predicted resource utilization of Figs. 4B-4C. This module can e.g. be implemented by the processor 60 of Fig. 8, when running the computer program.

The determination manager 102 is for causing determination of a weight for each of a plurality of hosts, in dependence on current resource utilization and predicted resource utilization. This module corresponds to the determination weight step 42, the further types of cloud computing resources decision 44 and the determine total weight step 45 of Figs. 4A-4C. This module can e.g. be implemented by the processor 60 of Fig. 8, when running the computer program.

The allocation manager 103 is for causing allocation of a cloud computing resources in dependence on the determined weight. This module corresponds to the allocate cloud computing resources step 43 of Figs. 4A-4C. This module can e.g. be implemented by the processor 60 of Fig. 8, when running the computer program.

The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.