DESCRIPTION Moving stateful applications

Field of the invention

The present invention relates to moving stateful applications. In particular, the present invention relates to apparatuses, methods and a program for moving stateful applications.

Background of the invention

A major trend in telecommunications, also in mobile networks, especially in the core is to use data center technologies for applications. One motivation of that is to reduce total cost of ownership (TCO) since one platform (data center) can be used for many (most) applications that by today are often running on distinct network nodes. Furthermore, it allows a better scaling and more elasticity since applications can be invoked and terminated flexibly according to networking demands. Fig. 1 shows a typical setup of a data center and its management entities.

The data center itself comprises of hardware including multi core processing units and switching facilities (D-Switch in Fig. 1) to interconnect different processing units on the multiple blades in the multiple racks that make up a data center. The multiple computing parts will be equipped with an operating system (host OS, e.g. Linux) on which one or several virtual machines (VM) can be established. These VMs may be equipped with application software running on top of yet another operating system (OS) (guest OS, e.g. Linux, Windows). The control of the different VMs is done by a piece of HiperVisor (HV) middleware which acts as a mediator between the guest OS/VM and the host operating system hiding the virtual nature of the platform to the guest OS. Virtual machines will be invoked/terminated and equipped with software images by an Infrastructure-as-a-service (IaaS) component, also denoted as Cloud Management System. On demand (of e.g. a cloud orchestration system) a specific software image (which may also include the guest OS) out of a list of software images that is stored in a database will be started on a virtual machine. The selection of the VM is done and controlled by this entity.

The Cloud Orchestration Function, e.g. NSN Cloud Framework CFW, stores templates for specific software that shall be deployed in a network which are stored in yet another database. Those templates comprise e.g. of information about how many applications make up a network function (e.g. three applications together form a voice communication server VCS), which of the images that are stored in the IaaS database do reflect this application(s), the starting order of the different applications, identifiers (IDs) that allow to identify running applications and more. Per screen level command or triggered by an external network control (e.g. via http based interface) new applications can be started / stopped / modified and monitored. The Cloud Orchestration System will communicate with the IaaS or cloud management system, respectively, and directly/indirectly with the application. Yet those interfaces are still subject of standardization, current solutions employ Quantum and Eucalyptus and derivates of those.

Another trend is gaining momentum in communication service provider (CSP) networks, namely software defined networking (SDN), i.e. decoupling of data forwarding and control.

By today, typical nodes in transport networks comprise of specific functionalities. A router, for example, comprises of data switching fabriques which move data packets between the different I/O ports. But it also handles all the complex routing protocols like resource reservation protocol (RSVP), it holds routing tables and more. All the complex functionality and the switching is encapsulated in one box. Another example would be a carrier Ethernet switch, providing data forwarding and control on layer 2. And more and more multilayer switches are used in transport networks providing (generalized) multi protocol label switching ((G-)MPLS) functionality which on top of the before mentioned router or switch functionality provide MPLS / G-MPLS signaling capability. Bottom line, depending for what purpose a transport node is used it is more or less complex providing data forwarding and control function in one monolithic node.

The basic idea of SDN is to decouple control functions from data forwarding functions, in other words, everything that makes a router being a router and everything that makes a switch being a switch is taken out of a node, let's call the remaining part network element (NE) and put the taken out part into a controller. What will be remaining in the NE is pure data forwarding functionality. With this philosophy, routers, switches, MPLS nodes would all have a similar look-alike NE for data forwarding, and a specific control element (which is outside the box) which makes it a router or a switch or whatsoever.

Some principles of SDN are described in Fig. 2.

At the bottom of the picture, an NE providing pure data forwarding functionality is shown. It comprises of the switching hardware (data path) which provides I/O ports, some pieces of software to allow configuration, a flow table which contains port based rules for data forwarding.

In the following there is a description of how to handle a packet depending on e.g. header information. For example, a rule may be that incoming packets on port 0 will be analyzed such that depending what information is in the header, the packet shall be forwarded to port 2 or 3. These rules, which are stored in a flow table, can be passed to the NE from a controller which resides out of the box (denoted as SDN control). For that, a protocol for exchange must be specified and both, the controller and the NE must be able to mutually understand the protocol (SDN client). A most prominent representative for an SDN control protocol is OpenFlow as specified in the Open Network Foundation (ONF). Another known representative is ForCES.

This way and with additional means a whole eco system for sharing transport equipment can be built up. NEs and controller can be cascaded and access can be limited. Introducing FlowVisors will limit access to certain parts of a Flow Table (e.g. ports 0 to 3). Controllers themselves may act as proxies to other controllers. Finally, SDN controllers may provide a northbound interface (I/F) to applications. By this, applications may acquire network resources via this interface in an abstracted way, e.g. "connectivity between topological point A and topological point B with a given bandwidth". SDN controllers may then instruct NEs out of a pool of NEs where there might be several options to solve the request - still hiding the network HW to the application by using this abstract interface.

There exist a variety of program languages, operating systems, platforms and shells on which applications may run. In real time environments many applications are built using C, C++, C# programming language. Here a program source code will be compiled (see Fig. 3) such that text based instructions are transformed into machine code that can be run on specific hardware platforms (such as Intel x86, for example).

At the end of a typical compilation procedure two types of data segments result - one comprising of executable code and another containing data variables that will frequently change during the application runtime. Most typically, after compilation, address relationships for data variables and relative addresses of subroutines are resolved by a re-locator. Since applications will most commonly make use of specific functions provided by the operating system they are running on, typically there are compiler/re-locators tailored for specific operating systems (such as Linux, for example). Most typically, during runtime, executable machine code and application data (variables) are stored in a local RAM (random access memory). Another possibility to create applications is to build them on top of OS independent platforms that hide / mediate OS specific functions (such as Java) via unified application programming interfaces (APIs) so that one and the same application may run on a variety of OS systems, provided the specific platform is implemented (e.g. Java).

Yet another way to create applications is to let a given source code be "interpreted" at runtime - instruction by instruction - thus avoiding the compilation phase (examples: Basic, QBasic, Visual Basic). In this case an interpreter will parse the source code instruction wise and generate the according machine code ad hoc.

Applications may also comprise of a mix of the above mentioned, e.g. user interface (less time critical) parts are implemented in Java while time critical components are implemented using C++ or else.

However, regardless which software creation set up was used, applications will make intensive use of data which is needed to perform the tasks they've been written for. Depending on the kind of data these may change very frequently. Typically, less time critical data like user data records or subscription data and content are stored in databases or on disks. More time critical data like session data are stored in the RAM and runtime-data like process states are stored alongside the executable program parts in the RAM.

Network virtualization, especially use of data center technologies allow for new technical and economical solutions - not only for TCO reduction. With the capability to dynamically setup applications in the cloud, operators can flexibly act on network demands. If further capacity, functionality or performance is required, additional applications can be invoked - and simply be uninstalled when they are no longer needed. Another huge value network virtualization is providing is on disaster / recovery scenarios. If a disaster is to be expected in a certain area (Hurricane, Tsunami) or simply if urgent repair needs to be done at certain spots, it might be necessary to move functionality from one place to another in a very short period of time - and to move it back afterwards. If, for example, a number of voice communication servers (VCS) handling millions of voice calls are operated in data centers at a coast and a Tsunami is expected to happen during the next hours, these servers should be moved to far-off data centers (which are used e.g. for value added services) in order to allow communication even in a disaster case.

Network virtualization would allow to do so - first new servers are started at the far-off data center before the coast side data center will be put out of operation. However, this would mean that all ongoing calls (might be millions and each may be at a different state) would be dropped and lost.

This would be unacceptable by two reasons: first, the quality of service / call completion rate would be unacceptable, and second, all users with dropped calls would try to re-establish the calls simultaneously and create a signaling flood that would impact the whole network and might even bring it down.

More generally, many information technology (IT) and telecommunication applications are stateful and proper operation can only be achieved if user / session / application states are present and accessible. If those applications / those functions shall be moved to other parts of the network, it is an essential requirement that ongoing services shall not be affected.

A most common method of moving applications with stateful services is to wait for de-population before functionality is moved. Most typically, repairs and swaps are done at time periods with low traffic (e.g. during the night). If e.g. an Mobile Switching Center (MSC) -Server shall be moved or swapped, it will be guaranteed (e.g. by load balancers) that newly established voice calls will be forwarded to different servers and ongoing voice calls will be handled as usual. Thus, after some hours, the majority of calls might have finished so the server may be shut down. However, the whole process may take very long and it cannot be guaranteed that all calls are terminated. Thus this is not an appropriate method to meet the requirements described in the foregoing.

Summary of the Invention

It is therefore an object of the present invention to overcome the above mentioned problems and to provide methods, apparatuses and a program for moving stateful applications.

According to an aspect of the present invention there is provided a method comprising :

performing compilation of source code so as to generate data variables, storing the data variables in an external database, and

storing a reference to the data variables stored in the external database in a local memory.

According to further refinements of the present invention as defined under the above aspect, the method further comprises

- receiving an inquiry for a value of a specific data variable stored in the external database, acquiring the value of the specific data variable based on the reference stored in the local memory, and forwarding the requested value of the specific data variable;

- receiving an instruction to update a value of a specific data variable stored in the external database, and instructing the external database to update the value of the specific data variable based on the reference stored in the local memory.

According to another aspect of the present invention there is provided a method for use in a network entity, comprising: evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data, and

if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, forwarding the data packet to a specific output port.

According to further refinements of the present invention as defined under the above aspect

- the specific type of data is data stored in an external database;

- the method further comprises recording, in an information field in the header of the data packet, forwarding information on input and output parameters in the network entity;

- the input and output parameters include information on input and output ports in the network entity;

- the specified output port is predefined;

- the specific output port is defined by an entry in a flowtable linking to the specific output port.

According to another aspect of the present invention there is provided a method for use in a network entity, comprising:

evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data,

if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, evaluating whether the header of the data packet includes a information field including forwarding information, and

routing the data packet based on the forwarding information included in the information field in the header of the data packet.

According to further refinements of the present invention as defined under the above aspect - the specific type of data is data stored in an external database;

- the routing is performed in a reverse direction with respect to a direction indicated by the forwarding information;

According to another aspect of the present invention there is provided a method for use in a network control entity, comprising:

receiving an instruction to move an application running on a first data center to a second data center,

sending an instruction to set up the application to the second data center, retrieving information on references to data variables stored in an external database from a local memory in the first data center,

pushing the retrieved information to the second data center.

According to further refinements of the present invention as defined under the above aspect, the method further comprises invoking the application at the second data center, instructing the second data center to read the data variables, and terminating the application at the first data center.

According to another aspect of the present invention there is provided a method for use in a system, the system comprising a software creation entity, a network entity, and a network control entity, wherein the system is configured to perform in the software creation entity, a method as defined above, in the network entity, a method as defined above, and in the network control entity, a method as defined above.

According to another aspect of the present invention there is provided an apparatus for use in a software creation entity, comprising :

at least one processor,

at least one interface to at least one other network element, and

at least one memory for storing instructions to be executed by the processor, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform :

performing compilation of source code so as to generate data variables, storing the data variables in an external database, and

storing a reference to the data variables stored in the external database in a local memory.

According to further refinements of the present invention as defined under the above aspect the at least one memory and the instructions are further configured to, with the at least one processor, cause the apparatus to perform :

- receiving an inquiry for a value of a specific data variable stored in the external database, acquiring the value of the specific data variable based on the reference stored in the local memory, and forwarding the requested value of the specific data variable;

- receiving an instruction to update a value of a specific data variable stored in the external database, instructing the external database to update the value of the specific data variable based on the reference stored in the local memory.

According to another aspect of the present invention there is provided an apparatus for use in a network entity, comprising :

at least one processor,

at least one interface to at least one other network element, and

at least one memory for storing instructions to be executed by the processor, wherein

the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform :

evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data, and

if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, forwarding the data packet to a specific output port. According to further refinements of the present invention as defined under the above aspect

- the specific type of data is data stored in an external database;

- the at least one memory and the instructions are further configured to, with the at least one processor, cause the apparatus to perform : recording, in an information field in the header of the data packet, forwarding information on input and output parameters in the network entity;

- the input and output parameters include information on input and output ports in the network entity;

- the specified output port is predefined;

- the specific output port is defined by an entry in a flowtable linking to the specific output port.

According to another aspect of the present invention there is provided an apparatus for use in a network entity, comprising :

at least one processor,

at least one interface to at least one other network element, and at least one memory for storing instructions to be executed by the processor, wherein

the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform :

evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data,

if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, evaluating whether the header of the data packet includes a information field including forwarding information, and

routing the data packet based on the forwarding information included in the information field in the header of the data packet. According to further refinements of the present invention as defined under the above aspect

- the specific type of data is data stored in an external database;

- the routing is performed in a reverse direction with respect to a direction indicated by the forwarding information.

According to another aspect of the present invention there is provided an apparatus for use in a network control entity, comprising :

at least one processor,

at least one interface to at least one other network element, and

at least one memory for storing instructions to be executed by the processor, wherein

the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform :

receiving an instruction to move an application running on a first data center to a second data center,

sending an instruction to set up the application to the second data center, retrieving information on references to data variables stored in an external database from a local memory in the first data center, and

pushing the retrieved information to the second data center.

According to further refinements of the present invention as defined under the above aspect, the at least one memory and the instructions are further configured to, with the at least one processor, cause the apparatus to perform : invoking the application at the second data center, instructing the second data center to read the data variables, and terminating the application at the first data center.

According to another aspect of the present invention there is provided an apparatus for use in a software creation entity, comprising :

means for performing compilation of source code so as to generate data variables, means for storing the data variables in an external database, and means for storing a reference to the data variables stored in the external database in a local memory.

According to another aspect of the present invention there is provided an apparatus for use in a network entity, comprising :

means for evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data, and

means for forwarding the data packet to a specific output port, if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data.

According to another aspect of the present invention there is provided an apparatus for use in a network entity, comprising :

means for evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data,

means for evaluating whether the header of the data packet includes a information field including forwarding information, if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, and

means for routing the data packet based on the forwarding information included in the information field in the header of the data packet.

According to another aspect of the present invention there is provided an apparatus for use in a network control entity, comprising :

means for receiving an instruction to move an application running on a first data center to a second data center,

means for sending an instruction to set up the application to the second data center,

means for retrieving information on references to data variables stored in an external database from a local memory in the first data center, and

means for pushing the retrieved information to the second data center. According to another aspect of the present invention there is provided a system, comprising an apparatus for use in a software creation entity as defined above, an apparatus for use in a network entity as defined above, and an apparatus for use in a network control entity as defined above.

According to another aspect of the present invention there is provided a computer program product comprising code means adapted to produce steps of any of the methods as described above when loaded into the memory of a computer.

According to a still further aspect of the invention there is provided a computer program product as defined above, wherein the computer program product comprises a computer-readable medium on which the software code portions are stored.

According to a still further aspect of the invention there is provided a computer program product as defined above, wherein the program is directly loadable into an internal memory of the processing device.

Brief Description of the Drawings

These and other objects, features, details and advantages will become more fully apparent from the following detailed description of aspects/embodiments of the present invention which is to be taken in conjunction with the appended drawings, in which:

Fig. 1 is a diagram showing a setup of a data center; Fig. 2 is a diagram illustrating some principles of SDN; Fig. 3 is a diagram illustrating an example of compilation of an application program source code;

Fig. 4 is a diagram illustrating an example of compilation of an application program source code according to certain embodiments of the present invention;

Fig. 5 is a diagram illustrating an example of interaction according to certain embodiments of the present invention;

Fig. 6 is a diagram illustrating an example of interaction at runtime according to certain embodiments of the present invention;

Fig. 7 is a diagram illustrating an example of a fast forwarding scheme according to certain embodiments of the present invention;

Fig. 8 is a diagram illustrating an example of forwarding of data packets and an example of a header structure in uplink direction according to certain embodiments of the present invention;

Fig. 9 is a diagram illustrating an example of forwarding of data packets and an example of a header structure in downlink direction according to certain embodiments of the present invention;

Fig. 10 is a diagram illustrating an example of a behavior of a packet handler according to certain embodiments of the present invention;

Fig. 11 is a diagram illustrating an example of interaction for enabling full administration of proximity data usage according to certain embodiments of the present invention;

Fig. 12 is a diagram illustrating an example of functionality in a cloud orchestration entity according to certain embodiments of the present invention; Fig. 13 is a diagram illustrating an implementation example according to certain embodiments of the present invention;

Fig. 14 is flowchart illustrating an example of a message flow in the implementation example according to certain embodiments of the present invention;

Fig. 15 is a flowchart illustrating an example of a method according to certain embodiments of the present invention;

Fig. 16 is a flowchart illustrating another example of a method according to certain embodiments of the present invention;

Fig. 17 is a flowchart illustrating another example of a method according to certain embodiments of the present invention;

Fig. 18 is a flowchart illustrating another example of a method according to certain embodiments of the present invention;

Fig. 19 is a diagram illustrating an example of an apparatus according to certain embodiments of the present invention;

Fig. 20 is a diagram illustrating an example of a system according to certain embodiments of the present invention.

Detailed Description

According to certain embodiments of the present invention, there are provided solutions on how to beneficially combine

- application creation processes/tools/environments (compiler, re-locator), - Network Virtualization Technologies, for example, data center technologies,

- Data packet transport techniques, for example, Software-Defined- Networking based transport, and

- Cloud Management Functions, such, that stateful applications can be moved between data centers in a real time manner without service interruption or degradation.

In order to achieve this, according to certain embodiments of the present invention, solutions for different technology domains (data center, transport, software creation) are proposed. These methods can be employed independently or in a combined way.

The relevant technology fields are Network Virtualization, Virtual Machines, Cloud Computing, Software Defined Network (SDN) - SDN transport, and software creation, which have been shortly described above.

According to certain embodiments of the present invention, there are provided solutions in the affected technology domains: software creation, transport and virtualized data centers that allow for moving of stateful applications/services.

It is noted that each solution can be applied independently to provide this functionality, or some or all of these solutions may be combined to gain maximum performance.

The problem can be addressed and solved on the transport level or on the application level or as part of the data center set up or as a combination of those.

One idea according to certain embodiments of the present invention is to store user states, session states and application states outside the environment where the application is running (datacenter-VM, local host) and to guarantee that this data will be accessible from any place of a network to allow an application to be run at different places still serving users/systems without service degradation when being moved.

One part of this idea is that data duplication is avoided during the move by only moving the references to stateful data rather than to move the data itself - this avoids inconsistencies which are a major obstacle in alike implementations today. Currently, there is no network/system available that is able to offer this sort of functionality as is needed to operate a system in a manner as described above.

Fig. 4 shows a typical setup as it is required for creating application software (as already described in connection with Fig. 3) with extensions according to certain embodiments of the present invention that allow to create applications that store and modify their (or a part of their) runtime variables outside their runtime environment. In addition to the setup shown in Fig. 3, the setup of Fig. 4 includes a proximity database in a data cloud.

Fig. 4 shows a Compiler/Re-locator which will - by instruction - perform compilation of source code such that variables are stored in a database (proximity database) and all usage of variables in the application at runtime will be referenced to this data (see also Fig. 6, which will be explained later).

In the simplest case, each instruction in the code that uses/manipulates variables will receive/update according data of the proximity database using a direct connection (e.g. TCP/IP connection).

However, handling such connections directly on application level will be slow and cumbersome. Rather than this it would be more beneficial if underlying platforms or middleware like Operating System or HiperVisor (in case of virtualized environments) will provide according support so that compilers can rely on OS support for proximity data handling through an API.

In this way, the existence of a proximity database would be hidden from the application allowing compiler builders to build more unified compilers matching to specific operating systems. Each time an application will access a variable, this will then be done via an OS API which allows a read/write to the according data record in the proximity database. Furthermore, the OS/middleware/HiperVisor can instruct the application to re-read its variables in case an update is needed.

Fig. 5 illustrates how OS/middleware/HiperVisors will interact beneficially in this context. When an application is invoked, all those variables that are not stored in local RAM but in a proximity database will have to be created and initialized. For this purpose, e.g. to instantiate a variable "i", a <reserve variable> procedure will be called (e.g. as an OS function). In order to identify the application that is related with that call, a process- or application ID can be used. The OS/middleware/HiperVisor will have an inbuilt proximity data scheduler PD-ID scheduler together with a PD-ID list in which all variables of all applications which shall be stored in a proximity database which are running on top of the OS/middleware/HiperVisor are referenced.

The proximity database is a database that will store all variables and states of all applications of all datacenters that are connected to it without having knowledge about individual applications. In order to allow a proper reference, each data record that contains a state or an application runtime variable will be assigned a unique identity which can be requested and which will be assigned by a PD- assignment center.

This function does not have any knowledge about the data record contents, its only knowledge is on which PD-IDs are already in use and which ones are free - it is the master of the pool of PD-IDs. Logically, this function can be placed anywhere in the network, physically it makes sense to place it nearby the proximity database, for example.

When an application requests for creating such an ID, PD-ID scheduler will contact the PD-ID assignment function with a <request PD-ID> message which may also contain the size (in bytes) of the data record to store and in return receive an ID by a <assign PD-ID> message. The OS/middleware/HiperVisor will then store this PD-ID in its PD-ID list linking the process ID and the variable name of the calling application and the newly assigned PD-ID.

Fig. 6 illustrates how applications, OS/middleware/HiperVisor and proximity database interact at runtime. It is assumed the application was built using C++ source code and a variable "i" is used to keep a state of an application feature. Typically, on an event the application checks the state, and in case its value is of "Valuel" it will do some instructions and in case it is of "Value 2" it will do some other instructions. This "switch" construct will in normal operation evaluate the value of variable "i" that is stored in local RAM. In case of proximity data, however, instead of accessing RAM data, the compiler implemented OS/middleware/HiperVisor API calls that

- will get a value of a variable (in case the variable value is needed for processing) via a <get variable> call which will be returned in a <send variable> response

- store a new value of the variable in case the application has changed the value by an <update variable> API call

Furthermore, the OS/middleware/HiperVisor can use a <re-read variable> command to make the application issue a <get variable> request.

Upon receipt of a <get variable> request from the application, referenced by a variable name ("i" in example) and a process ID, the PD-ID scheduler will look up the according PD-ID in its PD-ID list and issue a <request value> to the proximity database referenced by the PD-ID. The proximity database will return the according value in a <report value> message. If the application modifies the value of such a variable or state using the <update variable> call or method, the PD-ID scheduler will send an according <update value> message to the proximity database which will store the new value referenced by PD-ID.

In addition, Fig. 6 shows message flows for requesting, reporting and pushing of IDs from an outside entity.

Thus, as described above, a compiler is built in a way such that its output would be an application that can access frequently changing data (variables) via the references to the proximity database.

However, as an alternative, it is also possible to built the compiler in a way such that its output would be an application that can access frequently changing data (variables) either in the way described above via the references to the proximity database or locally (e.g. in RAM). For example, if for some reason there is no connection to the proximity database (e.g. because simply none is installed) the application may run using "traditional" local storage.

In principle, the method described above is sufficient to store and update proximity data that has only low latency requirements, i.e. typical database lookup cases. However, in real time or close to real time environments, additional delays caused by packet transport may occur.

Thus a very fast transport mechanism needs to be employed for this sort of data that requires very low latency. In best case, transport will only add little more than transmission delay to the overall delay budget. Typically, delay is caused by every hop in a packet transport network, either on layer 2 (Ethernet) or layer 3 (Internet Protocol (IP)) or in both.

Typically, each incoming packet will have to be analyzed according to its source and destination address fields and rules for further forwarding will have to be looked up in tables - which is a time consuming process. Even if new emerging technologies were employed, as e.g. SDN, the forwarding rules are either stored in Flow Tables that need to be explored, or even worse, a controller has to be inquired to help find the right forwarding rule.

Considering the delay, the downlink (proximity database towards application) direction likely performs worse than the uplink direction: since all requests will target the same point in the network (proximity database), SDN controllers could configure the flow tables in the network elements accordingly while as in the backward direction all connections will end up in millions of different endpoints which may make on-demand involvement of an SDN controller necessary, in worst case for each packet.

In order to provide a fast delivery of information in both directions, transport nodes shall be extended with specific functions solving specific problems for either direction.

Fig. 7 describes a fast forwarding scheme according to certain embodiments of the present invention that can be used in SDN deployments in uplink direction. However, this method can also be applied to any packet switched network technology. A Network Control entity, which could e.g. comprise of TNMS (transport network management system) or a OAM (operation and maintenance[system]), will configure all data forwarding network elements (shown at the bottom of Fig. 7) via SDN controller and FlowVisors to apply a very specific behavior in case proximity data is part of the data packets' payload. It is part of this method that specific fields in the packet header shall contain an indication that the payload will comprise of proximity data. In the example shown in Fig. 7, an Ethernet frame is considered and the EtherType field shall contain a specific Proximity Data indication.

For example, there could be a Type "P-Data Access" for uplink direction and "P- Data Report" for downlink direction (represented by HEX values FO and Fl in Fig. 7, see also Figs. 8 and 9). Via SDN controllers, the network elements could be instructed that all data packets with Type "P-Data Access" (uplink data to proximity data) shall be forwarded to a specific port which can even be reserved for P-Data access.

In one implementation one specific port could be reserved solely for proximity data forwarding (e.g. port 32 is reserved for P-Data), in another implementation there can be a fixed entry in the FlowTable that links to the relevant port which can flexibly be assigned and in yet another implementation the FlowTable entry will be assigned at runtime by the SDN controller which was interrogated by the network element upon detection of the first packet with a P-Data header indication.

Furthermore, the header field (or parts of the payload) shall be sub-structured as shown in Fig. 8 where the EtherType field is followed by a HopCount field and several forwarding Info fields (Fwd Info 1 to n in Figs. 8 and 9). Upon detection of the P-Data Access indicator, the forwarding network elements shall increment the Hop Counter field and fill the corresponding Forwarding Info field with information that allows to track the input and output parameters in the according network element for a predicted downlink transmission to come.

For example, if OpenFlow were used as forwarding technique, information on the input port (the port the packet came into the network element) and the output port (the port to which the packet will be forwarded) could be appropriate information. On its way from the source (e.g. application running in a datacenter) to the drain (proximity database) each data forwarding network element will act as a hop. Most typically, even in international connections, less than ten hops should be expected for conveying information from source to drain, for example. In the example shown in Fig. 8, three hops are involved. The source (datacenter) will reset the hop-counter to zero so that the last hop before the drain (proximity database) will increment the hop-counter to 3.

In accordance with the above, the payload of such a data packet will typically contain a request for the value of a variable or state or an update of such.

In the backward direction (see Fig. 9), e.g. when the proximity database will respond to a request and e.g. return the value to a corresponding PD-ID, it will send the packet with the type indicator "P-Data report" and copy the relevant fields of the received packet into the packet header of the packet to send back. All hops in the forwarding row between the Proximity Database and the data center will detect the type info "P-Data Report" in the header field and use the information found in the according forwarding fields to forward the packet from ingress to egress.

For example, if OpenFlow were used as forwarding technology, the information found in this field would be the ports through which the packet was previously sent and this port information could now be applied in the reverse order (input port becomes output port). This way, without further time consuming evaluation, the packet can be directly forwarded and no flow table has to be analyzed and no controller has to be interrogated.

With other words: in the uplink direction, all forwarding network elements add information to the data packet that can be used as routing information in the downlink direction at the appropriate network elements.

In case of SDN, that would mean that each downlink data packet would piggy- pack its own FlowTable.

Of course, this mechanism can also be applied to other forwarding technologies such as Ethernet and it can also be applied for uplink directions (symmetrical). Fig. 10 illustrates how a packet handler, which could be part of a proximity database, will act according to the method described above. Upon reception of the data packet with the "P-Data Access" indicator, the packet handler will look up the value for the requested information with the PD-ID as reference. It will then compile a response data packet with the requested information in the payload field and the indication "P-Data Report" set in the header type field. Furthermore it will copy the Hop Counter- and Forwarding Info-Fields from the received packet to this packet header.

In the foregoing, according to certain embodiments of the present invention, methods have been described

- how software can be built so that application data (variables, states) is stored centrally in the network,

- how operating systems/middleware/HiperVisors will interact, and

- how transport can be tailored to allow a delivery of according information with minimum delay.

Finally, the interaction with data centers will be described - which in the end will allow building a solution to solve the problem described above.

As described above, applications are invoked on virtual machines of a data center by a cloud orchestration tool such as NSN cloud framework CFW. This entity has knowledge which instantiations of which applications are running in what data center (see Fig. 10). This entity is the point of a network where applications are started, stopped and moved. Most typically, it also provides some means to track and monitor some key indicators of an application. However, it has no knowledge on which virtual machine the IaaS entity or cloud management system has invoked the application.

Fig. 11 shows all necessary dialogs of a data center, cloud orchestration, proximity database, proximity data ID assignment center, the application and the operation system(s)/middleware/HiperVisor to allow a full administration of proximity data usage according to certain embodiments of the present invention. Some of the dialogs have already been explained before.

First, the application may acquire (through the OS/middleware/HiperVisor) data storage for variables and state information resulting in assignments of PD-IDs from the PD-ID assignment center upon application start. The values of those can be requested/reported/updated (through the OS/middleware/HiperVisor) with the proximity database. Most beneficially, if an application is stopped, or if an operating system detects that an application has stopped or if a HiperVisor detects that a virtual machine is no longer running, all associated variables and state information shall be released and this shall be reported to the proximity database which in return shall report back the related PD-IDs to the pool of IDs of the PD-ID assignment center.

In addition, the cloud orchestration system shall be able to request all PD-IDs related to a specific application from the OS/middleware/HiperVisor and the according entity shall report back the PD-IDs of the requested application.

Finally, the cloud orchestration system and the OS/middleware/HiperVisor shall provide means that allow the cloud orchestration system to push a set of PD-IDs that are assigned to a specific application to the OS/middleware/HiperVisor and this entity shall, on receipt of such a command, build up a "variable-process ID- PD-ID" context list as described above in connection with Fig. 5.

If an application will then be started, an according request for creation of variables will then result in an already given context, i.e, no new PD-IDs have to be assigned.

Fig. 12 illustrates an example of functionality in a cloud orchestration entity according to certain embodiments of the present invention that will be required to allow this. The cloud orchestration system typically stores application templates alongside with a running application that was previously invoked comprising of information about this application, i.e. which software image was used, an application ID, dependencies with other applications and more. This information field, that is typically stored in a database, shall be amended with additional data records containing PD-IDs and variable-IDs that have been assigned to this application.

The cloud orchestration system shall provide the function to retrieve this information from the OS/middleware/HiperVisor on demand and store this information alongside with the application template. Furthermore, the cloud orchestration and OS/middleware/HiperVisor will provide means that allow pushing such data into this entity. Finally, the cloud orchestration shall have the functionality to either directly (using an appropriate command) or via OS/middleware/HiperVisor instruct the application to re-read or re-fresh the corresponding values. In the latter case, the OS/middleware/HiperVisor will instruct via API the application to re-read those variables as described in connection with Fig. 6.

In an alternative implementation, the Cloud Orchestration will request the values of the variables / state info instead of the PD-IDs and will push those down when demanded. In that case, proximity databases would not have to be involved. However, it is to be noted that in this case, there would be more than one point in the network where this volatile and frequently changing data would be stored and there may be a high risk of data inconsistency.

In the following, there will be described an implementation example according to certain embodiments of the present invention.

The implementation example chosen is a setup that allows solving the problem described above, i.e. moving of an application with all its relevant states. Fig. 13 shows a network where a voice communication system (VCS) is running in a data center located at the east cost of a country. The VCS application had been invoked by the cloud orchestration system, and the network control has pre-configured the SDN based transport network elements to allow access to a proximity database. When the application started, a number of variables and state-related data had been created resulting in PD-ID entries in the proximity database and the OS/middleware/HiperVisor. The system is assumed to be in a normal operation mode, i.e. that there are ongoing calls with each being in a different call state.

Now it is assumed that there is a hurricane warning which forces to move the VCS application to another data center located at the west coast of the country while keeping all ongoing calls.

Fig. 14 shows an example of the system behavior and message flow of this movement according to certain embodiments of the present invention. Out of the network control center there is the decision to move the VCS application. This instruction shall be reported to the cloud orchestration system, e.g. via HTTP interface/command. The cloud orchestration system will then setup a VCS application in the data center west, retrieve all PD-IDs of the VCS running in data center east and push this information to the OS/middleware/HiperVisor of data center west. In parallel, the network control will establish paths from the VCS in data center west to the proximity database by accordingly manipulating the flow tables of affected data forwarding network elements via the SDN controller.

Next, the VCS application in data center west will be invoked while all relevant proximity data is already present at the OS/middleware/HiperVisor since it was pushed down by the cloud orchestration system.

Then the cloud orchestration system will instruct the application (either directly or via OS/middleware/HiperVisor) to re-read the variables and state info. Finally the VCS application in data center east can be terminated (but in this case PD- IDs will not be released).

In another implementation according to certain embodiments of the present invention (not shown in Fig. 14), two or more VCS applications are invoked e.g. in different data centers with one being the active one and the others being in idle mode. PM-IDs are requested from the active application once during its runtime and pushed to the idle ones. Additionally the configuration of the (e.g. SDN based) transport is done in a way that all these applications have access to the proximity database but only the active application will modify/update the proximity data. By changing the activity state (idle/active) the cloud orchestration can now toggle which of the applications shall be used and thus a hot swap in real time would be achieved.

Fig. 15 is a flowchart illustrating an example of a method according to certain embodiments of the present invention.

According to certain embodiments of the present invention, the method may be implemented in a software creation entity or the like, and comprises performing compilation of source code so as to generate data variables in a step S151, storing the data variables in an external database in a step S152, and storing a reference to the data variables stored in the external database in a local memory in a step S153.

According to certain embodiments of the present invention, the method further comprises receiving an inquiry for a value of a specific data variable stored in the external database, acquiring the value of the specific data variable based on the reference stored in the local memory, and forwarding the requested value of the specific data variable.

According to certain embodiments of the present invention, the method further comprises receiving an instruction to update a value of a specific data variable stored in the external database, instructing the external database to update the value of the specific data variable based on the reference stored in the local memory.

Fig. 16 is a flowchart illustrating another example of a method according to certain embodiments of the present invention.

According to certain embodiments of the present invention, the method may be implemented in a network entity, like e.g. a forwarding node in a packet transport network or the like, and comprises evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data in a step S161, and if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, forwarding the data packet to a specific output port in a step S162.

According to certain embodiments of the present invention, the specific type of data is data stored in an external database.

According to certain embodiments of the present invention, the method further comprises recording, in an information field in the header of the data packet, forwarding information on input and output parameters in the network entity.

According to certain embodiments of the present invention, the input and output parameters include information on input and output ports in the network entity.

According to certain embodiments of the present invention, the specified output port is predefined.

According to certain embodiments of the present invention, the specific output port is defined by an entry in a flowtable linking to the specific output port. Fig. 17 is a flowchart illustrating another example of a method according to certain embodiments of the present invention.

According to certain embodiments of the present invention, the method may be implemented in a network entity, like e.g. a forwarding node in a packet transport network or the like, and comprises evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data in a step S171, if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, evaluating whether the header of the data packet includes a information field including forwarding information in a step S172, and routing the data packet based on the forwarding information included in the information field in the header of the data packet in a step S173.

According to certain embodiments of the present invention, the specific type of data is data stored in an external database

According to certain embodiments of the present invention, the routing is performed in a reverse direction with respect to a direction indicated by the forwarding information.

Fig. 18 is a flowchart illustrating another example of a method according to certain embodiments of the present invention.

According to certain embodiments of the present invention, the method may be implemented in a network control entity, like e.g. a cloud orchestration system or the like, and comprises receiving an instruction to move an application running on a first data center to a second data center in a step S181, sending an instruction to set up the application to the second data center in a step S182, retrieving information on references to data variables stored in an external database from a local memory in the first data center in a step S183, and pushing the retrieved information to the second data center in a step S184. According to certain embodiments of the present invention, the method further comprises invoking the application at the second data center, instructing the second data center to read the data variables, and terminating the application at the first data center.

As already mentioned above, the above described methods according to certain embodiments may be arbitrarily combined. Thus, according to certain embodiments of the present invention, there is further provided a method for use in a system, the system comprising a software creation entity, a network entity, and a network control entity, wherein the system is configured to perform any one of the steps described above.

Fig. 19 is a block diagram showing an example of an apparatus according to certain embodiments of the present invention.

In Fig. 19, a block circuit diagram illustrating a configuration of an apparatus 190, for use in a software creation entity, a network entity, and/or a network control entity, is shown, which is configured to implement the above described aspects of the invention. It is to be noted that the apparatus 190 shown in Fig. 19 may comprise several further elements or functions besides those described herein below, which are omitted herein for the sake of simplicity as they are not essential for understanding the invention. Furthermore, even though reference is made to an apparatus for use in a software creation entity, a network entity, and/or a network control entity, the apparatus may be also another device having a similar function, such as a chipset, a chip, a module etc., which can also be part of or attached as a separate element to the software creation entity, a network entity, and/or a network control entity or the like.

The apparatus 190 may comprise a processing function or processor 191, such as a CPU or the like, which executes instructions given by programs or the like related to the flow control mechanism. The processor 191 may comprise one or more processing portions dedicated to specific processing as described below, or the processing may be run in a single processor. Portions for executing such specific processing may be also provided as discrete elements or within one or more further processors or processing portions, such as in one physical processor like a CPU or in several physical entities, for example. Reference sign 192 denotes transceiver or input/output (I/O) units (interfaces) connected to the processor 191. The I/O units 192 may be used for communicating with one or more other network elements, or the like. The I/O units 192 may be a combined unit comprising communication equipment towards several network elements, or may comprise a distributed structure with a plurality of different interfaces for different network elements. Reference sign 193 denotes a memory usable, for example, for storing data and programs to be executed by the processor 191 and/or as a working storage of the processor 191.

The processor 191 is configured to execute processing related to the above described aspects.

According to certain embodiments of the present invention, the processor 191 is configured to perform compilation of source code so as to generate data variables, storing the data variables in an external database, and storing a reference to the data variables stored in the external database in a local memory.

According to certain embodiments of the present invention, the processor 191 is configured to perform evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data, and if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, forwarding the data packet to a specific output port.

According to certain embodiments of the present invention, the processor 191 is configured to perform evaluating whether a header of a data packet includes an indication that the data packet contains a specific type of data, if it is evaluated that the header of the data packet includes the indication that the data packet contains a specific type of data, evaluating whether the header of the data packet includes a information field including forwarding information, and routing the data packet based on the forwarding information included in the information field in the header of the data packet.

According to certain embodiments of the present invention, the processor 191 is configured to perform receiving an instruction to move an application running on a first data center to a second data center, sending an instruction to set up the application to the second data center, retrieving information on references to data variables stored in an external database from a local memory in the first data center, pushing the retrieved information to the second data center.

For further functions of the apparatus according to further example embodiments of the present invention, reference is made to the above description of methods according to certain embodiments of the present invention, as described in connection with Figs. 15 to 18.

Fig. 20 is a block diagram showing an example of system according to certain embodiments of the present invention.

A system 200 according to certain embodiments of the present invention, comprises at least one of an apparatus for use in a software creation entity 201, an apparatus for use in a network entity 202, and an apparatus for use in a network control entity 203 according to any one of the above described aspects.

The methods and apparatuses described above will allow a fast swap or move of functionality between data centers without service degradation. Without this functionality, only part of the economical potential of network virtualization would be harvested. The presented solutions according to certain embodiments of the present invention allow to

- act very fast on changing networking demands,

- build in per-se resilience, and

- handle networking in a disaster case.

One crucial or pain point in today's "hot-swap" technologies is that it must be avoided to have frequently changing data being stored/duplicated at different places since that will always lead to data inconsistencies.

Thus, it is an advantage of the proposed solutions according to certain embodiments of the present invention is that only links (pointer, IDs) that reference this data are swapped rather than data itself is copied or duplicated.

Another advantage is that the proposed methods according to certain embodiments of the present invention can be employed separately and each method will add its own benefits, while a combination would open the full potential. For example, compilers allow outsourcing critical data - but do not necessarily have to demand for a specific transport being in place. As another example, employing the piggy-pack routing information for data packet transmission, especially in the downlink, will also beneficial for other use cases.

In the foregoing exemplary description of the apparatus, only the units/means that are relevant for understanding the principles of the invention have been described using functional blocks. The apparatus may comprise further units/means that are necessary for its respective operation as base station or management entity, respectively. However, a description of these units/means is omitted in this specification. The arrangement of the functional blocks of the apparatus is not construed to limit the invention, and the functions may be performed by one block or further split into sub-blocks. When in the foregoing description it is stated that the apparatus (or some other means) is configured to perform some function, this is to be construed to be equivalent to a description stating that a (i.e. at least one) processor or corresponding circuitry, potentially in cooperation with computer program code stored in the memory of the respective apparatus, is configured to cause the apparatus to perform at least the thus mentioned function. Also, such function is to be construed to be equivalently implementable by specifically configured circuitry or means for performing the respective function (i.e. the expression "unit configured to" is construed to be equivalent to an expression such as "means for").

For the purpose of the present invention as described herein above, it should be noted that

- method steps likely to be implemented as software code portions and being run using a processor at an apparatus (as examples of devices, apparatuses and/or modules thereof, or as examples of entities including apparatuses and/or modules therefore), are software code independent and can be specified using any known or future developed programming language as long as the functionality defined by the method steps is preserved;

- generally, any method step is suitable to be implemented as software or by hardware without changing the idea of the aspects/embodiments and its modification in terms of the functionality implemented;

- method steps and/or devices, units or means likely to be implemented as hardware components at the above-defined apparatuses, or any module(s) thereof, (e.g., devices carrying out the functions of the apparatuses according to the aspects/embodiments as described above) are hardware independent and can be implemented using any known or future developed hardware technology or any hybrids of these, such as MOS (Metal Oxide Semiconductor), CMOS (Complementary MOS), BiMOS (Bipolar MOS), BiCMOS (Bipolar CMOS), ECL (Emitter Coupled Logic), TTL (Transistor-Transistor Logic), etc., using for example ASIC (Application Specific IC (Integrated Circuit)) components, FPGA (Field-programmable Gate Arrays) components, CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components;

- devices, units or means (e.g. the above-defined apparatuses, or any one of their respective units/means) can be implemented as individual devices, units or means, but this does not exclude that they are implemented in a distributed fashion throughout the system, as long as the functionality of the device, unit or means is preserved;

- an apparatus may be represented by a semiconductor chip, a chipset, or a (hardware) module comprising such chip or chipset; this, however, does not exclude the possibility that a functionality of an apparatus or module, instead of being hardware implemented, be implemented as software in a (software) module such as a computer program or a computer program product comprising executable software code portions for execution/being run on a processor;

- a device may be regarded as an apparatus or as an assembly of more than one apparatus, whether functionally in cooperation with each other or functionally independently of each other but in a same device housing, for example.

In general, it is to be noted that respective functional blocks or elements according to above-described aspects can be implemented by any known means, either in hardware and/or software, respectively, if it is only adapted to perform the described functions of the respective parts. The mentioned method steps can be realized in individual functional blocks or by individual devices, or one or more of the method steps can be realized in a single functional block or by a single device.

Generally, any method step is suitable to be implemented as software or by hardware without changing the idea of the present invention. Devices and means can be implemented as individual devices, but this does not exclude that they are implemented in a distributed fashion throughout the system, as long as the functionality of the device is preserved. Such and similar principles are to be considered as known to a skilled person. Software in the sense of the present description comprises software code as such comprising code means or portions or a computer program or a computer program product for performing the respective functions, as well as software (or a computer program or a computer program product) embodied on a tangible medium such as a computer-readable (storage) medium having stored thereon a respective data structure or code means/portions or embodied in a signal or in a chip, potentially during processing thereof.

It is noted that the aspects/embodiments and general and specific examples described above are provided for illustrative purposes only and are in no way intended that the present invention is restricted thereto. Rather, it is the intention that all variations and modifications which fall within the scope of the appended claims are covered.