Field

The present specification relates to a generating or obtaining an updated neural network from a base neural network.

Background

Neural networks are being utilized in a wide range of applications, for many different devices, such as mobile phones. Example applications of neural networks include image and video analysis and processing, social media data analysis, device usage data analysis etc. Although sending data to a centralised computation server for processing is appealing in terms of end- user computational complexity and battery power saving, other considerations, such as data privacy and weaknesses of centralised computation argue in favour of a more distributed computation scenario. To this end, improvements in the communication and updating of neural networks would be advantageous.

Summary

In a first aspect, this specification describes an apparatus comprising: means for obtaining or generating an updated multi-unit neural network from a base neural network, wherein the updated neural network is generated by updating at least one parameter within a plurality of units of the base neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; means for obtaining an order of unit updates for converting the base neural network into the updated neural network unit- by-unit (for example using a brute force methodology), wherein the order of layer updates is based on a metric (which metric may be such that each update improves the performance of the neural network); and means for providing neural network updating information, wherein the neural network updating information includes data relating to said order. The base neural network and the updated neural network may be feedforward neural networks. In some embodiments, providing neural network updating information includes sending said information to a remote device.

The neural network updating information may include the updated at least one parameter of the updated neural network.

The neural network updating information may comprise an order of updates preceded or followed by updated weights of the neural network and/or updated weights of the neural network, each preceded or followed by an indication of the unit of the neural network to which the updated at least one parameter relates.

The updated multi-unit neural network may be configured to receive data having a different distribution than the data that the base neural network is configured to receive.

The neural network updating information may include the updated at least one parameter.

The at least one parameter may comprise weights within the plurality of units of the base neural network.

In a second aspect, this specification describes an apparatus comprising: means for receiving neural network updating information, the neural network updating information including data relating to an order in which a plurality of units of a multi-unit base neural network are to be updated, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and means for updating the base neural network to generate an updated neural network by updating at least one parameter within the plurality of units of the base neural network in accordance with said neural network updating information. The base neural network and the updated neural network may be feedforward neural networks.

The apparatus may further comprising means for obtaining at least one parameter of a next unit of the neural network to be updated from said neural network updating information, wherein said means for updating the base neural network updates said neural network accordingly.

The neural network updating information may include the updated at least one parameter of the updated neural network. The neural network updating information may comprise an order of updates preceded or followed by updated weights of the neural network and/or updated weights of the neural network, each preceded or followed by an indication of the unit of the neural network to which the updated at least one parameter relates. The updated multi-unit neural network may be configured to receive data having a different distribution than the data that the base neural network is configured to receive.

The neural network updating information may include the updated at least one parameter. The at least one parameter may comprise weights within the plurality of units of the base neural network. In the first and second aspects, the means may comprise: at least one processor; and at least one memory including computing program code, the at least one memory and computer program code configured to, with the at least one processor, cause the performance of the apparatus. In a third aspect, this specification describes a method comprising: obtaining or generating an updated multi-unit neural network from a base neural network, wherein the updated neural network is generated by updating at least one parameter within a plurality of units of the base neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; obtaining an order of unit updates for converting the base neural network into the updated neural network unit-by-unit, wherein the order of layer updates is based on a metric; and providing neural network updating information, wherein the neural network updating information includes data relating to said order. The at least one parameter may comprise weights within the plurality of units of the base neural network.

In a fourth aspect, this specification describes a method comprising: receiving neural network updating information, the neural network updating information including data relating to an order in which a plurality of units of a multi-unit base neural network are to be updated, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and updating the base neural network to generate an updated neural network by updating at least one parameter within the plurality of units of the base neural network in accordance with said neural network updating information. The at least one parameter may comprise weights within the plurality of units of the base neural network.

The method may further comprise obtaining at least one parameter of a next unit of the neural network to be updated from said neural network updating information, wherein said means for updating the base neural network updates said neural network according. The method may further comprise repeating said obtaining and updating for a further next unit of the multi-unit network to be updated.

In a fifth aspect, this specification describes an apparatus configured to perform any method as described with reference to the third or fourth aspect. In a sixth aspect, this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform any method as described with reference to the third or fourth aspect.

In a seventh aspect, this specification describes a computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by the at least one processor, causes performance of: obtaining or generating an updated multi-unit neural network from a base neural network, wherein the updated neural network is generated by updating at least one parameter within a plurality of units of the base neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; obtaining an order of unit updates for converting the base neural network into the updated neural network unit-by-unit, wherein the order of layer updates is based on a metric; and providing neural network updating information, wherein the neural network updating information includes data relating to said order.

In an eighth aspect, this specification describes a computer-readable medium having computer-readable code stored thereon, the computer readable code, when executed by the at least one processor, causes performance of: receiving neural network updating

information, the neural network updating information including data relating to an order in which a plurality of units of a multi-unit base neural network are to be updated, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and updating the base neural network to generate an updated neural network by updating at least one parameter within the plurality of units of the base neural network in accordance with said neural network updating information.

In a ninth aspect, this specification describes a non-transitoiy computer readable medium comprising program instructions stored thereon for performing at least the following:

obtaining or generating an updated multi-unit neural network from a base neural network, wherein the updated neural network is generated by updating at least one parameter within a plurality of units of the base neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; obtaining an order of unit updates for converting the base neural network into the updated neural network unit-by- unit, wherein the order of layer updates is based on a metric; and providing neural network updating information, wherein the neural network updating information includes data relating to said order. In a tenth aspect, this specification describes a non-transitoiy computer readable medium comprising program instructions stored thereon for performing at least the following:

receiving neural network updating information, the neural network updating information including data relating to an order in which a plurality of units of a multi-unit base neural network are to be updated, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and updating the base neural network to generate an updated neural network by updating at least one parameter within the plurality of units of the base neural network in accordance with said neural network updating information.

In an eleventh aspect, this specification describes an apparatus comprising: means for obtaining or generating updating information for updating an n-unit neural network from a first n-unit base neural network by updating at least one parameter of each a plurality, i, of the n-units of the neural network in turn, wherein, for each i, an ith neural network is generated from an i-ith neural network, wherein the base neural network is the 0 ^th neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and means for providing neural network updating information, wherein the neural network updating information includes data relating to said updated at least one parameter.

In an twelfth aspect, this specification describes an apparatus comprising: means for receiving updating information for updating an n-unit neural network, the neural network updating information including data relating to updated at least one parameter for each of a plurality of units of an n-unit neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and means for updating at least one parameter of each of a plurality, i, of the n-units of the neural network in turn, in accordance with said updating information, wherein, for each i, an ith neural network is generated from an i-ith neural network, wherein the base neural network is the 0 ^th neural network.

In an thirteenth aspect, this specification describes a method comprising: obtaining or generating updating information for updating an n-unit neural network from a first n-unit base neural network by updating at least one parameter of each a plurality, i, of the n-units of the neural network in turn, wherein, for each i, an ith neural network is generated from an i- ith neural network, wherein the base neural network is the 0 ^th neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and providing neural network updating information, wherein the neural network updating information includes data relating to said updated at least one parameter. In a fourteenth aspect, this specification describes an apparatus comprising: receiving updating information for updating an n-unit neural network, the neural network updating information including data relating to updated at least one parameter for each of a plurality of units of an n-unit neural network, wherein each of the plurality of units is any of a neural network layer, a neural network filter or a neural network node; and updating at least one parameter of each of a plurality, i, of the n-units of the neural network in turn, in accordance with said updating information, wherein, for each i, an ith neural network is generated from an i-ith neural network, wherein the base neural network is the 0 ^th neural network.

In a fifteenth aspect, this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform any method as described with reference to the thirteenth aspect. In a sixteenth aspect, this specification describes a non-transitoiy computer readable medium comprising program instructions stored thereon for performing any method as described with reference to the thirteenth aspect.

Brief description of the drawings

Example embodiments will now be described, by way of example only, with reference to the following schematic drawings, in which:

FIG. i is a block diagram of a neural network system in accordance with an example embodiment;

FIG. 2 is a block diagram of a system in accordance with an example embodiment;

FIG. 3 is a flow chart showing an algorithm in accordance with an example embodiment;

FIG. 4 is a flow chart showing an algorithm in accordance with an example embodiment;

FIG. 5 is a flow chart showing an algorithm in accordance with an example embodiment;

FIG. 6 is a block diagram of a system in accordance with an example embodiment; and FIGS. 7a and 7b show tangible media, respectively a removable memory unit and a compact disc (CD) storing computer-readable code which when run by a computer perform operations according to embodiments.

Detailed description

A neural network is a computational graph including one or more layers of computation. One or more layers represent the input layers, and one or more layers represent the output layers. During the forward pass, the typical execution order of layers is from the input layers to the output layers. Feedforward neural networks are such that there is no feedback loop: each layer takes one or more inputs from one or more of the layers before and provides one or more outputs to one or more of the subsequent layers. Initial layers of a neural network (those close to the input data) may extract semantically low-level features, such as edges and textures in images. Intermediate and final layers may extract more semantically high-level features.

FIG. l is a block diagram of a neural network system, indicated generally by the reference numeral l, in accordance with an example embodiment. The system l includes a first neural network layer 2, a second neural network layer 4 and a third neural network layer 6. The system includes inputs that are provided to the first neural network layer 2 and outputs provided by the third neural network layer 6. As shown in Figure 1, each layer takes one or more input from the layer before and provides one or more outputs to the subsequent layer, such that the system 1 is a feedforward neural network. Of course, although the neural network 1 includes three neural network layers, the principles described herein are applicable to neural networks having any number of layers and taking inputs from one or more previous layers and providing outputs to one or more subsequent layers.

FIG. 2 is a block diagram of a system, indicated generally by the reference numeral 10, in accordance with an example embodiment. The system 10 includes a first entity 12 (such as a server) and a second entity 14 (such as a remote device). The second entity 14 may, for example be a user device, a mobile phone or some other device, such as an Internet of Things enabled device.

The first entity 12 has computational, memory and power capabilities for performing training of a neural network (such as the neural network 1 described above). The first entity may train the neural network itself, or may otherwise obtain an updated neural network (e.g. from some other device). The first entity may have one or more base neural networks, i.e., neural networks which may be updated. The second entity 14 has computational, memory and power capabilities for using a neural network, or is otherwise interested in receiving a neural network trained by the first entity 12. The second entity may have one or more base neural networks, i.e., neural networks which may be updated. One or more of these base neural networks may be the same as some of those present in the first entity. In particular, on the two devices the common base neural networks may have the same topology and/ or architecture (e.g., number and type of layers, number of units per layer, etc.) and in the values of each leamable and non-learnable parameters (e.g., the weights of the connections). FIG. 3 is a flow chart showing an algorithm, indicated generally by the reference numeral 20, in accordance with an example embodiment. The algorithm 20 starts at operation 22 where a base neural network is updated. This may be one of the base neural networks which are the same on the first entity and on the second entity. The algorithm 20 may, for example, be carried out at the first entity 12 of the system 10 (such as a server), with update information being provided to the second entity 14 of the system 10 (such as a remote device).

The updated neural network obtained or generated in operation 22 may be a fine-tuned version of a base neural network and may be generated for many different reasons such as: · To change the accuracy of the base neural network (usually to improve the accuracy)

• To perform a domain adaptation (such as from a certain data distribution to a different data distribution)

• To perform a transfer learning to a different but related task (e.g. from a classification task to a segmentation task).

In order to achieve the best results from fine-tuning, the first entity 12 may perform fine- tuning on all of the layers of the base neural network. As a result, the weights of all layers of the base neural network (such as the neural network layers 2, 4 and 6 described above) may change without imposing any restriction on the order of which weights should be changed. Such order may be decided implicitly by the training algorithm (e.g. stochastic gradient descent).

The first entity may use an accuracy or performance metric which fits the considered task.

For example, if the task is image classification, where the image is classified into one out of a predefined set of classes, the classification accuracy may be the ratio of the number of correctly predicted classes for all images in a held-out dataset (i.e., a dataset which was not used for training the neural network) over the number of all images in that dataset. Another metric may combine an accuracy metric (such as the classification accuracy) with a computational complexity metric, or other metric, where each metric is weighted separately and potentially differently.

With the neural network updated, the algorithm 20 moves to operation 24 where the updates are ordered according to a metric. The operation 24 defines an order in which units (e.g. layers) of the base neural network should be updated so that the base neural network can be updated unit-by-unit (as described further below) .

The order in which the units are updated may be such that each update improves the performance of the latest version of the neural network, i.e., either of the base neural network or of the previously updated neural network. For example, in an example updating of the example neural network system l, a first non-optimal ordering of updates to the neural network layers may be to update the first layer 2, the second layer 4 and the third layer 6 in turn. This may be non-optimal as the accuracy of the latest updated neural network may not increase at each update. In fact, the accuracy of such updates may be as follows:

• Accuracy after updating the first layer: 0.8.

• Accuracy after updating the first and second layers: 0.78.

• Accuracy after updating the first, second and third layers: 0.83.

Thus, this order is non-optimal, as the accuracy drops when updating the second layer after updating the first layer, with respect to the case of updating only the first layer.

Alternatively, updating the neural network system 1 in a different order (the second layer 4, then the first layer 2 and finally the third layer 6) may give the following accuracy:

• Accuracy after updating the second layer: 0.8.

• Accuracy after updating the second and first layers: 0.81.

• Accuracy after updating the second, first and third layers: 0.83. If the operation 24 uses a metric that requires the order to be selected such that each update improves the performance of the neural network, then the order second, first then third layers meets this requirement, but the order first, second then third layers does not. Thus, the neural network update order second, first, third may be selected in the operation 24. The algorithm 20 moves to operation 26 where neural network updates are provided (for example by the first entity 12 to the second entity 14).

FIG. 4 is a flow chart showing an algorithm, indicated generally by the reference numeral 30, in accordance with an example embodiment.

The algorithm 30 starts at operation 32 where neural network updates are received. The neural network updates received in operation 32 may be the updates provided in the operation 26 described above. The updates may, for example, be received at the second entity 14 of the system 10.

The second entity 14 may already have the base neural network for updating. The said base neural network may have been trained from uninitialized (e.g. random) weights or may have been previously fine-tuned. In any event, the base neural network is the starting point for the updating described herein, and it may be the same base neural network which is present in the first entity.

On receipt of the neural network updates, the algorithm 30 moves to operation 34, where the base neural network is updated in the order defined in operation 24 (e.g. the second, first and third neural network layers respectively, in the example described above).

In this way, should the process by which the neural network system 1 is being updated be interrupted, it can be ensured that the neural network can be improved, even if only part of a message output in the operation 26 is received in the operation 32. In one example, the process that is interrupted may refer to communication of updates from the first entity to the second entity. In another example, the process may refer to application of the updated weights on the latest neural network by the second entity. In both examples, if one layer is not received or applied to the neural network, the previous update(s) are guaranteed to have already improved the neural network.

The examples described above are applicable to layer-by-layer updating of neural networks, but this is not essential. For example, the invention is applicable to unit-by-unit updates of neural networks, where the units are any of a neural network layer, a neural network filter or a neural network node. Also, one unit may be a sub-set of layers. According to an

embodiment at least one parameter within a plurality of units of the base neural network is updated. The at least one parameter may for example comprise weight or weights associated with neural network nodes or connections. The at least one parameter may also indicate updating a topology or architecture of at least one layer of the neural network. For example, the at least one parameter may indicate an updated number of nodes or units of a neural network layer, for example a fully-connected layer, a number of convolution filters for one or more convolutional layers of the neural network, and/or one or more types of activation functions associated with the neural network. In an implementation of the system 10 described above, signalling may be required between the first entity 12 and the second entity 14 such that the second entity 14 is informed of the relevant order for updating the base neural network (for example, during the providing updates operation 26 and the receiving updates operation 32 described above). Such signalling can take many different forms, as described further below. Also, the first entity 12 may send information related to identifying the base neural network on which updates are intended to be applied, as the second entity 14 may have multiple base neural networks. This may comprise signalling a base neural network identifier either before sending the updates or together with at least one of the updates. The first entity 12 may include an order sequence in a protocol packet in the form of a signalling table that links each received layer to an actual layer identifier number. This table may be signalled either together with the first layer data or as a separate stream.

Alternatively, the first entity 12 may send a layer, but include one data element with the layer of the data (e.g. preceding the layer) which indicates the layer that is being sent. More in detail, each transmitted layer update may be associated with an update order number that indicates which layer (or other unit) of the neural network at the second entity 14 the data relates to. On receiving neural network layer update data, the second entity 14 may check the order number (e.g. layer identifier) for the layer data and apply the updated data to the corresponding layer of the neural network stored at the second entity.

In both examples described above (receiving the entire order list in advance or receiving layer identifiers for each received layer), the term“applying” may mean that the received layer weights will be used in place of the layer weights of the current neural network (e.g. base neural network). Alternatively, if the received layer contains only the update (difference between the new layer weights and the previous layer weights), then the second entity 14 may modify the weights of the corresponding layer accordingly (e.g. using a summation operation).

In a further variant, the first entity 12 may send both a table with the entire sequence before sending the update layer data and also send identifiers with each sent layer. This

embodiment is arguably more robust against temporaiy interruptions of the communication channel between the first and second entities, so that the second entity 14 can verify that the received layer is the correct one in the sequence order contained in the table. If a

communication channel interruption occurs and a layer gets lost, the user device will receive a layer which is not the expected layer according to the sequence order provided in the table and may, for example, request resending of the missing data.

In the examples described above, the relevant metric for ordering unit updates is such that each update improves the performance of the neural network. This is not essential to all embodiments. For example, in the case of limited bandwidth, the ordering process may take into account the size of the layers as an alternative, or additional, ordering criterion. For example, if a first fine-tuned layer provides a slightly inferior performance improvement with respect to a second fine-tuned layer, but at a much lower bandwidth cost (e.g. small layer or more sparsely updated weight matrix, thus more compressible), then the first fine-tuned layer may be sent before the second fine-tuned layer, despite having an inferior performance. FIG. 5 is a flow chart showing an algorithm, indicated generally by the reference numeral 40, in accordance with an example embodiment. The algorithm 40 relates to updating a n-unit base neural network (such as an n-layer neural network) from a first n-unit base neural network by updating weights of each of a plurality, i, of the n-units of the neural network in turn (e.g. a subset or all of the units of the neural network may be updated). The algorithm 40 starts at operation 42 where the value i is initialised. The value i may be equal to n (such that all units of the neural network are updated), but this is not essential.

Next, at operation 44, the layers of the neural network are frozen and, at operation 46, only the ith unit is fine-tuned. This ensures that when a unit is fine-tuned, the other units in the neural network do not change.

With the ith unit fine-tuned, update information is sent in operation 48 (for example from the first entity 12 to the second entity 14). Thus, in accordance with the algorithm 30 described above, the second entity can receive the update information and update the neural network accordingly.

At operation 50, it is determined whether the algorithm 40 is complete. The algorithm may be complete, for example, if all units (e.g. layers) within the neural network have been updated (or if all units that are to be updated have been updated). If the algorithm 40 is not complete, the value i is incremented (operation 52) and the algorithm returns to operation 44 described above and the operations 44 to 50 are repeated with the new value i. If the algorithm 40 is complete, the algorithm terminates at operation 54.

The operation 46 may involve sending the fine-tuned units in the original order of updating (e.g. the first neural network layer, closest to the input, then the second layer and so on until the last layer, closest to the output). In this way, should the process by which the neural network system is being updated be interrupted, it can be ensured that the neural network can be improved, even if only some instances of the operation 46 are received in the operation 32 described above.

Updating each of the neural network units in turn may be advantageous in some

embodiments, for example in order to reduce overheads in determining (and transmitting) an order to update the neural network units. In the algorithm 50, the value i is incremented. In a different embodiment, the values of i may be preset, but not necessarily incremented. Thus, the order in which the neural network units are to be updated (with the other modules frozen) may be preset and may be known to the receiver. In this way, the receiver may not need to be instructed of the order in which the neural network units are to be updated.

Some of the examples described above relate to layer-by-layer updating of neural networks. This is not essential. For example, the invention is applicable to unit-by-unit updates of neural networks, where the units are any of a neural network layer, a neural network filter or a neural network node.

For completeness, FIG. 6 is a schematic diagram of components of one or more of the modules described previously (e.g. the first entity 12 and/or the second entity 14), which hereafter are referred to generically as processing systems 300. A processing system 300 may have a processor 302, a memory 304 coupled to the processor and comprised of a RAM 314 and ROM 312, and, optionally, user inputs 310 and a display 318. The processing system 300 may comprise one or more network interfaces 308 for connection to a network, e.g. a modem which may be wired or wireless.

The processor 302 is connected to each of the other components in order to control operation thereof.

The memory 304 may comprise a non-volatile memoiy, a hard disk drive (HDD) or a solid state drive (SSD). The ROM 312 of the memory 304 stores, amongst other things, an operating system 315 and may store software applications 316. The RAM 314 of the memoiy 304 is used by the processor 302 for the temporary storage of data. The operating system 315 may contain code which, when executed by the processor, implements aspects of the algorithms 20, 30 or 40.

The processor 302 may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors. A processor may comprise processor circuitry. The processing system 300 may be a standalone computer, a server, a console, or a network thereof. In some embodiments, the processing system 300 may also be associated with external software applications. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications may be termed cloud-hosted applications. The processing system 300 may be in communication with the remote server device in order to utilize the software application stored there.

FIG. 7a and FIG. 7b show tangible media, respectively a removable memoiy unit 365 and a compact disc (CD) 368, storing computer-readable code which when run by a computer may perform methods according to embodiments described above. The removable memoiy unit 365 may be a memory stick, e.g. a USB memory stick, having internal memory 366 storing the computer-readable code. The memory 366 may be accessed by a computer system via a connector 367. The CD 368 may be a CD-ROM or a DVD or similar. Other forms of tangible storage media may be used. Embodiments of the present invention may be implemented in software, hardware, application logic or a combination of software, hardware and application logic. The software, application logic and/or hardware may reside on memory, or any computer media. In an example embodiment, the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media. In the context of this document, a “memoiy” or“computer-readable medium” may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.

Reference to, where relevant,“computer-readable storage medium”,“computer program product”,“tangibly embodied computer program” etc., or a“processor” or“processing circuitry” etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialised circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices and other devices. References to computer program, instructions, code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device as instructions for a processor or configured or configuration settings for a fixed function device, gate array, programmable logic device, etc. As used in this application, the term“circuitry” refers to all of the following: (a) hardware- only circuit implementations (such as implementations in only analogue and/or digital circuitry) and (b) to combinations of circuits and software (and/or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a server, to perform various functions) and (c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the above- described functions may be optional or may be combined. Similarly, it will also be

appreciated that the flow diagrams of FIGS. 3, 4 and 5 are examples only and that various operations depicted therein may be omitted, reordered and/ or combined.

It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.

Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.

Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

It is also noted herein that while the above describes various examples, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and

modifications which may be made without departing from the scope of the present invention as defined in the appended claims.