RELATED APPLICATION/S

This application claims the benefit of priority of U.S. Provisional Patent Application No. 63/405,447 filed on 11 September 2022, the contents of which are incorporated herein by reference in their entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention, in some embodiments thereof, relates to communication balancing over non- symmetric radio links.

In electronic communications, link aggregation is the combining or aggregating of multiple network connections in parallel by any of several methods. Such aggregation is generally carried out for one or both of two reasons, the first being in order to increase throughput beyond that which a single connection is able to sustain. A second reason is to provide redundancy in case one of the links should fail, or both. Link aggregation may be used with Ethernet, multiple modem-based links, DSL lines, wireless broadband, Wi-Fi and cellular networks and may be used to provide backbone links for networks, say linking together base stations.

A link aggregation group (LAG) is the combined collection of physical ports for such a multiple link communication, and generally the data to be transmitted is split between the available channels at the packet level. That is to say, individual packets are assigned to one or other of the channels for transmission. The arriving packets can be of different sizes, depending on the origin and purpose of the data, and different packets may have different priority levels. Furthermore, the various network connections, the transmission channels, may not be symmetric but may have different channel characteristics, such as different bandwidths and different latencies. Furthermore, the channel characteristics may change dynamically. Thus, one channel at any given time may be a high bandwidth faster channel, that is having a lower latency, and the other may be a low bandwidth slower channel, that is having a higher latency, or the channel bandwidths and latencies may change dynamically due to atmospherics or for other reasons.

Thus, packets sent via one channel may arrive at the destination much faster than those sent via another channel. Nevertheless, the receiving application needs to assemble the packets in order, and thus considerable buffering may be needed to retain earlier arriving packets that belong after later arriving packets, also affecting the throughput. Thus, imbalance between the connections may lead to increased memory size requirements and may reduce the throughput. Furthermore, some flow types use very large, so-called jumbo, packets. Smaller packets have to wait until the jumbo packets have passed in order to get their turn, leading to particular latency for flow types that use smaller packets.

It is possible to balance the network load across all links. Load balancing may involve dividing traffic on a per network socket basis. Another solution for balancing network load is channel bonding, which involves a division of traffic between physical interfaces at a lower level, either per packet (layer 3) or a data link (layer 2) basis. However, in order to avoid out-of-order delivery, and the consequent need for buffering, not all implementations take advantage of such load balancing.

It is possible to divide the packets according to flows, so that all packets belonging to the same flow, identified by hashing the packet header in the data link layer, may be assigned to the same link. However, the flows may be of very different sizes, and if the characteristics of the links are changing dynamically, any initial assignment of the flows is static and outdated. A reasonable assumption would be that it is not possible to divide optimally according to types of traffic flows verses link capacity.

One solution, is to build all of the data into fixed size packets, and assign the fixed size packets to the different links according to channel capacity, so that for example, slower links get fewer of the fixed size packets. The advantage over using the original packets is that the different incoming packets are of varying sizes so that computing how much capacity is being assigned to each channel is simplified by building the incoming data into fixed size packets. However, flow control for such a system requires an additional port, and although the system makes a reasonable distribution between the channels, it does not adapt for dynamic changes, so that as channel conditions change, the system continues to send the same proportions of the fixed size packets down the different links.

The question thus arises as to whether it is possible to reduce latency, and improve throughput while at the same time reducing the need for memory resources for buffering, when using a non-symmetric dual or multiple link.

SUMMARY OF THE INVENTION

The present embodiments build the incoming data into frames for sending down the links, however, the frames, are sized for the current characteristics of the link down which they are to be sent. The frames are thus adaptive to dynamic bandwidth and latency conditions while being agnostic to the content or type of traffic. Thus, the incoming data packets are split between the available radio link resources and dynamic changes in the radio link are automatically taken into account, simply by updating the frame size.

The frames are built from the packets making up the different flows arriving at the link, and since they are sized for the current channel conditions, all may be expected to arrive at more or less the same time. Jumbo and small packets are inserted alike into the next available frame space, so that small packets are not held up by the jumbo packets. The frames are filled according to the size of the frame with the traffic that arrived regardless of the end and beginning of a new packet.

That is to say, that defining the frame size may comprise considering the channel characteristics of the links, and assigning to each link a frame size that provides an equivalent throughput over all of the links. Thus, a slower link gets a smaller frame and a faster link gets a larger frame. The packets then fill the frames and all arrive more or less in order irrespective of which link they have been assigned to.

Then, as the channel conditions change, the frame size may be varied accordingly, but the packets continue to be assigned to available frames as before.

Jumbo packets are assigned to frames in the same way as any other packet, and if the frames are not large enough then the jumbo packets may be split between different frames.

According to an aspect of some embodiments of the present invention there is provided apparatus for relaying a signal in balanced manner over at least two non- symmetrical links, comprising: a first input for receiving data; a second input for dynamically receiving channel characteristics from each of at least two non-symmetrical links connected to the apparatus; a framer, connected to the second input and configured to assign a frame size to each of the at least two non-symmetrical links based on respective channel characteristics of the respective non-symmetric link, the apparatus configured to construct frames for each link and insert the incoming data into the frames, the inserting being according to the assigned frame sizes; and a transmitter, configured to transmit each frame via the respectively assigned link, thereby to transmit the data in balanced manner over the at least two non-symmetrical links.

In an embodiment, the data received at the first input is arranged in packets, and the apparatus is configured to build the packets into the frames in accordance with a size of a respective frame.

In an embodiment, the second input is connected to respective adaptive coding modulators, which adaptive coding modulators are configured to obtain respective channel characteristics.

In an embodiment, there may be a change a respective assigned frame size upon detection of changes in respective channel characteristics of the link. The apparatus may split the data received between separate frames in order to retain the respectively assigned frame size.

The apparatus may assign each frame with a frame header and a payload, the payload comprising packets assigned to the frame, and wherein the frame header and the payload add up to the respectively assigned frame size.

In an embodiment, the respectively assigned frame size is additionally selected to optimize between increased latency and decreased throughput.

In an embodiment, the respectively assigned frame size is limited by available stack resources at a receive end.

According to a second aspect of the present invention there is provided a method for relaying a signal in balanced manner over at least two non-symmetrical links, comprising: receiving data; dynamically receiving channel characteristics from each of at least two non-symmetrical links on which the data is to be relayed to a data destination; for each of the at least two non-symmetrical links defining a frame size based on respective received channel characteristics; assigning frames to each of the at least two non-symmetrical links, each frame being assigned a respective one of the defined frame sizes; building the incoming data into respectively assigned frames to fill the frame to the respectively assigned frame size; and transmitting each frame via the respectively assigned link, thereby to transmit the data in balanced manner over the at least two non-symmetrical links.

In an embodiment, the received data is arranged in packets.

In an embodiment of the method, the received data is received from a plurality of data connections and / or a plurality of data flows, each data connection or data flow having different characteristics.

The method may comprise obtaining the respective channel characteristics from parameters of an adaptive coding modulator assigned to the respective channel.

The method may comprise dynamically changing a defined frame size associated with a respective link upon detection of changes in respective channel characteristics of the link.

The method may comprise splitting packets between separate frames in order to retain the respectively assigned frame size. The method may comprise assigning each frame with a frame header and a payload, the payload comprising packets assigned to the frame, and wherein the frame header and the payload add up to the respectively assigned frame size.

In an embodiment of the method, the respectively assigned frame size is additionally selected to optimize between increased latency and decreased throughput.

The method may comprise limiting the respectively assigned frame size according to available stack resources at a receive end.

In an embodiment of the method the defining the frame size comprises: considering the channel characteristics of all of the at least two links; and assigning to each link a frame size to provide an equivalent throughput over all of the links.

In an embodiment of the method, the defining the frame size comprises: considering the channel characteristics of all of the at least two links; and assigning to each link a frame size to provide an equivalent latency over all of the links. In an embodiment of the method, the channel characteristics include channel bandwidth.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

Implementation of the method and/or system of embodiments of the invention can involve performing or completing selected tasks using hardware, software or firmware or by a combination thereof, for example using an operating system.

For example, hardware for performing selected tasks according to embodiments of the invention could be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the invention could be implemented as a plurality of software instructions being executed by a computer and which may use any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to exemplary embodiments of method and/or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk and/or removable media, for storing instructions and/or data. Optionally, a network connection is provided as well. A display and/or a user input device such as a keyboard or mouse are optionally provided as well.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

In the drawings:

FIG. 1 is a simplified diagram illustrating the general case of transmitting via more than one radio link, to which the present embodiments may be applied;

FIG. 2 is a simplified block diagram illustrating a device that creates frames sized according to the dynamic characteristics of different links and inserts packets therein, according to embodiments of the present invention;

FIG. 3 is a simplified flow chart illustrating operation of a device according to embodiments of the present invention for sending data in balanced fashion over non- symmetrical radio links;

FIG. 4 is a simplified block diagram showing a multi-link radio connection using the device of Fig. 2 or the method of Fig. 3 in order to balance the transmission of data between the links;

FIG. 5 is a simplified diagram showing a typical datastream being divided into frames according to embodiments of the present invention;

FIG. 6 is a simplified diagram illustrating a structure for a frame according to embodiments of the present invention;

FIG. 7 is a simplified diagram illustrating the use of an embodiment of the present invention over a link comprising a fast channel and a slow channel;

FIG. 8 is a simplified diagram showing a datastream including a jumbo packet being processed by the present embodiments; and

FIG. 9 is a simplified comparative diagram showing the prior art LAG system against a CRB load balancing system according to the present embodiments.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention, in some embodiments thereof, relates to electronic communications over parallel and non- symmetrical links, for example where the links are radio links. The present embodiments may provide a method for relaying a signal in balanced manner over two or more non- symmetrical links, as follows:

Data is received from connections or data flows. At the same time, channel characteristics are dynamically received from the links available to a data destination.

Then, for each of the available links a current frame size is defined, based on the channel characteristics current for that channel.

Frames are made available for the links based on the currently defined frame size for that link and the incoming data is inserted into the available frames, typically on a first come, first served basis.

The data is built into the assigned frames to fill up the assigned frame size, and the frame is then transmitted via the appropriate link. The assignment of packets into frames which are sized for the link characteristics may ensure that the packets arrive at their destination more or less in order and thus less memory resources are needed at the receive end to reconstruct the data flows in the correct order.

Thus, defining the frame size may be predefined per bandwidth or capacity of the specific link. In an alternative embodiment the frame size may be defined by considering the channel characteristics of all of the available links; and assigning to each link a frame size to provide an equivalent throughput or an equivalent latency over all of the links.

It is noted that the term non-symmetric links may refer to links with widely differing characteristics, but may also refer to links with generally identical characteristics but wherein the characteristics may change dynamically, say due to atmospheric conditions, as a function of different frequencies, or due to temporal interference. In these and other cases, there may be temporary periods in which the links are non-symmetric.

Embodiments may provide a method and apparatus for relaying a signal in balanced manner over at least two non- symmetrical links, involves receiving the data for sending, and dynamically receiving channel characteristics from two or more non-symmetrical links to a data destination. Then a frame size is defined for each link based on its current channel characteristics so that all frames transit their links a full capacity. The incoming data is then applied to the frames, each frame being assigned to and sized for a specific link. The frames are thus filled with payload up to their frame size and sent via the respective link. Thus, data may be transmitted in balanced manner over the available links even though the links are not symmetrical. In embodiments, the frame size can be changed for each frame. More practically, a change in frame size may be carried out in the event of changes in the capacity that the Modem/radio can support. For purposes of better understanding some embodiments of the present invention, reference is first made to the construction and operation of a known load balancer as illustrated in Figure 1.

In Fig. 1, node A, 10 and node B, 12 are connected via two radio links, 14 and 16, which are not symmetrical. Nodes A and B connect nodes C, 18 and D, 20, each of which has a respective Ethernet, 22 and 24. Nodes A and B serve as a link between the two Ethernets.

In order to pass the data between nodes A and B, it may be possible simply to split the data up on a per packet basis, as discussed in the background. However, if one of the two links is faster than the other, and the packets are evenly assigned between the links, then packets sent to one of the links will arrive later than the others, increasing the amount of holding needed at the receiving end to get all the packets in order, thus causing latency and greater variation in delay.

As discussed in the background a number of solutions are available. It is possible to balance the network load across all links. Load balancing may involve dividing traffic on a per network socket basis. Another solution for balancing network load is channel bonding, which involves a division of traffic between physical interfaces at a lower level, either per packet (layer 3) or a data link (layer 2) basis. These solutions however do not solve the problem of out of order delivery of packets and may even make it worse, since data flows may be split between the links. Thus, the need for buffering may in fact be increased. These solutions furthermore do not make optimal utilization of the links in terms of throughput and delay variation.

As further mentioned above, it is possible to divide the packets according to flows, so that all packets belonging to the same flow, identified by hashing the packet header (level 1), may be assigned to the same link. However, the flows may be of very different sizes, and if the characteristics of the links are changing dynamically, any initial assignment of the flows may need updating.

A further solution mentioned above is to build all of the data into fixed size packets, and assign the fixed size packets to the different links according to channel capacity. However, flow control for such a system requires an additional port, and although the system makes a reasonable distribution between the channels, it does not adapt for dynamic changes.

Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the Examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

Reference is now made to Fig. 2, which illustrates apparatus 30 for relaying a signal in balanced manner over two or more non- symmetrical links 32.1...32.n according to a first embodiment of the present invention. A data input 34 receives data from one or more data connections, typically in the form of packets that belong to several different data flows, and have different sizes and priority.

A channel characteristics input 36 dynamically receives channel characteristics from each of the channels or links connected to the apparatus across which it is intended to send the packets to their destinations.

A sizer 37 obtains the channel characteristics for the different channels 32.1...32.n and calculates a frame size for each channel. The frame size is calculated such that for the given channel characteristics, each frame takes more or less the same amount of time to cross the channel. Thus, slower channels have smaller frames and faster channels have larger frames. The load between the channels can further be managed in terms of the number of frames sent over each of the links. For example, if the first link is much faster than the second link then several frames may be sent over the first link while only one is sent over the slower link.

The incoming payload may then be inserted into the available frames using framer 38. The frames may be entirely agnostic to the packets that they carry and the packets know nothing about the frames. Nevertheless, packets from different flows may be assigned to frames that belong to different links, but the confidence that succeeding packets will arrive at the far side of the link more or less in order is high. The frames, once full, are assigned to the different channels in frame to channel assignor 40 and transmitted using transmitters for the respective channels, here symbolically illustrated by a single transmitter block 42.

Accordingly, the data may be transmitted in a balanced manner over the available non- symmetrical links, and the only adjustment that needs to be made as channel characteristics vary, is to change the size of the frame for that link.

The data for transmission is typically arranged in packets, and the packets are built into the frames in accordance with the size of the given frame. In the event that a given packet does not fit, as may typically happen with jumbo packets, then the packet may be split between succeeding frames. As all the frames generally arrive with low delay variation, the packet may be reconstructed at the receiving end without requiring too much in the way of buffering resources.

The channel characteristic input may be connected to an adaptive coding modulator ACM 44.1...44.n which is a convenient way to obtain the channel characteristics dynamically. Specifically, the ACM is part of the Modem which constructs the channel. The ACM adjusts the modulation of the Modem based on the quality of the link, thus changing the capacity that the link supports at that point. A higher ACM profile is equal to a higher capacity that can be sent over the link, meaning a faster link and vice versa. Accordingly, the frame size associated with any given channel may be changed upon detection of changes in the channel characteristics of the link as being used by the ACM.

Each frame may be assigned by the framer with a frame header, and the packets built into that frame form its payload. The frame header and the payload may add up to the assigned frame size.

The frame size relative to the other links is selected based on the channel conditions. However, the assigned frame size may further be influenced by an overall optimization for the system, between increased latency and decreased throughput. Thus, the system as a whole may be optimized for a given throughput, and then the frame sizes for the faster and slower links are adjusted in view of the given throughput for the system.

The specific frame size or overall throughput may be limited by available stack resources at the receiver end.

Reference is now made to Fig. 3, which illustrates a method for relaying a signal in balanced manner over at least two non-symmetrical links. Input data for transmission is received, 50. Channel characteristics are dynamically received from each of the two or more non-symmetrical links that lead to the destination for the incoming data, 52.

For each of the two or more non-symmetrical links a frame size is defined based on the channel characteristics current for that link, 54.

A frame is then set up for a given one of the links and is assigned the frame size defined for that link, 56. The incoming data is then built into the frames to fill each frame to the given frame size, 58, and then the frame is sent via the appropriate link or channel, 60. The method then continues with the next frame.

As discussed above, defining the frame size may comprise considering the channel characteristics of all of the links and assigning to each link a frame size to provide an equivalent throughput or latency over all of the links.

The channel characteristics may include the channel bandwidth.

Reference is now made to Fig. 4, which is a simplified diagram showing a device 70 for transmitting data over two or more non-symmetrical radio links according to embodiments of the present invention. Traffic arrives from associated Ethernet connections 72 and may include latency sensitive traffic 74 indicated as RE (Reduced Latency) The branch 74 uses a mechanism that cuts through a queue and handles the assignment of these packets to a frame so as to reduce delay and latency. The Ethernet traffic may be placed into frames using the generic framing protocol GFP. ESS 70 or Ethernet Sub-System, also known as NSS - Network Sub-System, may carry out certain processing on the data such as catch in band management packets or protocols that requires the system to react.

The Communication over non-symmetrical Radio links - Balancing (CRB) block 76 then carries out the functions discussed above in respect of Figs. 2 and 3 and places the packets from the connections into CRB frames that are sized for the individual radio links according to their current characteristics as dynamically measured. The data then splits into the separate radio links 78.

Reference is now made to Fig. 5, which illustrates a sequence of packets passing through various stages of the device 70 shown in Fig. 4, and three stages, (1), (2) and (3) are indicated by numbers in brackets. Stage 1 shows the ethemet packets A and B with the Ethernet overhead, Eth OH. The packets include Ethemet headers, Eth HDR, the payload and a checksum CRC. IPG indicates idle time between packets.

In stage 2, the Ethernet overhead is replaced by a header of the generic framing protocol (GFP) and the stream of packets is divided into frames, indicated by the scissors. In stage 3, a CRB header is added to each frame. It is seen that in this instance, the payload of Ethernet packet A is split between CRB frame 1 and CRB frame 2.

Reference is now made to Fig. 6, which illustrates an exemplary CRB frame structure 80. A CRB header 82 is provided as well as a pay load part 84. The CRB header encodes the length of the frame, and holds a sequence number, a reserved spot, an RL flag, which indicates payload from the RL reduce latency input and a checksum CRC.

It is noted that for a given link, a larger frame size leads to increased latency and requires larger buffering resources at the receiving end. A smaller value of frame size may improve latency and buffering requirements but may decrease throughput.

Reference is now made to Fig. 7, which is a simplified diagram showing how the embodiments of the present invention may achieve load balancing. As shown in Fig. 7, two Ethernets 90 and 92 are connected by a fast link 94 and a slow link 96. Frames, or as they are described in Fig. 7, frames, 0, 2 and 4 go via the fast link, and frames 1, 3 and 5 go via the slow link. The fast link has frames of size 2048 bytes and the slow link has frames of size 64 bytes, nevertheless the three frames on the fast link take as much time to travel as the three frames on the slow link, so that all packets, arrive more or less in the required order, irrespective of which link they are sent on.

Reference is now made to Fig. 8, which is a variation of Fig. 6 showing how the present embodiments may work when there is a very large or jumbo packet 100. As shown in (1) the jumbo packet has an Ethernet header ETH HDR, a pay load and a checksum CRC. In (2) GFP headers are added and the datastream is cut into frames. In (3) a CRB header is added to each frame, and the jumbo packet is now split between three different frames, each frame having its own CRB header.

Reference is now made to Fig. 9, which is a simplified diagram illustrating a comparison between link aggregation group (LAG) and embodiments of the present invention. As will be recalled, A link aggregation group (LAG) is the combined collection of physical ports for such a multiple link communication, and generally the data to be transmitted is split between the available channels at the packet level. That is to say, individual packets are assigned to one or other of the channels for transmission. The arriving packets can be of different sizes, depending on the origin and purpose of the data, and different packets may have different priority levels. Furthermore, the various network connections, the transmission channels, may not be symmetric but may have different channel characteristics, such as different bandwidths and different latencies. Furthermore, the channel characteristics may change dynamically. Nevertheless, the packets are simply assigned to the channels in turn and excessive latency may be caused by sending a large packet through the slower link. The stream of incoming packets 110 includes packet #3 which is very much larger than the others. Under LAG, packet #3 may be sent through either of the channels along with one of the smaller packets, Packet #1. The other two small packets, #2 and #4 go via the other link. However even if both links share the same capacity, it is clear that packet #4 will arrive before packet #3 and need to be buffered. Under CRB however the larger packets are simply cut up into frames sized for the link.

It is expected that during the life of a patent maturing from this application many relevant link and framing technologies will be developed and the scopes of these and other terms herein are intended to include all such new technologies a priori.

The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to".

The term “consisting of’ means “including and limited to”.

As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise.

It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment and the present description is to be construed as if such embodiments are explicitly set forth herein. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or may be suitable as a modification for any other described embodiment of the invention and the present description is to be construed as if such separate embodiments, subcombinations and modified embodiments are explicitly set forth herein. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is/are hereby incorporated herein by reference in its/their entirety.