Background

[0001] Very large layer 2 (VLL2) networking technology has been implemented in data center (DC) networks. VLL2 networking technologies such as the transparent interconnection of lots of links (TRILL) and the shortest path bridging (SPB) have been developed and have been standardized by different standards organizations. TRILL protocol is a layer-two network technology defined by Internet Engineering Task Force (IETF). The TRILL protocol introduces Intermediate System-to-intermediate System (IS-IS) adopting layer-three routing technology into a layer-two network. Accordingly, the simplicity and flexibility of layer- two network may be combined with the stability, extensibility and high-performance of layer-three network.

Brief Description of the Drawings

[0002] Features of the present disclosure are illustrated by way of example and not limited in the following figure(s), in which like numerals indicate like elements, in which:

[0003] FIG. 1 is a flowchart illustrating a packet forwarding process according to an example of the present disclosure.

[0004] FIG. 2 is a diagram illustrating a layer-two network in a data center according to an example of the present disclosure.

[0005] FIG. 3 is a diagram illustrating a layer-two network in a data center according to another example of the present disclosure.

[0006] FIG. 4 is a diagram illustrating the structure of a packet forwarding apparatus according to an example of the present disclosure.

[0007] FIG. 5 is a diagram illustrating the structure of a packet forwarding apparatus according to another example of the present disclosure. Detailed Description

[0008] For simplicity and illustrative purposes, the present disclosure is described by referring mainly to an example thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be readily apparent however, that the present disclosure may be practiced without limitation to these specific details. In other instances, some methods and structures have not been described in detail so as not to unnecessarily obscure the present disclosure. Throughout the present disclosure, the terms "a" and "an" are intended to denote at least one of a particular element. As used herein, the term "includes" means includes but not limited to, the term "including" means including but not limited to. The term "based on" means based at least in part on.

[0009] FIG. 1 is a flowchart illustrating a packet forwarding process according to an example of the present disclosure. The packet forwarding process includes following blocks.

[0010] At block 101, Equal-Cost Multipath Routings (ECMPs) are selected from multiple TRILL paths to a gateway.

[0011] At block 102, the source MAC address of a received Ethernet packet whose destination MAC address is the MAC address of the gateway is replaced with a local MAC address.

[0012] At block 103, a path is selected from the ECMPs, and the Ethernet packet whose source MAC address has been replaced is sent via the selected path.

[0013] According to the packet forwarding process shown in FIG. 1, the ECMPs selected from the multiple TRILL paths to the gateway may share in the sending of the packet to the gateway.

[0014] FIG. 2 is a diagram illustrating a TRILL network in a data center according to an example of the present disclosure.

[0015] As shown in FIG. 2, a TRILL network 200 includes gateway Routing Bridges (RBs) 231-234 and non-gateway RBs 221-226. Each of the gateway RBs 231-234 has a layer-three forwarding capability.

[0016] A virtual router 230 is composed of RBs 231-234. For a VLAN1 in the TRILL network 200, the virtual router 230 is assigned a virtual IPl and a virtual MACl, which are used as the IP address and MAC address of a gateway of the VLANl respectively. For a VLAN2 in the TRILL network 200, the virtual router 230 is assigned a virtual IP2 and a virtual MAC2, which are used as the IP address and MAC address of a gateway of the VLAN2. The RBs 231-234 may construct the virtual router 230 according to Hot Standby Router Protocol (HSRP) or Virtual Router Redundancy Protocol (VRRP). The RB 231 may be selected as a primary device of the virtual router 230. The RB 231 may send an Address Resolution Protocol (ARP) response packet corresponding to an ARP request packet for the IP address of the gateway of the VLANl or the IP address of the gateway of the VLAN2.

[0017] The RBs 231-234 all keep on an active state and are used as gateways of the VLANl and the VLAN2.

[0018] The IP address of a gateway of an end- station belonging to the VLANl is set as the virtual IP1, and the IP address of a gateway of an end-station belonging to the VLAN2 is set as the virtual IP2.

[0019] Each of the RBs 231-234 advertises the gateway information of the VLANl and the gateway information of the VLAN2 in the TRILL network 200. The gateway information advertised by each of the RBs 231-234 may include information for indicating the MAC addresses of the gateways and information for indicating the current RB to be used as the gateway. Accordingly, all RBs in the TRILL network 200 may learn that the virtual MAC1 is gateway MAC addresses of the VLANl, the virtual MAC2 is gateway MAC addresses of the VLAN2 are the virtual MAC2, and the gateway of the VLANl and the gateway of the VLAN2 are implemented by the RBs 231-234. Methods for advertising the gateway information by each of the RBs 231-234 are not limited in the present disclosure. In an example, each of the RBs 231-234 may send a conventional protocol packet in the TRILL network 200 through a TRILL multicast tree, and the gateway information may be contained in an undefined field of the conventional protocol packet. In another example, each of the RBs 231-234 may send a self-defined gateway advertisement message in the TRILL network 200 through the TRILL multicast tree, and the gateway information may be contained in the gateway advertisement message. Accordingly, it should be ensured that the protocol type of the self-defined gateway advertisement message can be recognized by all RBs in the TRILL network 200.

[0020] The RB 221 calculates TRILL paths from the RB 221 to the gateway (including the RBs 231-234) of the VLANl according to the topology of the TRILL network 200. Then, in a specific example corresponding to the method of block 101 of Figure 1, the RB 221 selects TRILL paths with the smallest number of hops as ECMPs to the gateways of the VLAN1.

[0021] In FIG. 2, the ECMPs selected by the RB 221 to the gateways of the VLAN1 include a path from the RB 221 to the RB 231, a path from the RB 221 to the RB 232, a path from the RB 221 to the RB 233 and a path from the RB 221 to the RB 234. The ECMPs selected by the RB 221 to the gateway of the VLAN2 are the same as the ECMPs to the gateway of the VLAN1.

[0022] The RBs 222-226 may select ECMPs to the gateway of the VLAN1 and ECMPs to the gateway of the VLAN2 based on the same principle.

[0023] An end- station 152 belonging to the VLAN1 is connected to the RB 222, and an end-station 153 belonging to the VLAN2 is connected to the RB 223.

[0024] If the end-station 152 does not find a MAC address corresponding to the virtual IP1, the end-station 152 generates an ARP request packet for the virtual IP1. In the ARP request packet, a target IP address is the virtual IP1, a sender IP address is IP152, a sender MAC address is MAC 152, a source MAC address is MAC 152, and a destination MAC address is an all F MAC address, i.e. the destination MAC address is a broadcast address. The end-station 152 sends out the ARP request packet for the virtual IP1.

[0025] The RB 222 receives the ARP request packet from the end- station 152, and learns a MAC address entry based on the VLAN1, MAC 152 (the source MAC address) and a port receiving the APR request packet. The RB 222 encapsulates the ARP request packet into a TRILL-encapsulated ARP request packet. In the TRILL-encapsulated ARP request packet, the ingress nickname and egress nickname of a TRILL header are nickname222 (the nickname of the RB222) and the nickname of the root of a distribution tree respectively. The distribution tree for sending the ARP request packet has been specified by the egress nickname. The source MAC address and destination MAC address of an outer Ethernet header of the TRILL-encapsulated ARP request packet are the MAC address of a TRILL interface of the RB 222 and an All-RBridges multicast address respectively.

[0026] The RBs 221 and 223-226 receive the TRILL-encapsulated ARP request packet, and learn MAC address entry based on nickname222, the VLAN1 and MAC152(the inner source MAC address). The RBs 221 and 223-226 remove the outer Ethernet header and the TRILL header, and broadcast the ARP request packet via a port associated with the VLAN1.

[0027] The RBs 231-234 receive the TRILL-encapsulated ARP request packet, learn MAC address entry based on nickname222, the VLANl(the inner VLAN ID) and MAC 152, and learn an ARP entry of IP 152 based on a port receiving the TRILL-encapsulated ARP request packet, MAC 152 (the sender MAC address), the VLAN 1 and IP152 (the sender IP address).

[0028] The RB 231 generates an ARP response packet for the virtual IP1 which is requested by the received ARP request packet. In the ARP response packet, a target IP address is IP152, a target MAC address is MAC152, a sender IP address is the virtual IP1, a sender MAC address is the virtual MACl, a source MAC address is MAC2312 and a destination MAC address is MAC 152. The RB 231 sets a second MAC address MAC2312 of a Virtual Switching Interface (VSI) of the VLAN1 on the RB 231 as the source MAC address of the ARP response packet, rather than sets the virtual MAC l as the source MAC address of the ARP response packet. In this way, other RBs may not learn an entry of the virtual MACl based on the source MAC address in the received ARP response packet, i.e., the MAC address entry of the gateway of the VLAN1.

[0029] The RB 231 encapsulates the ARP response packet into a TRILL-encapsulated ARP response packet based on the learned entry of MAC 152. In the TRILL-encapsulated ARP response packet, an ingress nickname is nickname231, an egress nickname is nickname222, and the outer source MAC address and outer destination MAC address of an outer Ethernet header are the MAC address of a TRILL interface of the RB 231 and the MAC address of a TRILL interface of the next-hop RB respectively. In the TRILL network 200, the outer source MAC address and outer destination MAC address of a unicast TRILL packet are modified hop by hop. The outer source MAC address and outer destination MAC address of the unicast TRILL packet denote each-hop RB sending the unicast TRILL packet and the next-hop RB to the egress nickname. In FIG.2, the RB 231 sends the TRILL-encapsulated ARP response packet to the RB 222.

[0030] The RB 222 receives the TRILL-encapsulated ARP response packet, determines that the outer destination MAC address and the egress nickname are the MAC address of a TRILL interface of the RB 222 and the nickname of the RB 222 respectively, removes the outer Ethernet header and the TRILL header, finds an entry matching the destination MAC address MAC 152 in a MAC address table, and sends the ARP response packet to the end-station 152.

[0031] The end-station 152 receives the ARP response packet, learns an ARP entry of the virtual IP1. Based on the ARP entry of the virtual IP1, the end- station 152 encapsulates the IP packet whose destination IP address is IP153 into an Ethernet packet whose destination MAC address is the virtual MAC1 and sends out the encapsulated Ethernet packet. That is, in the Ethernet packet sent by the end-station 152, the destination MAC address is the virtual MAC1, and the source MAC address is MAC 152.

[0032] The RB 222 receives the Ethernet packet whose destination MAC address is the virtual MAC1 from the end-station 152 and replaces the source MAC address with MAC222. This is a specific example of the method block 102 of Figure 2 where a MAC address of the gateway is replaced with a local MAC address. The RB 222 then selects a path and sends the Ethernet packet via the selected path as explained above in method block 103 of Figure 2. For example, the RB may select a path from the RB 222 to the RB 233, from ECMPs to the gateway of the VLANl, and send the Ethernet packet whose source MAC address has been replaced to the RB 233. In this way, in the TRILL network, the Ethernet packet of which destination is gateway MAC address of VLANl is sent to the VLANl gateway via one of ECMPs without being encapsulated into a TRILL packet. The VLANl gateway is implemented by the gateway RBs 231-234. As the Ethernet packet is not encapsulated in a TRILL packet before sending to the VLANl gateway, the gateway RBs 231-234 may not have to implement TRILL decapsulating before layer-three forwarding when they receive the Ethernet packet whose destination MAC address is MAC address of a gateway. The MAC address of the VSI of the VLANl to which the RB 222 belongs is MAC222, and is a local MAC address assigned to the RB 222.

[0033] In FIG. 2, on the RB 222 and other non-gateway RBs, the VSI of the VLANl and the VSI of the VLAN2 have different IP addresses, but have the same MAC address or have different MAC addresses.

[0034] The RB 222 may calculate a HASH value based on a five-tuple of the received Ethernet packet, select a path corresponding to the HASH value from the ECMPs, and send the Ethernet packet to the RB 233 via the selected path. The RB 222 modifies the source MAC address of the Ethernet packet to prevent the RBs 231-233 from modifying the egress nickname nickname222 in the learned entry of MAC152. Thus the learned entry of MAC 152 in the RB 231-233 will not be modified frequently. [0035] The RB 233 receives the Ethernet packet whose destination MAC address is the virtual MAC1, and learns an entry of the source MAC address MAC222.

[0036] The RB 233 removes the Ethernet header whose destination MAC address is the virtual MAC1, and performs downstream layer-three forwarding according to the destination IP address IP153. If the RB 233 does not find an ARP entry corresponding to the destination IP address IP153, the RB 233 caches the IP packet whose destination IP address is IP153, and generates an ARP request packet for requesting a MAC address associate with the IP address IP153. In the ARP request packet, the sender IP address is IP2332, the sender MAC address is MAC2331, the target IP address is IP153, the source MAC address is MAC2331, the destination MAC address is an all F MAC address, and the VLAN ID is VLAN2.

[0037] In FIG. 2, the VSI of the VLAN1 and the VSI of the VLAN 2 on the gateway RB 233 have different IP addresses IP2331 and IP2332 respectively, but the two VSIs on the gateway RB 233 have the same first MAC address MAC2331 and the same second MAC address MAC2332. In another example, for the gateway RB 233 and other gateway RBs, the VSI of the VLAN1 and the VSI of the VLAN2 have different IP addresses, different first MAC addresses and different second MAC addresses.

[0038] The RB 233 sets the first MAC address MAC2331 of the VSI of the VLAN2 on the RB 233 as the source MAC address and the sender MAC address of the ARP request packet which requests the MAC address associate with the IP address IP153, and sets the IP address of the VSI of the VLAN2 on the RB 233 as the sender IP address of the ARP request packet. In this way, other RBs may not learn the entry of the virtual MAC2(i.e. the MAC address of VLAN2 gateway) based on the source MAC address of the received ARP request packet, thereby preventing an end- station from learning the ARP entry of gateway based on the sender IP address and sender MAC address of the ARP request packet. Accordingly, the RB 233 does not set the virtual MAC2 as the source MAC address and sender MAC address of the ARP request packet, and also does not set the virtual IP2 as the target IP address of the ARP request packet. Only the RB 231 may send the MAC address responding to the ARP request packet for the IP address of eachgateway.

[0039] The RB 233 encapsulates the ARP request packet into a TRILL-encapsulated ARP request packet. In the TRILL-encapsulated ARP request packet, the ingress nickname and the egress nickname are nickname233 and the nickname of the root of the distribution tree respectively, and the outer source MAC address and the outer destination MAC address are the MAC address of a TRILL interface of the RB 233 and an All-RB ridges multicast address respectively.

[0040] The RBs 221-226 receive the TRILL-encapsulated ARP request packet, and learn the MAC address entry based on nickname233, the VLAN 2 and MAC2331. The RBs 221-226 remove the outer Ethernet header and the TRILL header, and broadcast the ARP request packet via a port associated with the VLAN2.

[0041] The end-station 153 receives the ARP request packet, learns an ARP entry of IP2332, generates an ARP response packet for the ARP request packet for IP153, and sends out the ARP response packet according to the ARP entry of IP2332. In the ARP response packet generated by the end-station 153, the sender IP address and the sender MAC address are IP153 and MAC 153 respectively, the target IP address and the target MAC address are IP2332 and MAC2331 respectively, and the source MAC address and the destination MAC address are MAC153 and MAC2331 respectively.

[0042] The RB 223 receives the ARP response packet, and learns a MAC address entry based on the VLAN 2, the source MAC address MAC 153 and a port receiving the ARP response packet.

[0043] After finding an MAC address entry matching the destination MAC address MAC2331, the RB 223 encapsulates the ARP response packet into a TRILL-encapsulated ARP response packet. In the TRILL-encapsulated ARP response packet, the ingress nickname and egress nickname of the TRILL header are nickname223 and nickname233 respectively, the outer source MAC address and the outer destination MAC address are the MAC address of the TRILL interface of the RB 223 and the MAC address of the TRILL interface of the RB 233.

[0044] Accordingly, the destination MAC address, source MAC address and VLAN ID of the Ethernet header in the ARP response packet are an inner destination MAC address, an inner source MAC address and an inner VLAN ID in the TRILL-encapsulated ARP response packet. The RB 223 sends the TRILL-encapsulated ARP response packet to the RB 233.

[0045] The RB 223 sends the TRILL-encapsulated ARP response packet to the RB 233 according to the outer destination MAC address.

[0046] The RB 233 receives the TRILL-encapsulated ARP response packet, learns a MAC address entry based on nickname223, the inner source MAC address and the inner VLAN ID, and learns the ARP entry of IP 153 based on the port of receiving the TRILL-encapsulated ARP response packet, the inner VLAN ID, the sender IP address IP 153 and the sender MAC address MAC 153.

[0047] The RB 233 encapsulates an IP packet whose destination IP address is IP153 into an Ethernet packet based on MAC2331, MAC153 and the VLAN2 in the ARP entry of the destination IP address IP153, and sends out the Ethernet packet via an egress port in the ARP entry of the destination IP address IP153. When performing downstream layer-three forwarding, the RB 233 encapsulates the IP packet into the Ethernet packet and sends out the Ethernet packet without implementing TRILL encapsulating.

[0048] Since the RB 223 sends the TRILL-encapsulated ARP response packet to the RB 233 via a unicast path in the TRILL network 200, the RB 233 receives the TRILL-encapsulated ARP response packet from a port on a TRILL path which is the shortest path. Therefore, when the RB 233 sends out the Ethernet packet via the egress port in the ARP entry of the destination IP address IP153, the packet is to be forwarded via the shortest path.

[0049] The RB 223 receives the Ethernet packet whose destination MAC address is MAC 153, finds an entry matching MAC 153, and sends the Ethernet packet to the end-station 153.

[0050] When the RBs 231-234 receive the Ethernet packet/IP packet from the outer network of the data center and determines to perform downstream layer-three forwarding according to the destination IP address, the IP packet may be encapsulated into an Ethernet packet according to the above mentioned method and the Ethernet packet is sent out.

[0051] Abnormality processing will be described hereinafter.

[0052] The RBs 221-226 check uplinks which connect the RBs 221-226 with gateway RBs 231-234. The RB 223 detects a failed uplink, and the failed uplink is located on a path between the RB 223 and the RB 233. The path between the RB 223 and the RB 233 is one of the ECMPs from the RB 223 to the VLAN1 gateway and the VLAN2 gateway. The RB 223 deletes the path where the failed uplink is located from the ECMPs to the VLAN1 and the VLAN2.

[0053] The RBs 231-234 check uplinks which connect the outer network and downlinks which connect the RBs 231-234 with the RBs 221-226. The RB 233 detects a failed downlink, and deletes an ARP entry associated with a physical port on the failed downlink. In this example, the RB 233 deletes the ARP entry of IP153.

[0054] When the destination IP address of the Ethernet packet received by the RB 233 is IP153, the RB 233 generates an ARP request packet for IP153, and encapsulates the ARP request packet into the TRILL-encapsulated ARP request packet.

[0055] The RBs 221-226 receive the TRILL-encapsulated ARP request packet, remove the outer Ethernet header and the TRILL header, and broadcast the ARP request packet via a port associated with the VLAN2.

[0056] The end-station 153 receives the ARP request packet, finds the ARP entry of IP2332, generates an ARP response packet for the ARP request packet for IP153, and sends out the ARP response packet according to the ARP entry of IP2332.

[0057] The RB 223 receives the ARP response packet, finds the MAC address entry matching the destination MAC address MAC2331, and encapsulates the ARP response packet into the TRILL-encapsulated ARP response packet. In the TRILL-encapsulated ARP response packet, the ingress nickname and egress nickname of the TRILL header are nickname223 and nickname233 respectively, the source MAC address and destination MAC address of the outer Ethernet header are the MAC address of the TRILL interface of the RB 223 and the MAC address of the TRILL interface of the RB 232 respectively.

[0058] The RB 223 sends the TRILL-encapsulated ARP response packet to the next-hop RB 232. The RB 232 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry associated with MAC153 based on nickname223, the inner VLAN ID and the inner source MAC address, modifies the outer source MAC address and the outer destination MAC address based on egress nickname233, and sends the TRILL-encapsulated ARP response packet to the next-hop RB 222.

[0059] The RB 222 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry associated with MAC 153 based on nickname223, the inner VLAN ID and the inner source MAC address, and sends the TRILL-encapsulated ARP response packet to the next-hop RB 233.

[0060] The RB 233 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry associated with MAC 153 based on nickname223, the inner VLAN ID and the inner source MAC address, and learns the ARP entry of IP153 based on the port receiving the TRILL-encapsulated ARP response packet, the inner VLAN ID, the sender IP address IP153 and the sender MAC address MAC153.

[0061] The RB 233 generates an Ethernet header based on MAC2331 and the ARP entry of the destination IP address IP153. In the Ethernet header, the source MAC address is MAC2331, the destination MAC address is MAC153, the VLAN ID is the VLAN2. The RB 233 adds the Ethernet header into the IP packet whose destination IP address is IP153 to create an Ethernet packet, and sends out the Ethernet packet via an egress port in the ARP entry of the destination IP address IP153.

[0062] The RB 223 sends the Ethernet packet whose destination MAC address is MAC 153 to the RB 222. In this way, when the RB 223 performs downstream layer-three forwarding, what is sent is an Ethernet packet instead of a TRILL packet.

[00633] For a network apparatus whose forwarding chip may not posses sufficient processing capacity, it can not perform layer-3 forwarding immediately after TRILL decapcasulation. When such network apparatus is used the gateway RBs, it has to send Ethernet unicast frame decapsulated from TRILL packet to its loopback-interfaces first, and then performs layer-3 forwarding based on the Ethernet unicast frame which received at its loopback-interfaces. However, available user interfaces on such network apparatus are reduced due to some of the user interfaces are configured as loopback-interfaces, layer- 3 forwarding capacity of such network apparatus might be restricted by the number of the loopback- interfaces.

[00644] In above examples, non-gateway RBs 221-226 calculate ECMPs towards gateway RBs 231-234 based on TRILL network topology, and transmit Ethernet unicast frames needed layer-3 forwarding to the gateway RBs through the ECMPs. Thus, the gateway RBs 231-234 don't have to implement TRILL decapsulation before performing layer-3 forwarding and TRILL encapsulation after performing layer-3 forwarding. The layer-3 forwarding capacity of the gateway RBs 231-234 may not be affected.

[0065] The RB 222 receives the Ethernet packet whose destination MAC address is MAC 153, finds an entry matching MAC 153 in a MAC table, and encapsulates the Ethernet packet into the TRILL-encapsulated Ethernet packet based on the entry matching MAC 153. In the TRILL-encapsulated Ethernet packet, the ingress nickname and egress nickname of the TRILL header are nickname222 and nickname223 respectively, the outer source MAC address and the outer destination MAC address are the MAC address of the TRILL interface of the RB 222 and the MAC address of the TRILL interface of the RB 232. The RB 222 sends the TRILL-encapsulated Ethernet packet to the next-hop RB 232.

[0066] The RB 232 receives the TRILL-encapsulated Ethernet packet, replaces the outer source MAC address and the outer destination MAC with the MAC address of the TRILL interface of the RB 232 and the MAC address of the TRILL interface of the RB 223 respectively, and sends the TRILL-encapsulated Ethernet packet to the next-hop RB 223.

[0067] The RB 223 receives the TRILL-encapsulated Ethernet packet, removes the outer Ethernet header and the TRILL header, finds the entry matching MAC 153 in the MAC table, and sends the Ethernet packet to the end-station 153.

[0068] When the destination MAC address of the Ethernet packet received by the RBs 222 and 223 is not the MAC address of the gateway, routine layer-two forwarding processing is performed on the Ethernet packet.

[0069] The process of requesting for the virtual MAC2 corresponding to the virtual IP2 by the end-station 153 is the same as the process of requesting for the virtual MAC1 corresponding to the virtual IP1 by the end-station 152.

[0070] The RB 231 generates an ARP response packet for the ARP request packet for the virtual IP2, and sets the virtual MAC2 and MAC2312 as the sender MAC address and the source MAC address respectively.

[0071] Based on the learned ARP entry of the virtual IP2, the end-station 153 perform encapsulation and forwarding for an IP packet on which layer-three forwarding is to be performed. The RB 223 receives the Ethernet packet whose destination MAC address is the virtual MAC2, replaces the source MAC address with the MAC address MAC223 of the VSI of the VLAN2 on the RB 223, and selects a path from the RB 223 to the RB 232 to forward the Ethernet packet whose source MAC address has been replaced and whose destination MAC address is the MAC address of the gateway of the VLAN2. The selected path from the RB 223 to the RB232 is one of the ECMPs from the RB223 to the gateway RBs 231-234 Accordingly, the ECMPs may share in the sending of the Ethernet packet to the gateway of the VLAN2.

[0072] When detecting that all uplinks of the RB 233 are failed, the RB 233 advertises the information that all uplinks of the RB 233 are failed to other RBs in the TRILL network 200. The RB 221 deletes the path between the RB 221 and the RB 233 from the ECMPs to the VLAN1 gateway and the VLAN2 gateway. Each of the RBs 222-226 also performs the same processing, that is, deletes a path between the current RB and the RB 233 from the ECMPs to the VLAN1 gateway and the VLAN 2 gateway.

[0073] In this example, an initial packet sent by a gateway RB through perform downstream layer-three forwarding is always an Ethernet packet. When the destination MAC address of an Ethernet packet received by a non-gateway RB is not the MAC address of the gateway, the Ethernet packet is performed according to a routine layer- two forwarding method.

[0074] FIG. 3 is a diagram illustrating a layer-two network in a data center according to another example of the present disclosure.

[0075] As shown in FIG. 3, a TRILL network 300 includes gateway RBs 341-342 and non-gateway RBs 331-334 and 321-326. Each of the gateway RBs 341-342 has a layer-three forwarding capability.

[0076] A virtual router 340 is composed of RBs 341-342. For a VLAN10 in the TRILL network 300, the virtual router 340 is assigned a virtual IP3 and a virtual MAC3, which are used as the IP address and MAC address of a gateway of the VLAN10 respectively. For a VLAN20 in the TRILL network 300, the virtual router 340 is assigned a virtual IP4 and a virtual MAC4, which are used as the IP address and MAC address of a gateway of the VLAN20.

[0077] The RBs 341-342 may construct the virtual router 340 according to the HSRP or the VRRP The RB 341 may be selected as a primary device of the virtual router 340. The RB 341 may send an ARP response packet corresponding to an ARP request packet for the virtual IP3 or the virtual IP4. The RBs 341-342 both keep on an active state and are used as gateway of the VLAN10 and the VLAN20.

[0078] The IP address of a gateway of an end-station belonging to the VLAN10 is set as the virtual IP3, and the IP address of a gateway of an end-station belonging to the VLAN20 is set as the virtual IP4.

[0079] All RBs in the TRILL network 300 know that the gateway of the VLAN10 and the gateway of the VLAN20 are the virtual MAC3 and the virtual MAC4 respectively, and both the gateway of the VLAN10 and the gateway of the VLAN20 are the RBs 341 and 342.

[0080] Each of the RBs 341-342 advertises the gateway information of the VLAN10 and the gateway information of the VLAN20 in the TRILL network 300. The gateway information advertised by each of the RBs 341-342 may include information for indicating the MAC addresses of the gateway and information for indicating the current RB to be used as the gateway. Accordingly, all RBs in the TRILL network 300 may learn that the virtual MAC3 is gateway MAC address of the VLAN10 are the virtual MAC3, the virtual MAC4 is gateway MAC addresses of the VLAN20, and the gateway of the VLANIO and the gateway of the VLAN20 are implemented by the RBs 341-342.

[0081] In an example, each of the RBs 341-342 may send a conventional protocol packet in the TRILL network 300 through a TRILL multicast tree, and the gateway information may be contained in an undefined field of the conventional protocol packet. In another example, each of the RBs 341-342 may send a self-defined gateway advertisement message in the TRILL network 300 through the TRILL multicast tree, and the gateway information may be contained in the gateway advertisement message. Accordingly, it should be ensured that the protocol type of the self-defined gateway advertisement message can be recognized by all RBs in the TRILL network 300.

[0082] The RB 321 calculates TRILL paths between the RB 321 and the gateway (including the RBs 341-342) of the VLANIO according to the topology of the TRILL network 300, and selects TRILL paths with the smallest number of hops as ECMPs to the gateway of the VLANIO.

[0083] The RB 321 calculates TRILL paths between the RB 321 and the gateways of the VLAN20 according to the topology of the TRILL network 300, and selects TRILL paths with the smallest number of hops as ECMPs to the gateways of the VLAN20.

[0084] In FIG. 3, the ECMPs selected by the RB 321 to the gateways of the VLANIO include four ECMPs between the RB 321 and the RB 341, and sending RBs on the ECMPs are the RBs 331-334 respectively. The ECMPs selected by the RB 321 to the gateway of the VLANIO further include four ECMPs between the RB 321 and the RB 342, and transiting RBs on the ECMPs are the RBs 331-334 respectively. The ECMPs selected by the RB 321 to the gateway of the VLAN20 are the same as the ECMPs to the gateways of the VLANIO.

[0085] The RBs 322-326 may select ECMPs to the gateways of the VLANIO and ECMPs to the gateways of the VLAN20 based on the same principle.

[0086] In FIG. 3, the ECMPs selected by the RB 331 to the gateway of the VLANIO and the ECMPs selected by the RB 331 to the gateway of the VLAN20 include a path between the RB 331 and the RB 341 and a path between the RB 331 and the RB 342.

[00875] An end-station 352 belonging to the VLANIO is connected to the RB 322, and an end-station 353 belonging to the VLAN20 is connected to the RB 323.

[0088] If the end-station 352 does not find a MAC address corresponding to the virtual IP3, the end-station 352 generates an ARP request packet for the virtual IP3. In the ARP request packet, a target IP address is the virtual IP3, a sender IP address is IP352, a sender MAC address is MAC352, the source MAC address and destination MAC address of an Ethernet header of the ARP request packet are the MAC352 and an all F MAC address respectively. The end-station 352 sends out the ARP request packet for the virtual IP3 to the RB 322.

[0089] The RB 322 receives the ARP request packet from the end-station 352, and learns an entry of MAC352 based on the VLANIO, the source MAC address of the ARP request packet and a port receiving the APR request packet. The RB 322 encapsulates the ARP request packet into a TRILL-encapsulated ARP request packet, and broadcasts the TRILL-encapsulated ARP request packet in the TRILL network 300.

[0090] The RBs 321 and 323-326 receive the TRILL-encapsulated ARP request packet, learn the MAC address entry based on nickname322, the VLANIO and MAC352. The RBs 321 and 323-326 remove the outer Ethernet header and the TRILL header, and broadcast the ARP request packet via a port associated with the VLANIO.

[0091] The RBs 331-341 being used as transiting RBs learns the MAC address entry of MAC352 based on the ingress nickname, the inner destination MAC address and the inner VLAN ID, and forwards the TRILL-encapsulated ARP request packet based on the egress nickname in the TRILL network 300.

[0092] The RBs 341 and 342 receive the TRILL-encapsulated ARP request packet, learn the MAC address entry based on nickname322, the VLANIO and MAC352, and learn an ARP entry of IP352 based on a port receiving the TRILL-encapsulated ARP request packet, MAC352 (the sender MAC address), an inner VLAN ID and IP352 (the sender IP address).

[0093] The RB 341 generates an ARP response packet for the ARP request packet. In the ARP response packet, a target IP address is IP352, a target MAC address is MAC352, a sender IP address is the virtual IP3, and a sender MAC address is the virtual MAC3. The RB 341 sets the source MAC address and destination MAC address of the Ethernet header in the ARP response packet as the second MAC address MAC3412 of the VSI of the VLANIO on the RB 341 and MAC 352. The RB 341 encapsulates the ARP response packet into a TRILL-encapsulated ARP response packet.

[0094] In FIG. 3, the IP address of the VSI of the VLANIO on the RB 341 and the IP address of the VSI of the VLAN20 on the RB 341 are IP3411 and IP3412 respectively, but the two VSIs have the same first MAC address MAC3411 and the same second MAC address MAC3412. The VSI of the VLAN10 on the gateway RB 342 and the VSI of the VLAN20 on the gateway RB 342 have different IP addresses, but have the same first MAC address and the same second MAC address. In another example, the two VSIs on the gateway RB 341 have different first MAC addresses and different second MAC addresses; the two VSIs on the gateway RB 342 have different first MAC addresses and different second MAC addresses.

[0095] The RB 341 sets the ingress nickname as nickname341, and sets egress nickname as nickname322 according to the learned MAC address entry, and sets the outer source MAC address and the outer destination MAC address as the MAC address of the TRILL interface of the RB 341 and the MAC address of the TRILL interface of the next-hop RB 332 respectively. In FIG. 3, the RB 341 sends the TRILL-encapsulated ARP response packet to the RB 332.

[0096] The RB 332 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry of MAC3412 based on the ingress nickname, the inner destination MAC address and the inner VLAN ID, modifies the outer source MAC address and the outer destination MAC address, and sends the TRILL-encapsulated ARP response packet to the RB 322.

[0097] The RB 322 receives the TRILL-encapsulated ARP response packet, determines that the outer MAC address and the egress nickname are the MAC address and nickname of the TRILL interface of the RB 322 respectively, removes the outer Ethernet header and the TRILL header, finds an entry matching the destination MAC address MAC 352 in a MAC table, and sends the ARP response packet to the end-station 352.

[0098] The end-station 352 receives the ARP response packet, learns an ARP entry of the virtual IP3, encapsulates an IP packet whose destination IP address is IP353 in to a Ethernet packet whose destination MAC address is the virtual MAC3 based on the ARP entry of the virtual IP3, and sends out the Ethernet packet. That is, in the Ethernet packet sent by the end-station 352, the destination MAC address is the virtual MAC3, and the source MAC address is MAC352.

[0099] The RB 322 receives the Ethernet packet whose destination MAC address is the virtual MAC3 from the end-station 352, replaces the source MAC address with MAC322, selects a path, for example, the path RB 322 ->RB 332-> the RB 341, from ECMPs to the gateway of the VLAN10, and sends out the Ethernet packet whose source MAC address has been replaced. The RB 322 modifies the source MAC address to prevent the next-hop RB or the gateway from modifying the egress port in the learned entry of MAC352, that is, nickname322.

[0100] The RB 332 receives the Ethernet packet whose destination MAC address is the virtual MAC3, replaces the source MAC address with MAC332, selects a path, for example, the path from the RB 332 to the RB 341, from ECMPs to the gateway of the VLAN10, and sends out the Ethernet packet whose source MAC address has been replaced. The RB 332 modifies the source MAC address to prevent the gateway from modifying the egress port in the learned entry of MAC352.

[0101] In FIG. 3, on the non-gateway RBs 321-326 and 331-334, the VSI of VLAN 10 and the VSI of the VLAN20 have different IP addresses, but have the same MAC address or have different MAC addresses.

[0102] In this way, the RBs 322 and 332 may utilize the ECMPs to the VLAN10 gateway to share in the sending of the Ethernet packet to the VLAN 10 gateway. The RBs 322 and 333 do not need to encapsulate the Ethernet packet sent to the VLAN10 into the TRILL packet. Correspondingly, the gateway of the VLAN 10 may perform layer-three forwarding without decapsulating the TRILL packet.

[0103] The RBs 322 and 332 may calculate a HASH value according to a five-tuple of the received Ethernet packet, and select a path corresponding to the HASH value from the ECMPs. The RB 341 receives the Ethernet packet whose destination MAC address is the virtual MAC3, and learns the MAC address entry of the source MAC address MAC332.

[0104] The RB 341 removes the Ethernet header of the Ethernet packet whose destination MAC address is the virtual MAC3, and determines to perform downstream layer-three forwarding on the IP packet whose destination IP address is IP353 according to the destination IP address IP353.

[0105] If the RB 341 does not find an ARP entry of the destination IP address IP353, the RB 341 caches the IP packet whose destination IP address is IP353, and generates an ARP request packet for IP353. In the ARP request packet, the sender IP address is the IP address IP3412 of the VSI of the VLAN20 on the RB 341, the sender MAC address is the first MAC address MAC3411 of the VSI of the VLAN20 on the RB 341, and the target IP address is IP352. The VLAN ID, source MAC address and destination MAC address of the Ethernet header in the ARP request packet are the VLAN20, MAC3411 and an all F MAC address respectively.

[0106] The RB 341 encapsulates the ARP request packet into a TRILL-encapsulated ARP request packet, and broadcasts the TRILL-encapsulated ARP request packet in the TRILL network 300.

[0107] The RBs 331-334 receive the TRILL-encapsulated ARP request packet, and learn the MAC address entry based on the ingress nickname, the VLAN20 and MAC3411. The RBs 331-334 forward the TRILL-encapsulated ARP request packet according to the egress nickname.

[0108] The RBs 321-326 receive the TRILL-encapsulated ARP request packet, and learn the MAC address entry based on the ingress nickname, the VLAN20 and MAC3411. The RBs 321-326 remove the outer Ethernet header and TRILL header of the TRILL-encapsulated ARP request packet, and broadcasts the ARP request packet via a port associated with the VLAN20.

[0109] The end-station 353 receives the ARP request packet, learns the ARP entry of IP3412, generates an ARP response packet corresponding to the ARP request packet for the IP353, and sends the ARP response packet according to the ARP entry of IP3412. In the ARP response packet generated by the end-station 353, the sender IP address and the sender MAC address are IP353 and MAC353 respectively, the target IP address and the target MAC address are IP3412 and MAC3411 respectively, and the source MAC address and the destination MAC address are MAC353 and MAC3411 respectively.

[0110] The RB 323 receives the ARP response packet, and learns the MAC address entry based on the VLAN 20, the source MAC address MAC353 and a port receiving the ARP response packet.

[0111] The RB 323 finds an entry matching the MAC address MAC3411, and encapsulates the ARP response packet into a TRILL-encapsulated ARP response packet. In the TRILL-encapsulated ARP response packet, the ingress nickname and egress nickname of the TRILL header are nickname323 and nickname341 respectively, the outer source MAC address and outer destination MAC address of the Ethernet header are the MAC address of the TRILL interface of the RB 323 and the MAC address of the TRILL interface of the next-hop RB 332. The RB 323 sends the TRILL-encapsulated ARP response packet to the next-hop RB 332 based on the outer destination MAC address.

[0112] The RB 332 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry based on ingress nickname323, MAC353 and the VLAN20, modifies the outer source MAC address and the outer destination MAC address, and sends the TRILL-encapsulated ARP response packet to the RB 341.

[0113] The RB 341 receives the TRILL-encapsulated ARP response packet, learns the MAC address entry based on ingress nickname323, MAC353 and the VLAN20, and learns the ARP entry of IP353 based on a port of receiving the TRILL-encapsulated ARP response packet, the VLAN20, IP353 and MAC353.

[0114] The RB 341 generates an Ethernet header based on MAC3411, MAC353 and the VLAN20, adds the Ethernet header into the IP packet whose destination IP address is IP353, and sends out the Ethernet packet via an egress port in the ARP entry of IP353. In this way, when the RB 341 performs downstream layer-three forwarding, what is sent is the Ethernet packet.

[0115] The RB 332 receives the Ethernet packet, finds an entry matching MAC353, encapsulates the Ethernet packet into a TRILL-encapsulated Ethernet packet, and sends the TRILL-encapsulated Ethernet packet to the RB 323.

[0116] The RB 323 receives the TRILL-encapsulated Ethernet packet, removes the outer Ethernet packet and the TRILL header, finds the entry matching MAC353, and sends the TRILL-encapsulated Ethernet packet to the RB 353.

[0117] Abnormality processing will be described hereinafter.

[0118] The RBs 321-326 and 331-334 check uplinks.

[0119] The RB 321 detects a failed uplink, and the failed uplink is located on a path the RB 321->the RB 332-> the RB 341 and a path the RB 321 ->the RB 332->the RB 342.

The RB 321 deletes the two paths where the failed uplink is located on from the ECMPs towards the VLAN 10 gateway and the VLAN20 gateway.

[0120] The RBs 321-326 and 331-334 may send the packet whose destination MAC address is the gateway MAC address of the VLAN10 or the gateway MAC address of the

VLAN20 via the ECMPs to the RB 341.

[0121] In the TRILL network 300 shown in FIG. 3, even if one gateway RB is used as the gateway of each VLAN in the TRILL network 300, the RBs 321-326 may still utilize the ECMPs to the gateway of each VLAN to share in the sending of Ethernet packets to each VLAN, and do not need to encapsulate the received Ethernet packet into the TRILL-encapsulated Ethernet packet. The RBs 331-334 also do not need to encapsulate the received Ethernet packet into the TRILL-encapsulated Ethernet packet. The RB 332 detects a failed uplink, and the failed uplink is located on a path between the RB 332 and the RB 341 and is one of the ECMPs from the RB 332 to the VLAN10 and the VLAN20. The RB 332 deletes the path where the failed uplink is located from the ECMPs to the VLAN10 and the VLAN20.

[0122] Each of the RBs 341 and 342 checks its uplinks and downlinks. The RB 341 detects that a downlink connected to the RB 332 is failed, and deletes the ARP entry of IP353 associated with a physical port of the failed downlink.

[0123] When the destination IP address of the Ethernet packet received by the RB 341 is IP353, the RB 341 resends an ARP request packet for IP353, learns the ARP entry of IP353 again based on the received TRILL-encapsulated ARP response packet, encapsulates the IP packet whose destination IP address is IP353 into the Ethernet packet according to the learned ARP entry, and sends out the Ethernet packet.

[0124] A process of requesting for the MAC address of the gateway of the VLAN20 by the end-station 353 is the same as the process of requesting for the MAC address of the gateway of the VLAN10 by the end-station 352.

[0125] When detecting that all uplinks of the RB 341 are failed, the RB 341 advertises the information that all uplinks of the RB 341 are failed in the TRILL network 300. Each of the RBs 321-326 delete the path between the current RB and the RB 341 from the ECMPs to the VLAN10 and the VLAN20. Each of the RBs 331-334 also performs the same processing, that is, deletes a path between the current RB and the RB 341 from the ECMPs.

[0126] It can be seen from the examples shown in FIGs. 2 and 3 that, when a non-gateway RB in a data center receives an Ethernet packet whose destination MAC address is the MAC address of the gateway, the non-gateway RB replaces the source MAC address with a local MAC address, and forwards the Ethernet packet whose source MAC address has been replaced and whose destination MAC address is the gateway MAC address via the ECMPs to the gateway. In this way, the non-gateway RBs may send to the gateway RBs the Ethernet packet on which the layer-three forwarding is to be performed without encapsulating the Ethernet packet into the TRILL-encapsulated Ethernet packet.

[0127] FIG. 4 is a diagram illustrating the structure of a packet forwarding apparatus according to an example of the present disclosure. The packet forwarding apparatus includes following modules. [0128] A path selecting module 401 may select ECMPs from multiple TRILL paths to a gateway.

[0129] A packet forwarding module 402 may replace the source MAC address of a received Ethernet packet whose destination MAC address is the MAC address of the gateway with a local MAC address, select a path from the ECMPs, and send the Ethernet packet whose source MAC address has been replaced via the selected path.

[0130] A path checking module 403 may check an uplink, and send the state of the uplink to the path selecting module 401.

[0131] When determining that a failed uplink is located on one of the ECMPs, the path selecting module 401 may delete the path where the failed uplink is located from the ECMPs.

[0132] The packet forwarding module 402 may obtain the MAC address of the gateway and more than one gateway RB used as the gateway.

[0133] The path selecting module 401 may calculate multiple TRILL paths to each gateway RB, and select more than one path with the smallest number of hops as the ECMPs from the multiple TRILL paths to each gateway RB.

[0134] The path selecting module 401 may obtain a gateway RB whose uplink is failed, and determine whether the ECMPs include a path to the gateway RB whose uplink is failed. When determining that the ECMPs include a path to the gateway RB whose uplink is failed, the path selecting module 401 deletes the path to the gateway RB whose uplink is failed from the ECMPs.

[0135] The packet forwarding apparatus provided by the example of the present disclosure may utilize the ECMPs in the multiple TRILL paths to the gateway to share in the sending of the packet to the gateway, and does not need to encapsulate the packet sent to the gateway into the TRILL-encapsulated Ethernet packet.

[0136] FIG. 5 is a diagram illustrating the structure of a packet forwarding apparatus according to another example of the present disclosure. As shown in FIG. 5, the packet forwarding apparatus at least includes a memory 501, a processor 502 communicating with the memory 501 and a packet forwarding chip 503. The memory 501, the processor 502 and the packet forwarding chip 503 are connected via a bus. The memory 501 for example stores machine readable instructions that are executed to by the processor 502. The machine readable instructions for example include path selecting instructions. The memory 501 for example is a non-transitory computer storage readable medium. The path selecting instructions may be machine-readable instructions stored in the memory 501. The processor 502 may execute the machine-readable instructions stored in the memory 501.

[0137] The path selecting instructions may select ECMPs from multiple TRILL paths to a gateway. The packet forwarding chip 503 may replace the source MAC address of a received Ethernet packet whose destination MAC address is the MAC address of the gateway with a local MAC address, select a path from the ECMPs, and send the Ethernet packet whose source MAC address has been replaced via the selected path.

[0138] The packet forwarding apparatus also includes path checking instructions, which may check an uplink, and send the state of the uplink to the path selecting instructions. When determining that a failed uplink is located on one of the ECMPs, the path selecting instructions may delete the path where the failed uplink is located from the ECMPs.

[0139] The packet forwarding chip 503 may obtain the MAC address of the gateway and more than one gateway RBs used as the gateway. The path selecting instructions may calculate multiple TRILL paths to each gateway RB, and select more than one path with the smallest number of hops as the ECMPs from the multiple TRILL paths to each gateway RB.

[0140] The path selecting instructions may obtain a gateway RB whose uplink is failed, and determine whether the ECMPs include a path to the gateway RB whose uplink is failed. When determining that the ECMPs include a path to the gateway RB whose uplink is failed, the path selecting instructions delete the path to the gateway RB whose uplink is failed from the ECMPs. [0141] Although described specifically throughout the entirety of the instant disclosure, representative examples of the present disclosure have utility over a wide range of applications, and the above discussion is not intended and should not be construed to be limiting, but is offered as an illustrative discussion of aspects of the disclosure. [0142] What has been described and illustrated herein is an example along with some of its variations. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Many variations are possible within the spirit and scope of the subject matter, which is intended to be defined by the following claims— and their equivalents— in which all terms are meant in their broadest reasonable sense unless otherwise indicated.