SYSTEMS, APPARATUS AND METHODS FOR

MANAGING CONNECTIVITY OF NETWORKED DEVICES

CROSS- REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of, and priority to, United States Provisional Application Serial No. 62/574,396, filed October 19, 2017, the disclosure of which is hereby incorporated herein by reference in its entirety and for all purposes.

FIELD

[0002] This specification and the accompanying figures relate generally to connectivity of networked devices and the associated protocols, including systems, apparatus and methods for the same.

[0003] The systems, apparatus and methods disclosed herein and depicted in the

accompanying figures describe particular embodiments and are not intended to be exhaustive of the contemplated configurations and processes.

BACKGROUND

[0004] In traditional computer system network architectures, devices such as computer servers and storage devices are interconnected through wires (e.g., copper or optical) which transport data between the devices. FIG. 1 A shows an example of a wire 101 over which formatted data is transported, for example, using the Ethernet protocol. The data is typically in the form of "bits" which are represented as "l 's" and "0's" and are physically formed by voltage, current and/or light. Devices can be connected, for example, locally on the same motherboard, or between different pieces of equipment through longer connectors such as copper wires or optical fiber using connectors such as Small Form-factor Pluggable (SFP), Quad Small Form-factor Pluggable (QSFP), or Gigabit Interface Converter (GBIC) modules. FIG. IB shows an exemplary QSFP connected cable 111. [0005] The data on an interconnect or wire is generally encoded into a standard protocol (such as, for example, Ethernet, Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Express (PCIe or PCI-E), Serial Attached SCSI (SAS), Serial AT Attachment (Serial ATA or SATA), FiberChannel, or Infiniband) to accomplish reliable data transfer and interoperability between or among different devices and manufacturers. Two devices using the same protocol can be considered akin to two persons speaking the same language, and, when different protocols or languages are used, one or more translators may be required.

[0006] In certain configurations, there may exist several devices and/or computer applications that require multiple protocols. For example, consider a "server" architecture, in which the server (e.g., a box containing at least one motherboard with microprocessor and running computer programs) is connected to, and intended to provide functionality or services (e.g., performing computations or sharing data or resources) for, other "client" devices or programs.

[0007] The server needs to communicate, through the network, with one or more other devices or applications, which can include a hard disk drive (HDD) or solid-state drive (SSD), a general- purpose graphics processing unit (GP-GPU, GPGPU or GPGP), and/or other devices, applications or servers. To enable such communications, interfaces are used to create physical connections, such as from an on-board processor within the server to a peripheral device. These interfaces run protocols, such as PCIe, DDR (double data rate) and/or others. Each of the devices or other servers can have its own interface protocol. For example, the HDD may employ (connect through) SATA or SAS, and the SSD may employ SAS or PCIe, while the GP-GPU may employ PCIe. The network connecting these devices together may be Ethernet, which is generally considered a reliable and pervasive networking protocol, or another suitable protocol.

[0008] One approach to dealing with a configuration such as described above is to use translators to allow communications involving multiple protocols. For example, a Network Interface Card (NIC) may be used to connect together a server with an HDD or SSD. In that case, the NIC may be inserted between the server and the network connection to translate, for example, from the PCIe to Fiberchannel protocol. On the other end of the network, the HDD or SSD is connected through another NIC, which would translate from Fiberchannel to PCIe, or to a Host Bus Adapter which translates Fiberchannel to SAS or SATA. This translation process can be both inefficient and costly. For example, the NICs introduce latency and overhead in the data transactions which reduces overall system performance and increases power and cost.

[0009] Another approach is to create multiple separate, standalone networks, with each network using a particular protocol. For example, a configuration may include an Ethernet network and a FiberChannel network, each separate and optimized for the type of data being transmitted. However, this approach also has downsides, for example, the need to setup and maintain multiple networks.

[0010] Another approach, sometimes referred to as "tunneling," is to embed one protocol inside another, for example, putting PCIe data "frames" into Ethernet frames to be transported over Ethernet and then returned to PCIe on the other end of the connection. This approach generally reduces the performance of the connection to the lowest common denominator, or the slowest protocol. In this PCIe/Ethernet example, enhanced PCIe performance is lost as the data is transmitted as Ethernet packets during transmission, which results, for example, in certain latency burdens inherent in the Ethernet protocol.

[0011] In addition, the industry has generally standardized or shown a preference for several connector or module types and extensive infrastructure is already in place utilizing these types of connectors. For example, QSFP (such as shown in FIG. IB and discussed above) is a pervasive module/connector that is used for Ethernet connectivity and carries four "lanes" of data, typically Ethernet. There is an inherent bias in the industry to continue using such connectors already in place, if possible, even as connectivity architectures evolve.

[0012] There exists a need for improved multi -protocol connectivity systems, apparatus and methods to overcome obstacles and deficiencies associated with conventional computer networks and environments involving multiple devices and associated protocols. SUMMARY

[0013] This specification and the accompanying figures describe new technologies relating to the management and transportation of data in a networked environment or solution, such as data centers and cloud (or multi-cloud) services and systems. According to various embodiments, these technologies are used to transport data across individual wires or connections using uniform connection technology but different protocols.

[0014] For example, these technologies introduce a more efficient solution by allowing optionality in the connection between devices. The translation between protocols may be needed for certain network connections, but, for other network connections, remaining in the original protocol format would be preferred due to higher performance. The present disclosure solves this problem by enabling multiple protocols to be transmitted over a particular media without requiring translation. The decision as to which protocol is transmitted can be determined through software (e.g., a software-based controller), thus providing flexibility by allowing, for example, a software-defined protocol selection and reconfiguration solution based on application priorities and need.

[0015] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of, for example, receiving data corresponding to at least two communication protocols, detecting the at least two communication protocols, and automatically provisioning communication interfaces according to the detected communication protocols. Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.

[0016] As also disclosed herein, for example, the subject matter further includes methods, systems, apparatus, and computer program products whereby data is received from multiple data communication inputs, at least two types of corresponding communication paths are determined, at least two data paths are automatically combined via multiplexing to result in fewer data streams, and data from the combined data paths is automatically detected and de-multiplexed to be output to a destination. [0017] In accordance with a first aspect disclosed herein, there is set forth a method for managing data connectivity among diverse network devices, comprising: identifying a first communication protocol of received first data; and converting the received first data into a second communication protocol among a second plurality of communication protocols.

[0018] In some embodiments of the disclosed method, said identifying the first communication protocol includes identifying the first communication protocol from among a first plurality of communication protocols and/or identifying the first communication protocol from among a first plurality of communication protocols supported by a first input-output interface.

[0019] In some embodiments of the disclosed method, the method further comprises receiving the first data via the first input-output interface and/or receiving the first data being provided by a first network device via the first input-output interface.

[0020] In some embodiments of the disclosed method, said converting the received first data comprises converting the received first data into the second communication protocol among a second plurality of communication protocols supported by a second input-output interface.

[0021] In some embodiments of the disclosed method, the method further comprises transmitting the converted first data to a second network device via the second input-output interface.

[0022] In some embodiments of the disclosed method, said transmitting the converted first data comprises transmitting the converted first data to the second network device via a second communication connection coupling the second input-output interface with the second network device. Additionally and/or alternatively, said transmitting the converted first data includes transmitting the converted first data to the second network device via a second communication connection that supports the second communication protocol.

[0023] In some embodiments of the disclosed method, the method further comprises identifying a predetermined number of second communication lanes associated with the second communication connection for transmitting the converted first data based upon the second communication protocol. In some embodiments, the predetermined number of the second communication lanes can be a subset of a total number of the second communication lanes.

[0024] In some embodiments of the disclosed method, the method further comprises associating a respective predetermined number of the second communication lanes with each of the second plurality of communication protocols.

[0025] In some embodiments of the disclosed method, the method further comprises associating a first predetermined number of the second communication lanes with a Peripheral Component Interconnect communication protocol and associating a second predetermined number of the second communication lanes with an Ethernet communication protocol. In some embodiments, the first predetermined number of the second communication lanes can be greater than the second predetermined number of the second communication lanes.

[0026] In some embodiments of the disclosed method, the method further comprises combining the first predetermined number of the second communication lanes to reduce the second predetermined number of the second communication lanes.

[0027] In some embodiments of the disclosed method, said transmitting the converted first data includes transmitting the converted first data on a single wire of the second communication connection, said transmitting the converted first data includes combining the converted first data with other data encoded in the second communication protocol and/or said transmitting the converted first data includes multiplexing the converted first data with other data encoded in the second communication protocol.

[0028] In some embodiments of the disclosed method, the method further comprises identifying the second communication protocol based upon one or more communication protocols supported by the second communication connection. In some embodiments, the second communication connection supports only the second communication protocol.

[0029] In some embodiments of the disclosed method, the method further comprises: receiving second data via the second input-output interface; identifying a communication protocol of received second data and converting the received second data into the first communication protocol.

[0030] In some embodiments of the disclosed method, said receiving the second data includes receiving the second data provided by the second network device via the second input-output interface and/or said identifying the communication protocol of the received second data comprises identifying the communication protocol of the received second data as being the second communication protocol.

[0031] In some embodiments of the disclosed method, the method further comprises transmitting the converted second data to the first network device via the first input-output interface. For example, said transmitting the converted second data can comprise transmitting the converted second data to the first network device via a first communication connection coupling the first input-output interface with the first network device and/or said transmitting the converted second data can include transmitting the converted second data to the first network device via a first communication connection that supports the first communication protocol.

[0032] In some embodiments of the disclosed method, the method further comprises identifying a predetermined number of first communication lanes associated with the first communication connection for transmitting the converted second data based upon the first communication protocol. The predetermined number of the first communication lanes can be a subset of a total number of the first communication lanes.

[0033] In some embodiments of the disclosed method, the method further comprises associating a respective predetermined number of the first communication lanes with each of the first plurality of communication protocols.

[0034] In some embodiments of the disclosed method, the method further comprises associating a first predetermined number of the first communication lanes with a Peripheral Component Interconnect communication protocol and associating a second predetermined number of the first communication lanes with an Ethernet communication protocol. The first predetermined number of the first communication lanes can be greater than the second predetermined number of the first communication lanes.

[0035] In some embodiments of the disclosed method, the method further comprises combining the first predetermined number of the first communication lanes to reduce the second predetermined number of the first communication lanes. Additionally and/or alternatively, said transmitting the converted second data includes transmitting the converted second data on a single wire of the first communication connection, said transmitting the converted second data includes combining the converted second data with other data encoded in the first

communication protocol and/or said transmitting the converted second data includes multiplexing the converted second data with other data encoded in the first communication protocol.

[0036] In some embodiments of the disclosed method, the method further comprises identifying the first communication protocol based upon one or more communication protocols supported by the first communication connection. The first communication connection can support only the first communication protocol.

[0037] In some embodiments of the disclosed method, the first plurality of communication protocols and the second plurality of communication protocols share at least one common communication protocol.

[0038] In some embodiments of the disclosed method, the first input-output interface comprises a multiprotocol-capable serializer/deserializer that supports an Ethernet

communication protocol, a Peripheral Component Interconnect communication protocol, a Peripheral Component Interconnect Express communication protocol, a Serial Attached SCSI communication protocol, a Serial AT Attachment communication protocol, a FiberChannel communication protocol, an Infiniband communication protocol or a combination thereof.

[0039] In some embodiments of the disclosed method, the second input-output interface comprises an optical input-output interface device, a copper input-output interface device or a combination thereof. [0040] In accordance with a second aspect disclosed herein, there is set forth a system for managing data connectivity among diverse network devices, comprising means for performing the method of the first aspect.

[0041] In accordance with a third aspect disclosed herein, there is set forth a computer program product for managing data connectivity among diverse network devices, comprising one or more instructions for performing the method of the first aspect.

[0042] In accordance with a fourth aspect disclosed herein, there is set forth a converter device for managing data connectivity among diverse network devices, comprising: a first input-output interface supporting each of a first plurality of communication protocols and for receiving first data provided by a first network device; a second input-output interface supporting each of a second plurality of communication protocols and for communicating with a second network device; and a protocol management circuit for identifying a first communication protocol of the received first data and converting the received first data into a second communication protocol among the second plurality of communication protocols, wherein said second input-output interface transmits the converted first data to the second network device.

[0043] In some embodiments of the disclosed converter device, said second input-output interface communicates with the second network device via a second communication connection that supports the second communication protocol.

[0044] In some embodiments of the disclosed converter device, the second communication connection comprises a plurality of second communication lanes, and wherein said protocol management circuit identifies a predetermined number of the second communication lanes for transmitting the converted first data to the second network device based upon the second communication protocol.

[0045] In some embodiments of the disclosed converter device, the predetermined number of the second communication lanes is a subset of the plurality of second communication lanes. [0046] In some embodiments of the disclosed converter device, said protocol management circuit associates a respective predetermined number of the second communication lanes with each of the second plurality of communication protocols.

[0047] In some embodiments of the disclosed converter device, said protocol management circuit identifies the second communication protocol based upon one or more communication protocols supported by the second communication connection.

[0048] In some embodiments of the disclosed converter device, the second communication connection supports only the second communication protocol.

[0049] In some embodiments of the disclosed converter device, said second input-output interface receives second data provided by the second network device, wherein said protocol management circuit identifies a communication protocol of the received second data and converts the received second data into the first communication protocol, and wherein said first input-output interface transmits the converted second data to the first network device.

[0050] In some embodiments of the disclosed converter device, said protocol management circuit identifies the communication protocol of the received second data as being the second communication protocol.

[0051] In some embodiments of the disclosed converter device, said first input-output interface communicates with the first network device via a first communication connection that supports the first communication protocol.

[0052] In some embodiments of the disclosed converter device, the first communication connection comprises a plurality of first communication lanes, and wherein said protocol management circuit identifies a predetermined number of the first communication lanes for transmitting the converted first data to the first network device based upon the first

communication protocol.

[0053] In some embodiments of the disclosed converter device, the predetermined number of the first communication lanes is a subset of the plurality of first communication lanes. [0054] In some embodiments of the disclosed converter device, said protocol management circuit associates a respective predetermined number of the first communication lanes with each of the first plurality of communication protocols.

[0055] In some embodiments of the disclosed converter device, said protocol management circuit identifies the first communication protocol based upon one or more communication protocols supported by the first communication connection.

[0056] In some embodiments of the disclosed converter device, the first communication connection supports only the first communication protocol.

[0057] In some embodiments of the disclosed converter device, the first plurality of communication protocols and the second plurality of communication protocols share at least one common communication protocol.

[0058] In some embodiments of the disclosed converter device, said first input-output interface comprises a multiprotocol-capable serializer/deserializer that supports an Ethernet

communication protocol, a Peripheral Component Interconnect communication protocol, a Peripheral Component Interconnect Express communication protocol, a Serial Attached SCSI communication protocol, a Serial AT Attachment communication protocol, a FiberChannel communication protocol, an Infiniband communication protocol or a combination thereof.

[0059] In some embodiments of the disclosed converter device, said second input-output interface comprises an optical input-output interface device, a copper input-output interface device or a combination thereof.

[0060] In accordance with a fifth aspect disclosed herein, there is set forth a method for managing data connectivity among diverse network devices, comprising steps for performing the operations associated with the converter device of the fourth aspect.

[0061] In one embodiment, for example, the disclosed method can comprise: receiving first data provided by a first network device via a first input-output interface supporting each of a first plurality of communication protocols; identifying a first communication protocol of the received first data; converting the received first data into a second communication protocol among a second plurality of communication protocols supported by a second input-output interface; and transmitting the converted first data to a second network device via the second input-output interface.

[0062] In accordance with a sixth aspect disclosed herein, there is set forth a computer program product for managing data connectivity among diverse network devices, comprising one or more instructions for performing the operations associated with the converter device of the fourth aspect.

[0063] In one embodiment, for example, the disclosed computer program product can comprise: instruction for receiving first data provided by a first network device via a first input- output interface supporting each of a first plurality of communication protocols; instruction for identifying a first communication protocol of the received first data; instruction for converting the received first data into a second communication protocol among a second plurality of communication protocols supported by a second input-output interface; and instruction for transmitting the converted first data to a second network device via the second input-output interface.

[0064] Additional details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying figures and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the descriptions contained herein and the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG. 1 A shows an example of an exemplary wire over which data is carried.

[0066] FIG. IB shows an industry standard cable with QSFP module connectors.

[0067] FIG. 2 shows an exemplary connection of two networked devices that includes eight wires for respective lanes of data traffic.

[0068] FIG. 3 shows an exemplary embodiment in which eight lanes of data traffic are converted into three lanes of data traffic. [0069] FIG. 4 shows an exemplary embodiment involving the usage of a QSFP module.

[0070] FIG. 5 shows an exemplary embodiment involving the usage of a disaggregated storage solution.

[0071] FIG. 6 shows an exemplary network environment in which certain principles disclosed herein are implemented.

[0072] FIGS. 7-9 show additional exemplary network environments in which certain principles disclosed herein are implemented.

[0073] FIG. 10 shows an image of an exemplary row of racks in which certain principles disclosed herein are implemented.

[0074] It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. It also should be noted that the figures are only intended to facilitate the description of the preferred embodiments. The figures do not illustrate every aspect of the described embodiments and do not limit the scope of the present disclosure.

DETAILED DESCRIPTION

[0075] According to various embodiments, multiple communications protocols are enabled for transporting data on the same medium. For illustrative purposes only and not for purposes of limitation, the technologies contemplated and described herein are now described with respect to a network environment in which both PCIe and Ethernet protocols are supported. One or more devices involved in this environment have an input-output (I/O) interface capable of supporting multiple protocols. The number of I/Os can vary. Additionally, the I/Os (or interface) of the device are connected to the I/Os (or interface) of one or more other devices using a variety of protocols. For example, the connection can include four lanes of PCIe data traffic and four lanes of Ethernet data traffic. As discussed herein, there are multiple ways to handle this environment (and other environments involving multiple devices and multiple protocols) in accordance with the principles disclosed herein. [0076] To aid the description of various principles, consider the above-described illustrative configuration, in which four lanes of PCIe traffic and four lanes of Ethernet traffic need to be transported from a server to (1) a local device with four lanes of PCIe and (2) a remotely connected device, connected by Ethernet.

[0077] According to various embodiments, such as shown in FIG. 2, a system 200 can include a first device 201 and a second device 202 coupled by one or more intermediate wires or other communication connections 203. The first device 201, for example, can include a first I/O (or interface) (not shown) for communicating with the communication connections 203; whereas, the second device 202 can include a second I/O (or interface) (not shown) for communicating with the communication connections 203. The first and second devices 201, 202 thereby can exchange data or other information via the communication connections 203.

[0078] FIG. 2 shows that eight I/Os are connected between the first device 201 and the second device 202 through eight respective wires 203, and the protocol on each I/O is determined.

Then, data is transported across and to the end of the eight wires 203 (including, for example, one or more copper or optical wires) and output in their original format. In addition, each device 201, 202 optionally can be equipped with a capability to automatically determine the protocol of each I/O and automatically provision each I/O to the proper protocol. This manner of implementation allows system implementations to mix protocols and have respective data transported in the same environment and not be limited to specific configurations.

[0079] According to various other embodiments, a server (or other device) (not shown) can take inputs and combine the data from different streams into fewer streams, using a proprietary protocol or format. For example, a path carrying PCIe data traffic can be multiplexed (e.g., time multiplexed or combined) with an Ethernet path and transported on a single wire to the destination and de-multiplexed and output to the receiving server or device.

[0080] FIG. 3 shows an illustrative embodiment of a converter device 300. The converter device 300 can be disposed between a first predetermined number of communication connections 301 and a second predetermined number of communication connections 302, wherein the first predetermined number can be greater than, less than or equal to the second predetermined number. Additionally and/or alternatively, the communication connections 301, 302 can support a common communication protocol and/or one or more different communication protocols. As illustrated in FIG. 3, for example, the converter device 300 can be disposed between a first predetermined number of PCIe lanes and a second predetermined number of Ethernet lanes. Stated somewhat differently, the communication connections 301 can include one or more PCIe lanes; whereas, the communication connections 302 can include one or more Ethernet lanes, wherein the number of PCIe lanes is greater than the number of Ethernet lanes. The converter device 300 thereby can combine the multiple PCIe lanes to form fewer Ethernet lanes. The combined data paths could utilize a standard format, such as Ethernet, but may suffer some performance degradation.

[0081] Alternatively, a proprietary protocol, such as a proprietary protocol that removes much of the networking overhead of Ethernet and/or other protocols and/or a proprietary protocol that improves or maximizes performance, may be used in accordance with the principles disclosed herein. According to various embodiments, a proprietary protocol can be chosen that simplifies data stream format to carry reduced, or only essential, information required at the other end of the connection. Because the connection is proprietary and point to point, the protocol can be simplified beyond standard formats.

[0082] According to yet other embodiments, either of the foregoing mechanisms or techniques may be incorporated into an industry standard connector/module, such as an SFP or QSFP module. In this scenario, for example, the module and associated cable function as a standard cable but is also able to automatically detect the traffic on each I/O and handle it appropriately per its associated protocol.

[0083] FIG. 4 shows an illustrative embodiment according to the principles disclosed herein, wherein the converter device 300 (shown in FIG. 3) can be provided as a customized QSFP module 401. Turning to FIG. 4, the QSFP module 401 can be disposed between a first predetermined number of communication connections 301 and a second predetermined number of communication connections 302 in the manner discussed in more detail above with reference to FIG. 3. The QSFP module 401 can include a first QSFP I/O (or electrical interface) 403 for communicating with the communication connections 301 and a second QSFP I/O (or electrical interface) 407 for communicating with the communication connections 302. If the

communication connections 301 comprise one or more transmit lanes and one or more receive lanes, for example, the first QSFP I/O 403 can receive incoming data and other information via the receive lanes of the communication connections 301 and/or transmit outgoing data and other information via the transmit lanes of the communication connections 301. Additionally and/or alternatively, the second QSFP I/O 407 likewise can receive incoming data and other information via receive lanes of the communication connections 302 and/or transmit outgoing data and other information via transmit lanes of the communication connections 302.

[0084] As shown in FIG. 4, the first QSFP I/O 403 of the QSFP module 401 can include at least one multi-protocol capable SerDes (serializer/deserializer). The multi -protocol capable SerDes, for example, can support selected communication protocols, such as PCIe and/or Ethernet. The second QSFP I/O 407 advantageously can support one or more communication protocols, such as fiber optic and/or copper. The second QSFP I/O 407 is shown as being coupled with an optical I/O device, such as an optical module 405, via an optional fiber bundle 406. In selected embodiments, the second QSFP I/O 407 can include one or more fiber- optical connectors, such as a MXC® brand connector or plug manufactured by US Conec Ltd. located in Hickory, North Carolina. Additionally and/or alternatively, the QSFP module 401 can include an optional copper I/O module (not shown) for supporting communications between the second QSFP I/O 407 and a wired copper communication connection (not shown). In other words, the communication protocols supported by the second QSFP I/O 407 can be the same as, or different from, the communication protocols supported by the first QSFP I/O 403.

[0085] The QSFP module 401 can further include a protocol management device 404. The protocol management device 404 can be disposed between, and communicate with, the first QSFP I/O 403 and the second QSFP I/O 407. In the embodiment shown in FIG. 4, for example, the protocol management device 404 is illustrated as including a fiber management component. The protocol management device 404 can determine the protocol(s) of the incoming data and other information received by the QSFP module 401 and/or the protocol(s) of the outgoing data and other information transmitted by the QSFP module 401.

[0086] These and related embodiments and principles can be used in any of many potential use cases, some examples of which are now provided.

[0087] According to various embodiments, the principles described herein are used for enterprise or cloud, rack scale solutions in which a PCIe fabric (network) is desired to maximize performance (speed, power or cost) and a traditional Ethernet network is desired as well. In this scenario, as different types of equipment are installed in the rack, the principles described herein can be used to provide flexibility in how the individual devices within the rack are

interconnected to achieve one or more overall system performance goals or metrics.

[0088] According to various embodiments, the principles described herein are used in connection with a disaggregated storage solution, such as developed by R-Stor Inc. located in Saratoga, California, where a "Storage Server" node which performs storage management and control functions is located adjacent to the application servers and would maximize performance through a PCIe connection from the application server but also require an Ethernet connection to the broader network (between racks and rows).

[0089] FIG. 5 shows an illustrative embodiment of a network 500 according to the principles disclosed herein involving the usage of a disaggregated storage solution. As shown, the network 500 can include a Top of Rack (ToR) switching system 503 that can enable one or more servers 502 and one or more storage servers 501 to communicate. In other words, the ToR switching system 503 can permit a selected server 502 to communicate with a second server 502, with at least one predetermined storage server 501 and/or with a separate network 510. The selected server 502 is shown as communicating with the ToR switching system 503 via a predetermined number of intermediate communication connections 504. [0090] In some embodiments, the communication connections 504 can be provided in the manner discussed above with regard to the communication connections 301, 302 (shown in FIGS. 3 and 4) and can support at least one selected communication protocol. The

communication connections 504, for example, preferably can comprise multi -protocol connector lanes in the manner discussed above with reference to the communication connections 203 (shown in FIG. 2). The communication connections 504 preferably include an optical communication connection or other high-speed communication connection. Two or more servers 502 can communicate with the ToR switching system 503 via common (or shared) communication connections 504, and/or individual (or separate) communication connections 504 can be provided between the ToR switching system 503 and the selected server 502.

[0091] Although the server 502 can communicate with the communication connections 504 directly, the network 500 advantageously can include a first multi-protocol converter

device 300A for facilitating communications between the server 502 and the communication connections 504. The first converter device 300A, for example, can be provided in the manner set forth above with reference the multi -protocol converter device 300 (shown in FIGS. 3 and 4) and/or can be separate from, or at least partially integrated with, the server 502. As described in more detail above, the first converter device 300 A can receive data and other information transmitted by the server 502 in a first communication protocol and convert the received data and other information into the selected communication protocol of the communication

connections 504. The first converter device 300A likewise can receive data and other information in the selected communication protocol from the communication connections 504 and convert the received data and other information into the first communication protocol of the server 502.

[0092] In the embodiment shown in FIG. 5, the server 502 can communicate with the communication connections 504 via an optional network interface controller (or NIC or

Super NIC) 506. The network interface controller 506 can be provided in any conventional manner and can help to further facilitate communications between the server 502 and the communication connections 504. If disposed between the server 502 and the first converter device 300A, for example, the network interface controller 506 provide an additional layer of communication protocol conversion.

[0093] The network interface controller 506 can receive data and other information transmitted by the server 502 in the first communication protocol and convert the received data and other information into one of the communication protocols supported by the first converter

device 300A. Similarly, the network interface controller 506 can receive data and other information in the supported communication protocol from the first converter device 300 A and convert the received data and other information into the first communication protocol of the server 502. Although shown and described as comprising a separate device in FIG. 5 for purposes of illustration only, the network interface controller 506 can be at least partially integrated with the server 502 and/or the first converter device 300A as desired.

[0094] Additionally and/or alternatively, the network 500 can include a second multi -protocol converter device 300B for facilitating communications between the communication

connections 504 and the ToR switching system 503. Although the ToR switching system 503 can communicate with the communication connections 504 directly in some embodiments, the second converter device 300B advantageously can facilitate communications between the ToR switching system 503 and the communication connections 504. The second converter device 300B, for example, can be provided in the manner set forth above with reference the multi -protocol converter device 300 (shown in FIGS. 3 and 4) and/or can be separate from, or at least partially integrated with, the ToR switching system 503.

[0095] As described in more detail above, the second converter device 300B can receive data and other information transmitted by the ToR switching system 503 in a second communication protocol and convert the received data and other information into the selected communication protocol of the communication connections 504. The second converter device 300B likewise can receive data and other information in the selected communication protocol from the communication connections 504 and convert the received data and other information into the second communication protocol of the ToR switching system 503.

[0096] In some embodiments, the ToR switching system 503 can permit a selected storage server 501 to communicate with a second storage server 501, with at least one predetermined server 502 and/or with the separate network 510. The selected storage server 501 is shown as communicating with the ToR switching system 503 via a predetermined number of intermediate communication connections 505. In some embodiments, the communication connections 505 can be provided in the manner discussed above with regard to the communication

connections 301, 302 (shown in FIGS. 3 and 4) and can support at least one second selected communication protocol.

[0097] The communication connections 505, for example, preferably can comprise multiprotocol connector lanes in the manner discussed above with reference to the communication connections 203 (shown in FIG. 2). The communication connections 505 can be the same as, or different from, the second selected communication protocol supported by the communication connections 504 and preferably include an optical communication connection or other highspeed communication connection. Two or more storage servers 501 can communicate with the ToR switching system 503 via common (or shared) communication connections 505, and/or individual (or separate) communication connections 505 can be provided between the ToR switching system 503 and the selected storage server 501.

[0098] Although the storage server 501 can communicate with the communication

connections 505 directly, the network 500 advantageously can include a third multi -protocol converter device 300 for facilitating communications between the storage server 501 and the communication connections 505. The third converter device 300C, for example, can be provided in the manner set forth above with reference the multi -protocol converter device 300 (shown in FIGS. 3 and 4) and/or can be separate from, or at least partially integrated with, the storage server 501. [0099] As described in more detail above, the third converter device 300C can receive data and other information transmitted by the storage server 501 in a third communication protocol and convert the received data and other information into the second selected communication protocol of the communication connections 505. The third converter device 300C likewise can receive data and other information in the second selected communication protocol from the

communication connections 505 and convert the received data and other information into the third communication protocol of the storage server 501.

[0100] In the embodiment shown in FIG. 5, the storage server 501 can communicate with the communication connections 505 via an optional network interface controller (or NIC or

Super NIC) 507. The network interface controller 507 can be provided in any conventional manner and can help to further facilitate communications between the storage server 501 and the communication connections 505. If disposed between the storage server 501 and the third converter device 300C, for example, the network interface controller 507 provide an additional layer of communication protocol conversion.

[0101] The network interface controller 507 can receive data and other information transmitted by the storage server 501 in the third communication protocol and convert the received data and other information into one of the communication protocols supported by the third converter device 300C. Similarly, the network interface controller 507 can receive data and other information in the supported communication protocol from the third converter device 300C and convert the received data and other information into the third communication protocol of the storage server 501. Although shown and described as comprising a separate device in FIG. 5 for purposes of illustration only, the network interface controller 507 can be at least partially integrated with the storage server 501 and/or the third converter device 300C as desired.

[0102] Additionally and/or alternatively, the network 500 can include a fourth multi-protocol converter device 300 for facilitating communications between the communication

connections 505 and the ToR switching system 503. Although the ToR switching system 503 can communicate with the communication connections 505 directly in some embodiments, the fourth converter device 300D advantageously can facilitate communications between the ToR switching system 503 and the communication connections 505. The fourth converter device 300D, for example, can be provided in the manner set forth above with reference the multi -protocol converter device 300 (shown in FIGS. 3 and 4) and/or can be separate from, or at least partially integrated with, the ToR switching system 503.

[0103] As described in more detail above, the fourth converter device 300D can receive data and other information transmitted by the ToR switching system 503 in a fourth communication protocol and convert the received data and other information into the second selected

communication protocol of the communication connections 505. The fourth converter device 300D likewise can receive data and other information in the second selected

communication protocol from the communication connections 505 and convert the received data and other information into the fourth communication protocol of the ToR switching system 503.

[0104] The ToR switching system 503 can support any predetermined number of different switching protocols (or functionalities). In the embodiment shown in FIG. 5, for example, the ToR switching system 503 is illustrated as including PCIe and Ethernet switching functions. The ToR switching system 503 can include a PCIe switch module (or device) 503B that can be coupled with the second multi -protocol converter device 300B and the fourth multi -protocol converter device 300D. The second multi-protocol converter device 300B can convert the data and other information received from the selected server 502 via the communication

connections 504 into a PCIe protocol and provide the converted data and other information in the PCIe protocol to the PCIe switch device 503B. The PCIe switch device 503B likewise can receive data and other information from the selected storage server 501, the network 510 and/or other data sources and convert the received data and other information into the selected communication protocol of the communication connections 504 for transmission to the selected server 502.

[0105] Similarly, the fourth multi -protocol converter device 300D can convert the data and other information received from the selected storage server 501 via the communication connections 505 into the PCIe protocol and provide the converted data and other information in the PCIe protocol to the PCIe switch device 503B. The PCIe switch device 503B likewise can receive data and other information from the selected server 502, the network 510 and/or the other data sources and convert the received data and other information into the second selected communication protocol of the communication connections 505 for transmission to the selected storage server 501. Although shown and described as including the PCIe switch module (or device) 503B for performing PCIe protocol conversions for purposes of illustration only, the ToR switching system 503 can include any predetermined number of switch modules (or devices) each supporting a respective communication protocol. The communication protocols can be the same and/or different among the switch devices.

[0106] The ToR switching system 503 also is shown as including a ToR switch module (or device) 503 A. The ToR switch device 503 A can be coupled with the second multi -protocol converter device 300B and the fourth multi -protocol converter device 300D. The second multiprotocol converter device 300B can convert the data and other information received from the selected server 502 via the communication connections 504 into an Ethernet protocol, such as and provide the converted data and other information in the Ethernet protocol to the ToR switch device 503 A. The ToR switch device 503 A can transmit the converted data and other information to the PCIe switch device 503B, the network 510, the fourth multi -protocol converter device 300D for conversion and transmission to the selected storage server 501 via the communication connections 505 and/or other data destinations. Additionally and/or

alternatively, the ToR switch device 503 A likewise can receive data and other information from the selected storage server 501, the PCIe switch device 503B, the network 510 and/or the other data sources and convert the received data and other information into the selected

communication protocol of the communication connections 504 for transmission to the selected server 502.

[0107] Similarly, the fourth multi -protocol converter device 300D can convert the data and other information received from the selected storage server 501 via the communication connections 505 into the Ethernet protocol and provide the converted data and other information in the Ethernet protocol to the ToR switch device 503 A. The converted data and other information can comprise data and other information received via remote direct memory access (RDMA) communications. The ToR switch device 503 A can transmit the converted data and other information to the PCIe switch device 503B, the network 510, the second multiprotocol converter device 300B for conversion and transmission to the selected server 502 via the communication connections 504 and/or the other data destinations. The ToR switch device 503 A likewise can receive data and other information from the selected server 502, the PCIe switch device 503B, the network 510 and/or the other data sources and convert the received data and other information into the second selected communication protocol of the

communication connections 505 for transmission to the selected storage server 501.

[0108] Advantageously, the first and third converter devices 300A, 300C can selectively choose which protocol is carried by the communication connections 504, 505 during data transmission from the selected server 502 and the selected storage server 501, respectively, to the ToR switching system 503. Additionally and/or alternatively, the second and fourth converter devices 300B, 300D can selectively choose which protocol is carried by the respective communication connections 504, 505 during data transmission from the ToR switching system 503 to the selected server 502 and the selected storage server 501.

[0109] The respective communication connections 504, 505 preferably comprise multiprotocol connector lanes. In one preferred embodiment, the communication connection 504 includes four data lanes, wherein the selected server 502 can utilize two of the four data lanes for exchanging data and other information in the PCIe communication protocol and the other two data lanes for exchanging data and other information in the Ethernet communication protocol. Additionally and/or alternatively, the communication connection 505 includes four data lanes, wherein the selected storage server 501 can utilize two of the four data lanes for exchanging data and other information in the PCIe communication protocol and the other two data lanes for exchanging data and other information in the Ethernet communication protocol. [0110] According to various embodiments, the principles described herein are for suppliers or other individuals or entities preferring to use "standard" interconnects, such as QSFP modules. Many of these modules are currently set up to support Ethernet only, and the principles discussed herein can be applied to allow the use of standard modules and connectors (e.g., QSFP) to transport whatever standard (or mix of standards) is desired.

[0111] FIG. 6 shows an exemplary network environment 600 in accordance with the principles disclosed herein. Turning to FIG. 6, network environment 600 includes Application Servers 611, Storage Servers 612, one or more types of storage arrays 613, 614 (including, e.g., an All Flash Array, Hybrid Array and/or HDD Array), as well one or more other devices 615, such as a GP- GPU. In this example, one or more of the elements are connected through respective multiprotocol modules and a mesh network centered in the ToR (Top of Rack) 616. This example shows both a PCIe network and an Ethernet based Network, although it will be understood that the principles disclosed herein are not limited to such a configuration.

[0112] As shown, for example, Application Servers 611 and Storage Servers 612 are connected both through Ethernet as well as PCIe protocols. In this example, the other devices, e.g. All Flash Array or Hybrid Array, HDD array, and/or GP-GPU, are connected through PCIe only. The PCIe network is local to the rack in this example and the Ethernet network extends beyond the rack to connect with other racks, either on the storage network or the egress network. The multi -protocol modules allow standardization of the connection mechanics and electronics while maintaining flexibility on the protocol which is being transmitted.

[0113] FIGS. 7-9 show additional exemplary network environments 700, 800 and 900 in accordance with the principles disclosed herein. As shown and will be understood to persons of ordinary skill in the art, the principles disclosed herein can be applied to PCIe, Ethernet, and/or mixed network environments.

[0114] FIG. 10 shows an image of an exemplary row of racks 1001 (e.g., server racks) which can be implemented using the principles disclosed herein. [0115] Persons of ordinary skill in the art will understand that, although particular embodiments have been described above and shown in the figures, the principles described herein can be applied to different types of networked solutions, including but not limited to those involving data centers and cloud services.

[0116] As another example, the principles disclosed herein are used to facilitate data communications involving a multiple hard drive storage configuration, in which multiple internal hard drives are located in a single box or enclosure or otherwise grouped, and are accessed through a common controller (e.g., SAS) that is associated with the enclosed or grouped hard drives. According to various embodiments, the controller is disaggregated from any particular hard drive, such that the language can be customized between application and storage

controllers. It will be understood that, although drives are used in this example, the principles discussed apply to other types of devices and configurations.

[0117] Additionally, although specific network environments (number and type of devices, connections, protocols, etc.) have been described above for the purpose of simplifying the description, it will be understood to persons of ordinary skill in the art that this is illustrative only. It will also be understood that reference to a "device" herein can refer to a server or any other type of suitable device. The principles discussed herein can be generalized to any number and configuration of devices and protocols.

[0118] Embodiments of the subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed and shown in this specification and the accompanying figures, and their structural equivalents, or in combinations of one or more of the foregoing. Moreover, the subject matter can be implemented as part of one or more computer program modules or products, including computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, a data processing apparatus. The tangible program carrier can be a computer readable medium suitable for storing computer program instructions, such as a machine-readable storage device, a machine-readable storage substrate, a memory device, other forms of non-volatile memory, or a combination of one or more of them.

[0119] Terms such as "processor" encompasses all apparatus, devices, and machines for processing data, such as a programmable processor, general and special purpose

microprocessors, a computer, or multiple processors or computers. Generally, a processor will receive instructions and data from a read only memory (ROM) or a random access memory (RAM), or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. A computer will also generally include or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (such as magnetic, magneto optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer can be embedded in another device, such as a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, for example.

[0120] The embodiments contemplated herein can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0121] Software (also referred to as programs, applications, script, code, etc.) according to the principles described herein can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program need not correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (such as one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (such as files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0122] To provide for interaction with a software programmer or other use, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, such as a mouse, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Additionally, input from the user can be received in any form, including acoustic, speech, or tactile input.

[00100] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, such as a data server, or that includes a middleware component, such as an application server, or that includes a front end component, such as a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium or communication network, such as a local area network (LAN), or wide area network (WAN) such as the Internet, or both.

[0123] While this specification contains particular implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a subcombination.

[0124] Similarly, while certain details are depicted in the figures, this should not be understood as requiring that any or all such details be included (or any or all identified features or steps be performed in a particular order), to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0125] Particular embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the principles discussed herein. For example, the embodiments shown in the figures can be modified or combined and still achieve desirable results.