Inventors: Cameron Turner and Giffin Symko

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of and priority from U.S. Application No. 17/582,987, filed January 24, 2022 which is hereby incorporated by reference in its entirety as if fully set forth herein, for all purposes. Other pertinent disclosures include, by way of example, U.S. Patent No. 9,052,252, issued June 9, 2015, U.S. Patent No. 10,364,809, issued July 30, 2019, and 10,365,667, issued July 30, 2019. Each of the foregoing references is hereby incorporated by reference in its entirety as if fully set forth herein, for all purposes.

FIELD

[0002] This application and the subject matter disclosed herein (collectively referred to as the “disclosure”), generally concern liquid-based heat-transfer systems. More particularly, but not exclusively, this disclosure pertains to systems, methods, and components for cooling electronics.

BACKGROUND INFORMATION

[0003] Component and overall heat dissipation, together with computing performance, increases with each successive generation of server (including each successive generation of processing component, power-delivery component, chipset component, memory controller component, memory component, and other components) within those servers. Consequently, liquid-cooling technologies have become desirable within data centers and other computing installations for their ability to efficiently remove heat dissipated by processing units and other server components.

SUMMARY

[0004] Notwithstanding that component and overall heat dissipation may increase for each successive generation of server or component, not all servers and components operate at full capacity throughout their useful life. Rather, heat dissipation by a given server (and by each component within the server) typically varies according to the workload the server (or the component) is called to perform (sometimes referred to generally in the art as an “IT workload”). In some respects, disclosed principles pertain to tailoring a liquid-cooling system to match an expected IT workload. In other respects, disclosed principles pertain to adjusting operation of liquid-cooling system in real-time to match an observed or inferred IT workload. By tailoring or adjusting operation of liquid-cooling systems to correspond to observed, expected or inferred IT workloads, relatively high over-all system efficiency can be achieved, as unnecessary excess cooling can be avoided.

[0005] A heat-transfer system includes a cooling circuit configured to convey heated coolant from one or more cooling nodes to one or more heat-rejection devices, and to convey the cooled coolant from the one or more heat-rejection devices to the one or more cooling nodes. Each cooling node facilitates a transfer of heat to the coolant, the heat being from one or more heat-dissipation devices and a corresponding heat load on the respective cooling node. Each heat-rejection device facilitates heat transfer from the coolant to another medium. The heat-transfer system also has a selectively operable flow-control device configured to control a flow rate of the coolant through a segment of the coolant circuit. A control logic selectively operates the flow-control device responsive to an output from one or more sensors to tailor a cooling capacity available to each cooling node to the real-time heat load on the respective cooling node.

[0006] The coolant in the cooling circuit can be a facility coolant. At least one of the one or more cooling nodes can include a coolant heat-exchange unit configured to transfer heat from a secondary coolant to the facility coolant, cooling the secondary coolant. The heat-transfer system can also include a secondary cooling circuit having a plurality of server-cooling nodes. A secondary distribution manifold can be configured to distribute the secondary coolant among the plurality of server-cooling nodes. A secondary collection manifold can be configured to collect the secondary coolant from the plurality of server-cooling nodes. The coolant heat-exchange unit can fluidicly couple with the secondary collection manifold to receive heated secondary coolant from the plurality of server-cooling nodes. The coolant heat-exchange unit can fluidicly couple with the secondary distribution manifold to distribute cooled secondary coolant among the plurality of server-cooling nodes.

[0007] The coolant heat-exchange unit can be a first coolant heat-exchange unit and the secondary cooling circuit can be a first secondary cooling circuit. At least one of the one or more cooling nodes can include a second coolant heat-exchange unit and the heat-transfer system can further include a second secondary cooling circuit cooled by the facility coolant.

[0008] The one or more cooling nodes can include one or more component-cooling nodes corresponding to each of a plurality of servers. Each component-cooling node can be configured to transfer heat from a processing unit to the coolant. The one or more heat-rejection devices can include a coolant heat-exchange unit configured to transfer heat from the coolant to a facility coolant.

[0009] The selectively operable flow-control device can include an adjustable valve. The selectively operable flow-control device can include a controllable pump.

[0010] The one or more heat-rejection devices can include a waste -heat recovery device configured to transfer heat from the coolant to another working fluid.

[0011] The heat-transfer system can also include a gateway having a first communication connection with the control logic, the plurality of sensors, or both. The gateway can also include a second communication connection with a Data Center Information Manager, a Building Management System, or both. In some embodiments, the gateway is configured to receive first information over the first communication connection and to communicate the received first information over the second communication connection, to receive second information over the second communication connection and to communicate the received second information over the first communication connection, or both.

[0012] In some embodiments, the first received information can correspond to a measure of cooling margin available from the heat-transfer system. In some embodiments, the second received information corresponds to an IT workload scheduled for one or more servers. For example, the heat load on the respective cooling node can correspond to the IT workload.

[0013] According to other aspects, a heat-transfer system includes a plurality of cooling nodes. Each cooling node is configured to directly or indirectly transfer heat to a coolant. The transferred heat is generated by one or more electronic components corresponding to each cooling node. A flow controller is configured to control a flow rate of coolant to a selected one or more of the plurality of cooling nodes in correspondence with a real-time determination of the heat generated by the one or more electronic components that correspond to the selected one or more of the plurality of cooling nodes.

[0014] In some embodiments, the heat-transfer system also includes a communication gateway configured to receive information from the flow controller and to communicate the information to a Building Management System, a Data Center Information Manager, or both. For example, in some embodiments, the heat-transfer system includes a plurality of sensors. Each sensor in the first plurality of sensors can be configured to observe a corresponding one or more parameters of a selected one of the plurality of cooling nodes. The communication gateway can be configured to determine a state of-operational-health of the selected cooling node from the one or more parameters observed by the plurality of sensors. The communication gateway can be further configured to communicate an indication of the state of-operational-health to the Building Management System, the Data Center Information Manager, or both.

[0015] In some embodiments, the communication gateway is configured to receive information from the Building Management System, the Data Center Information Manager, or both. In some embodiments, the communication gateway can also be configured to communicate the information to the flow controller. For example, the flow controller can be configured to control the flow rate of coolant to the selected one or more of the plurality of cooling nodes responsive to information received from the communication gateway.

[0016] In some embodiments, the flow controller is configured to isolate one of the one or more of the plurality of cooling nodes responsive to information received from the communication gateway. For example, information received from the communication gateway can correspond to information received by the gateway from the Building Management System, the Data Center Information Manager, or both. Such received information can indicate a leak or other system operating fault.

[0017] In some embodiments, a real-time determination of a generated heat load to be cooled corresponds to information received from the Building Management System, the Data Center Information Manager, or both.

[0018] In some embodiments, the flow controller includes a control logic and a flow-control device. For example, the flow-control device comprises a valve, a pump, or both.

[0019] Also disclosed are associated methods, as well as tangible, non-transitory computer-readable media including computer executable instructions that, when executed, cause a computing environment to implement one or more methods disclosed herein. Digital signal processors embodied in software, firmware, or hardware and being suitable for implementing such instructions also are disclosed. [0020] The foregoing and other features and advantages will become more apparent from the following detailed description, which proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Referring to the drawings, wherein like numerals refer to like parts throughout the several views and this specification, aspects of presently disclosed principles are illustrated by way of example, and not by way of limitation.

[0022] FIG. 1 illustrates a modular heat-transfer system configured to cool a plurality of independently operable, rack-mounted servers.

[0023] FIG. 2 shows an isometric view of a branch of the cooling system within a server shown in FIG. 1 , together with two cooling nodes for cooling two corresponding electronic components in the server.

[0024] FIG. 3 schematically illustrates a portion of another embodiment of a modular heat-transfer system.

[0025] FIG. 4 schematically illustrates a portion of yet another embodiment of a modular heattransfer system.

[0026] FIG. 5 schematically illustrates a further embodiment of a modular heat-transfer system incorporating a waste-heat recovery branch. The heat-transfer system in FIG. 5 can be analogous to the system shown in any of FIGs. 1, 3 and 4 with an added waste -heat recovery branch.

[0027] FIG. 6 shows a plot of pressure head (or loss) as a function of flow rate for a pump and for a given coolant loop under different operating conditions.

[0028] FIG. 7 shows a family of pump curves for a pump operating at several different speeds.

[0029] FIG. 8 shows a block diagram of a computing environment suitable for implementing disclosed methods.

[0030] FIGs. 9 and 10 show operating conditions for a working example under different conditions. DETAILED DESCRIPTION

[0031] The following describes various principles related to smart components, systems and methods for transferring heat with liquid. For example, certain aspects of disclosed principles pertain to tailoring a cooling system’s operation to an observed heat load or distribution of heat loads throughout the cooling system. That said, descriptions herein of specific apparatus configurations and combinations of method acts are but particular examples of contemplated components, systems, and methods chosen as being convenient illustrative examples of disclosed principles. One or more of the disclosed principles can be incorporated in various other embodiments to achieve any of a variety of corresponding characteristics.

[0032] Thus, embodiments having attributes that are different from those specific examples discussed herein can incorporate one or more presently disclosed principles, and can be used in applications not described herein in detail. Accordingly, such alternative embodiments also fall within the scope of this disclosure.

I. OVERVIEW

[0033] Component and overall heat dissipation during computationally intensive workloads may be much higher than component and overall heat dissipation during periods of low computation activity, as when a server is primarily transferring data rather than rendering graphics or performing other computationally intensive work. Such “I/O intensive” workloads can arise, for example, when a server serves data over a network connection for a streaming application.

[0034] And, although a typical server rack of the type used in a data center can accommodate 42 individual servers, some server racks can accommodate more or fewer individual servers. Further, some server racks might not be fully populated regardless of their capacity.

[0035] Thus, a cooling system designed to provide a static rate of cooling to each server (or component or rack) assumed to be present in a given installation, e.g., based on an anticipated upper threshold level of heat dissipation by each component in each server in each rack, may adequately cool the various components when all are operating under a heavy or maximum computational workload. However, that cooling system may provide unnecessary excess cooling to the various components under other workloads, e.g., that cause one or more servers (or components) to dissipate less heat. Stated differently, a cooling system designed for an upper threshold power dissipation by all servers in a given rack (or all servers in group of racks in a datacenter) may be over designed when the upper-threshold power dissipation only occasionally occurs on a per-server basis (and perhaps rarely if at all across all servers associated with the cooling system). That is to say, the cooling system may have excess cooling capacity, or margin, under “typical” datacenter workloads.

[0036] While a selected degree of margin or excess cooling capacity may be desirable, liquid cooling systems consume power to operate, e.g., to pump a coolant through the various cooling system components. Accordingly, providing “too much” excess cooling to a server (or a group of servers) can result in less-than-optimal overall efficiency for the aggregate installation. Accordingly, disclosed principles provide a means for tailoring the cooling capacity provided by a given cooling system to an actual amount of heat being dissipated by each component, server, and/or rack in a datacenter. Such tailoring of cooling capacity can improve overall efficiency for the server installation compared to a server installation that relies on a cooling system that provides “too much” excess cooling.

[0037] According to one aspect, tailoring the cooling capacity of a cooling system to a server installation can involve reducing the rate of cooling provided by a cooling system to match a server’s (or a component’s or a rack’s) lower rate of heat dissipation during times of, for example, IO intensive operation. According to another aspect, tailoring the cooling capacity can involve matching the number of servers cooled by a given cooling system to that cooling system’s overall cooling capacity. According to yet another aspect, tailoring the cooling capacity can involve adjusting one or more coolant temperatures, coolant flow rates, and/or other cooling-system parameters to adjust the rate of cooling provided by the cooling system to a selected one or more heat-dissipating components. Aspects of this disclosure pertain to systems, methods, and components for tailoring a liquid-cooling system’s operation to an actual or estimated heat load that the liquid-cooling systems is called on to cool.

[0038] As but one illustrative example, a disclosed liquid-cooling system can tailor operation of one or more cooling nodes to the workload of a corresponding group of electronic components, group of servers incorporating or associated with such components, and/or group of racks incorporating or associated with such servers. For example, a liquid-cooling system can control a flow rate of coolant throughout the cooling system, through one or more selected branches of a coolant circuit, and/or through one or more selected cooling nodes of the cooling system. In some embodiments, a controller can adjust operation of one or more pumps, one or more valves, or a combination of one or more pumps and one or more valves to tailor the cooling capacity of a selected cooling nodes to correspond to the heat dissipated by the components, servers, and/or racks cooled by the cooling nodes.

[0039] Other, related principles also are disclosed. For example, the following describes machine- readable media containing instructions that, when executed, cause a processor of, e.g., a controller or other computing environment, to perform one or more disclosed methods. Such instructions can be embedded in software, firmware, or hardware. In addition, disclosed methods and techniques can be carried out in a variety of forms of processor or controller, as in software, firmware, or hardware.

II. MODULAR HEAT-TRANSFER SYSTEMS

[0040] By way of example, FIG. 1 shows an array 50 of independently operable servers 12a, 12b ... 12n mounted in a rack, or chassis, together with aspects of a modular heat-transfer system for cooling the servers. In FIG. 1, each server 12a-n contains one or more corresponding electronic components that dissipate heat while operating. A heat-transfer (e.g., cooling) system can use a liquid coolant to collect heat from each heat source and carry the heat to a suitable heat sink, e.g., a radiator that rejects the heat from the coolant to a facility liquid, atmospheric air, and/or air in a conditioned room containing the rack of servers. Such an arrangement for cooling rack mounted servers is described in further detail in U.S. Patent No. 9,496,200.

[0041] FIG. 1 also shows a rack-level coolant-distribution unit 10. A coolant-distribution unit is sometimes referred to as a “coolant heat-exchange unit” when it incorporates a heat-exchanger to reject heat from the coolant passing through the coolant-distribution unit. In the embodiment depicted in in FIG. 1, the coolant-distribution unit 10 can have at least one pump and can also incorporate a reservoir and other components, regardless of whether the coolant-distribution unit incorporates a heat exchanger.

[0042] In the embodiment shown in FIG. 1, the coolant-distribution unit 10 incorporates a heat exchanger. When a heat exchanger is included within the confines of the coolant-distribution unit 10, as in FIG. 1, the coolant-distribution unit 10 can receive a warm coolant carrying heat from the servers 12a-n and facilitate transfer of that heat from the warm coolant to another medium. The coolant-distribution unit 10 can return the cooled coolant to the servers to collect further heat from the servers. [0043] In other embodiments, a coolant-distribution unit receives cooled coolant from a heat exchanger outside the confines of the coolant-distribution unit or delivers warm coolant to a n external (e.g., outside the confines of the coolant-distribution unit) heat exchanger. The external heat exchanger cools the coolant before being distributed among and through the servers 12a-n.

[0044] A modular cooling system as shown in FIGs. 1 and 2 can provide at least one cooling node for each server. As used herein, the term “node” means an identifiable component (or an identifiable group of components) within a system and the term “cooling node” means an identifiable component (or an identifiable group of components) that absorb(s) heat from an external source (e.g., that cools the external heat source).

[0045] For example, in context of a modular heat-transfer system for cooling one rack of 42 individual servers, the cooling system can have a cooling node for each server. Stated differently, the cooling system can have 42 server-cooling nodes, with each server-cooling node corresponding to one of the 42 servers in the rack. For example, the portion of the modular cooling system shown in FIGs. 1 and 2 provides a server-cooling node 11 for each server 12a-n.

[0046] Similarly, in context of a system for cooling a plurality of racks of servers (as described more fully below), a modular cooling system can provide a rack-cooling node for each rack of servers. In FIG. 1, for example, a rack-cooling node encompasses all 42 of the server-cooling nodes 11.

[0047] In similar fashion, a given server-cooling node (or more than one of them, or all of them) can incorporate one or more component-cooling nodes. For example, if a given server has two electronic components (e.g., two processors) to be cooled by that server’s server-cooling node, that server’s server-cooling node can provide one component-cooling node for each electronic component to be cooled. As FIG. 2 shows, the server cooling node 11 provides a first componentcooling node 20a and a second component-cooling node 20b. The first component-cooling node 20a is thermally coupled with a first processor to transfer heat from the first processor to a liquid coolant passing through the first component-cooling node 20a. Similarly, the second component-cooling node 20b is thermally coupled with a second processor to transfer heat from the second processor to a liquid coolant passing through the second component-cooling node 20b. Representative component-cooling nodes are described in further detail in U.S. Patent Nos. 8,746,330 and 9,453,691. The component-cooling nodes can be passive, as in the ’330 Patent, or they can be active, e.g., include a pump, as in the ’691 Patent. [0048] FIG. 2 also depicts a portion of a coolant loop, or coolant circuit, that conveys coolant to and from the server-cooling node 11, as well as to and from each component-cooling node 20a, 20b. For example, the coolant-distribution unit 10 conveys cool coolant to a distribution manifold and receives warmed coolant from a collection manifold. The coolant loop shown in FIGs. 1 and 2 provides a branch of a fluid circuit for each server 12a-n. Each fluid-circuit branch receives cool coolant from the distribution manifold and conveys the cool coolant to the server-cooling node 11 where the coolant absorbs heat. Further, each fluid-circuit branch conveys warm coolant exiting from the server-cooling node 11 to the collection manifold, which returns the warmed coolant to the coolant-distribution unit 10. In the system shown in FIGs. 1 and 2, the fluid-circuit branch for each server-cooling node is fluidically coupled in parallel with the fluid-circuit branches for each of the other server-cooling nodes.

[0049] But, within the fluid-circuit branch shown in FIG. 2, the component-cooling nodes 20a, 20b are fluidically coupled with each other in series. For example, in FIG. 2, the component cooling node 20a receives cool coolant arriving from the coolant distribution manifold and heats the coolant with heat dissipated by the first processor. After exiting the first component-cooling node 20a, coolant heated by the first processor enters the second component-cooling node 20b, where the coolant is further heated by the second processor before returning to the coolant-distribution unit 10 by way of the collection manifold. Although not shown in FIG. 2, the component-cooling nodes 20a, 20b can be fluidically coupled with each other in parallel, which each component-cooling node receiving coolant from a corresponding further branch of the coolant circuit.

[0050] The cooling capacity of a given cooling node depends on many parameters. But, in a general sense, the available cooling capacity corresponds to a temperature of coolant entering the cooling node, a permissible increase in coolant temperature as it passes through the cooling node, and a flow rate of coolant passing through the cooling node. With all else being equal, a cooling node with a higher mass-flow rate of coolant passing through has a higher cooling capacity than it does with a lower mass-flow rate of coolant passing through. Accordingly, a cooling node that adequately cools a heat source (e.g., an electronic component, a server, or a rack of servers) that dissipates an upper threshold rate of heat will provide excess cooling to the heat source if the rate of heat dissipation by the source falls and the mass-flow rate of coolant through the cooling node remains unchanged.

[0051] Stated differently, as the rate of heat dissipated by a heat source falls, a mass-flow rate of coolant through the corresponding cooling node can be reduced. As a consequence of reducing the flow rate through the cooling node, the pressure (or head) loss due to friction through the portion of the coolant loop corresponding to the cooling node also is reduced. With such a reduction in head loss, the source of the pressure head (e.g., the pump) that drives the coolant through that cooling node may reduce its work on the coolant and thus may operate at a reduced power. For example, if a mass-flow rate of coolant through a portion of a coolant circuit can be reduced, less pressure head may be needed to urge coolant throughout the coolant loop and so it may be possible to reduce a pump speed, in turn reducing the amount of energy consumed by the cooling system. For a given rate of heat dissipated by a heat source, a reduction in mass-flow rate of coolant through the cooling node will lead to a correspondingly higher coolant-return temperature. For example, for a given rate of heat absorbed by a coolant that is assumed to be incompressible (e.g., water, a water-glycol mixture), the increase in coolant temperature across the region of heat transfer is linearly proportional to the flow rate of coolant through the region. In addition to saving energy expended by moving coolant through a cooling node, a relatively higher coolant-return temperature improves the quality of waste heat, increasing the ability to recover waste heat for useful purposes. By way of example, recovered waste heat can be used to heat domestic hot water, to heat a working fluid (e.g., water) in a hydronic heating system, and/or to perform useful work (e.g., in a power-generation or power-conversion device).

[0052] According to a disclosed aspect, a control system can receive real-time information that pertains to the rate of heat dissipated by a heat source, as well as real-time information that pertains to the cooling capacity provided by the cooling node that corresponds to the heat source. Responsive to the received real-time information, the control system can adjust, for example, the mass-flow rate of coolant through the cooling node to match the cooling capacity of the cooling node to the rate of heat dissipated by the heat source. Similarly, the control system can adjust the mass-flow rate of coolant through the cooling node to improve the quality of waste heat for useful purposes.

[0053] In an embodiment, the control system can reduce a pump speed or partially close a valve, or both, to reduce a flow rate of coolant through a given cooling node (as when the rate of heat dissipation by the heat source falls). Similarly, the control system can increase a pump speed or partially (or wholly) open a valve, or both, to increase a flow rate of coolant through the cooling node, as when the rate of heat dissipation by the heat source increases. III. A “SMART” MODULAR HEAT-TRANSFER SYSTEM

[0054] FIG. 3 schematically illustrates a portion of another embodiment of a modular heat-transfer system 100. The depiction in FIG. 3 shows four server-cooling nodes 110a, 110b, 110c, HOd, within a rack-level cooling loop, despite that the system 200 can have more or fewer server-cooling nodes within the rack-level cooling loop. For example, as with the modular heat-transfer system shown in FIG. 1, the heat-transfer system 100 can have 42 server-cooling nodes, with each servercooling node corresponding to one rack-mounted server. Each server-cooling node HOa-d can have one or more component cooling nodes (e.g., analogous to the component-level cooling nodes 20a, 20b in FIG. 1).

[0055] The heat-transfer system 100 has a rack-mounted coolant-distribution unit 120. As depicted, the coolant-distribution unit 120 receives heated coolant from a collection manifold 122 and delivers cool coolant to a distribution manifold 124. The coolant-distribution unit 120 has within a liquid-to- liquid heat exchanger 125 that rejects heat, Q, from the coolant received from the collection manifold 122 to cool facility coolant received by the facility supply inlet 121. As the facility coolant passes through the heat exchanger 125, it absorbs the heat, Q, and increases in temperature, eventually exiting the heat exchanger through the facility return outlet 123. After rejecting the heat, Q, the now cooled coolant enters a central circulation pump 126. An outlet from the pump is fluidically coupled with an inlet to the distribution manifold 124, allowing the cooled coolant to return to the several server-cooling nodes HOa-d.

[0056] Each server-cooling node HOa-d receives cool coolant from the distribution manifold 124 and returns heated coolant to the collection manifold 122. For example, each server-cooling node HOa-d has a supply connection 114a-d with the distribution manifold 124 and a return connection 115a-d with the collection manifold 122. In the illustrated embodiment of the heat-transfer system 100, a variable-position, controllable valve 112a-d is positioned intermediate the supply connection 114a-d with the distribution manifold and inlet 11 la-d to the cooling node 1 lOa-d. Stated differently, the branch of the coolant loop that conveys coolant to and from each server-cooling node HOa-d has a flow-control valve for adjusting a mass-flow rate of coolant that passes through each server-cooling node HOa-d.

[0057] The heat-transfer system 100 also has a controller 127, together with one or more communication connections (e.g., a signal bus) 128 that communicatively couples the controller 127 with one or more sensors as well as one or more flow-control devices. For example, based on information received from one or more sensors, the controller can output a control signal to adjust operation of one or more flow-control devices. As an example of such adjustments, an output signal from the controller can cause a valve to change or to maintain its opening within a range from 0% open (e.g., closed) to 100% open (e.g., unobstructed). As another example, the control output signal can cause a pump to speed up, slow down, start, or stop operation. For example, a coolantdistribution unit may have one or more pumps hydraulically coupled with each other in parallel, in series, or a combination of parallel and series to provide suited to maintain stable operation over a wide range of pressure-drop and flow-rate conditions. With such a coolant-distribution unit, the controller can adjust operation of one or more of the pumps to deliver a target pressure head and flow rate to the coolant loop of the cooling system 100.

[0058] Although the coolant-distribution unit 120 of the heat-transfer system 100 is depicted as incorporating a liquid-to-liquid heat exchanger 125, other embodiments of coolant-distribution units lack an internal heat exchanger, as discussed above in Section II. Further, the heat-transfer system 100 is depicted as having a central pump 126, but other embodiments of “smart” modular heattransfer systems have no central pump and instead incorporate distributed pumps as described for example in U.S. Patent No. 9,496,200. In such embodiments, the controller enjoys additional degrees of freedom to tailor cooling capacity through each cooling node. That is to say, the controller can adjust a speed or operating point of one or more pumps, e.g., a central pump and/or one or more distributed pumps (e.g., as a group or independently) to tailor the degree of cooling provided by each cooling node in the heat-transfer system.

IV. ANOTHER “SMART” MODULAR HEAT-TRANSFER SYSTEM

[0059] FIG. 4 schematically illustrates a portion of yet another embodiment of a modular heattransfer system 200. The depiction in FIG. 4 shows four rack-cooling nodes 210a, 210b, 210c, 210d, within a datacenter-level cooling loop, despite that the system 200 can have more or fewer rack-cooling nodes within the datacenter-level cooling loop.

[0060] In FIG. 4, each server rack corresponding to a rack-level cooling node 210a, b, c, d contains an array of rack-mounted servers similar to the array of rack-mounted servers shown in FIGS. 1 and 3. However, unlike the heat-transfer system 100 shown in FIG. 3, which provides on-rack cooling to one array of rack-mounted servers, the heat-transfer system 200 shown in FIG. 4 provides cooling to several arrays of rack-mounted servers.

[0061] For example, as with the modular heat-transfer system shown in FIG. 1 , each rack-level cooling node 210a-d can have 42 server-cooling nodes (e.g., analogous to server-cooling node 11), with each server-cooling node corresponding to one rack-mounted server. Further, each servercooling node within each rack-level cooling node 210a-d can have one or more component cooling nodes (e.g., analogous to the component-level cooling nodes 20a, 20b in FIG. 1).

[0062] The heat-transfer system 200 includes an off-rack (e.g., stand-alone) coolant-distribution unit 220. As depicted, the coolant-distribution unit 220 receives heated coolant from a datacenter-level collection manifold 222 and delivers cool coolant to a datacenter-level distribution manifold 224. The coolant-distribution unit 220 has within a liquid-to-liquid heat exchanger 225 that rejects heat, Q , from the coolant received from the collection manifold 222 to cool facility coolant received by the facility supply inlet 221. As the facility coolant passes through the heat exchanger 225, it absorbs the heat, Q, and increases in temperature, eventually exiting the heat exchanger through the facility return outlet 223. After rejecting the heat, Q, the now cooled coolant enters a central circulation pump 226. An outlet from the pump is fluidically coupled with an inlet to the distribution manifold 224, allowing the cooled coolant to return to the several rack-cooling nodes 210a-d.

[0063] Each rack-cooling node 210a-d receives cool coolant from the distribution manifold 224 and returns heated coolant to the collection manifold 222. For example, each rack-cooling node 210a-d has a supply connection 214a-d with the distribution manifold 224 and a return connection 215a-d with the collection manifold 222. In the illustrated embodiment of the heat-transfer system 200, a variable-position, controllable valve 212a-d is positioned intermediate the supply connection 214a-d with the distribution manifold and inlet 21 la-d to the rack-cooling node 210a-d. Stated differently, the branch of the coolant loop that conveys coolant to and from each rack-cooling node 210a-d has a flow-control valve for adjusting a mass-flow rate of coolant that passes through each rack-cooling node 210a-d. In other embodiments, one or more of the controllable valves 212a-d can be positioned intermediate the return-manifold connection 213a-d. For example, all of the controllable valves can be positioned intermediate the return-manifold connection, or in other embodiments, one or more of the controllable valves can be positioned intermediate the supply connection 214a-d and one or more other of the controllable valves can be positioned intermediate the return-manifold connection. [0064] As with the cooling system 100 shown in FIG. 3, one or more server-cooling nodes (not shown) within one or more of the rack-cooling nodes 210a-d can also have a flow-control valve for adjusting a mass-flow rate of coolant that passes through the server-cooling node(s) within the rackcooling node. Alternatively (or additionally), one or more server-cooling nodes among the rackcooling nodes 210a-d can have one or more pumps. Such server-level valves and pumps can allow a cooling-system operator to tailor the cooling capacity delivered to each server-cooling node.

[0065] Referring again to FIG. 4, the heat-transfer system 200 also has a controller 227, together with one or more communication connections (e.g., a signal bus) that communicatively couple the controller 227 with one or more sensors as well as one or more flow-control devices. For example, based on information received from one or more sensors, the controller 227 can output a control signal to adjust operation of one or more flow-control devices. As an example of such adjustments, an output signal from the controller can cause a valve to change or to maintain its opening within a range from 0% open (e.g., closed) to 100% open (e.g., unobstructed). As another example, the control output signal can cause a pump to speed up, slow down, start, or stop operation. For example, a coolant-distribution unit may have one or more pumps hydraulically coupled with each other in parallel, in series, or a combination of parallel and series to provide suited to maintain stable operation over a wide range of pressure-drop and flow-rate conditions. With such a coolantdistribution unit, the controller can adjust operation of one or more of the pumps to deliver a target pressure head and flow rate to the coolant loop of the cooling system 200.

[0066] Although the coolant-distribution unit 220 of the heat-transfer system 200 is depicted as incorporating a liquid-to-liquid heat exchanger 225, other embodiments of coolant-distribution units lack an internal heat exchanger. Further, the heat-transfer system 200 is depicted as having a central pump 226, but other embodiments of “smart” modular heat-transfer systems have no central pump and instead incorporate a plurality of pumps distributed among the rack-cooling nodes 210a-d and/or among the server-cooling nodes (not shown but analogous to the server-cooling nodes HOa-d) among the rack-cooling nodes 210a-d. In such embodiments, the controller enjoys additional degrees of freedom to tailor cooling capacity through each rack- and/or server-cooling node. That is to say, the controller can adjust a speed or operating point of one or more distributed pumps (e.g., as a group or independently) to tailor the degree of cooling provided by each cooling node in the heattransfer system. V. SENSORS AND CONTROLLERS FOR “SMART” HEAT-TRANSFER SYSTEMS

[0067] As noted, the controller 127 (FIG. 3) and the controller 227 (FIG. 4) has a communication connection with one or more sensors (not shown) associated with its corresponding cooling system. Such sensors can determine, for example, one or more of an external air temperature, a dew point temperature, an external temperature of a facility supply inlet 121 / 221, a temperature of incoming facility coolant, a temperature of the facility coolant returned by the heat exchanger 125/225, a temperature of coolant entering the heat exchanger 125/225 from the collection manifold 122/222, a temperature of coolant leaving the heat exchanger 125/225 (e.g., entering the pump 126/226), a temperature of coolant exiting the coolant-distribution unit 120/220 and entering the coolant distribution manifold 124/224, a temperature of coolant entering each branch of the cooling loop (e.g., at the connections 114a-d/214a-d), a temperature of coolant entering each server-cooling node (e.g., at the inlets llla-d/211a-d), a temperature of coolant entering, within, or exiting servercooling node or a component-cooling node or a surface temperature of the component-cooling node (e.g., nodes 20a, 20b in FIG. 2), a case-temperature of an electronic component, a die-temperature of an electronic component, a temperature of a valve 112a-d/212a-d, a temperature of coolant exiting each server-cooling node HOa-d or rack-cooling node 210a-d, and a temperature of coolant entering the collection manifold 122/222 from each server-cooling node HOa-d (e.g., at the branch connections 115a-d) or from each rack-cooling node. Although temperature sensors are specified in the immediately preceding sentence, exemplary sensors associated with the modular cooling system 100/200 also include pressure sensors, which can measure static pressure, dynamic pressure, or a difference therebetween, at or around each of the foregoing positions, whether internal to the conduits of the cooling system (and thus indicating pressure within the coolant) or external to the conduits (and thus indicating pressure within the datacenter environment to which the cooling system 100/200 is exposed, e.g., a barometric pressure). Other representative sensors include, by way of example, leak-detection sensors located at one or more of the foregoing or other locations, flow-rate or fluid-speed sensors, humidity sensors, sensors for measuring a wet-bulb temperature or dew point, etc. Still other sensors can measure one or more physical parameters indicative of a coolant’s “health,” e.g., cooling capacity and/or remaining coolant life. Such sensors can include, by way of example, pH sensors for measuring coolant pH, hygrometers for measuring data-center humidity, hydrometers for measuring coolant density (or volume- or weight-fraction of water-to- additive (e.g., water-to-glycol)), refractomers for measuring coolant density (or volume- or weightfraction of water-to-additive (e.g., water-to-glycol)), and other sensors suited for determining, for example, pH, reserve alkalinity, turbidity, electrical conductivity, volume- or weight-fraction of water-to-additive (e.g., water-to-glycol) and/or any other measure of “health” of the coolant within the cooling system 100/200.

[0068] Further exemplary sensors include sensors configured to determine an operating speed of the pump 126/226, power being delivered to a component, a server and/or a rack of servers, IT workload, and even a configuration or type of component, server, or rack of servers. In some embodiments, sensors can be hardware sensors or sensors can be embodied in software or firmware (e.g., a software or firmware sensor can output information relating to IT workload on or among a group of components, servers or racks, and the controller 127/227 can receive such information over a communication connection (e.g., communication connection 128/228 or 129/229).

[0069] In still another embodiment, the controller can receive configuration information (as may be stored in read-only memory or other memory) pertaining to one or more thermal characteristics or cooling requirements for a component, a server, or a rack within a given datacenter installation. For example, a such a memory may store information relating to component or server power dissipation under different IT workloads. As another example, such a memory may store information relating to cooling system parameters, such as which model of pump is installed in a given coolant-distribution unit, or even what size of impeller is provided in a pump. Such information pertaining to pumps can be used by the controller to identify pump performance curves (e.g., to assess or to determine available pressure head for across a range of output flow-rate) for various pump speeds and/or impeller sizes.

[0070] Similarly, read-only memory can provide the controller with information pertaining to the number of rack-cooling nodes, server-cooling nodes within each rack, and component-cooling nodes within each server. The controller can use such information, combined with information pertaining to the plumbing configuration for each branch of the coolant loop, to determine or to estimate how pressure varies throughout the coolant loop under various pump-outlet flow rates. With knowledge of a pump curve and system pressure curve, output flow-rate from a coolant-distribution unit 120/220 can be estimated for a given coolant loop.

[0071] Moreover, a network-flow-modeling analysis can inform the controller as to coolant flowrate through each branch of the coolant loop for a given output flow-rate from the coolantdistribution unit 120/220. For example, based on a mass-balance, the mass-flow rate through a segment 131/231 of a coolant loop equals the mass-flow rate through the segment 124/224 less the mass-flow rate through the valve 112a/212a. Mass-flow rate through the segments 133/233, 135/235, 137/237 can be similarly determined. Further, mass flow-rate through the segment 132/232 equals the mass-flow rate entering the coolant distribution unit 120/220 from the collection manifold 122/222 less the mass-flow rate returning to the collection manifold through the segment 113a/213a. Mass-flow rate through the segments 234, 236, and 238 can be similarly determined based in part on the mass-flow rates returning from the rack-cooling nodes through the segments 113b-d/213b-d.

[0072] And still further, the controller can estimate a cooling capacity of each cooling node based in part on an estimated coolant flow rate through each cooling node. If the cooling capacity for any cooling node exceeds a cooling demand estimated from sensor inputs and configuration information, the controller can output a control signal to adjust one or more valves (e.g., to increase or to decrease a degree of valve openness or closed-ness) and/or to adjust a pump speed.

VI. WASTE-HEAT RECOVERY SYSTEMS

[0073] FIG. 5 schematically illustrates an installation of one or more heat transfer systems as described above in connection with FIGs. 1 through 4. As with systems described above, the installed heat-transfer system 300 provides cool facility coolant to a coolant distribution unit 320. The coolant distribution unit 320 is analogous to the coolant distribution units 120, 220 described above.

[0074] For example, the coolant distribution unit 320 facilitates cooling of a server-cooling loop, receiving coolant heated by one or more heat-dissipation nodes, rejecting heat Q to a facility coolant and returning the cooled coolant to the one or more heat-dissipation nodes, e.g., server nodes as in FIG. 3 or rack nodes as in FIG. 4. The installed heat-transfer system 300 conveys the now-heated facility coolant to a facility-level cooler 340, which facilitates rejecting heat 345 from the facility coolant to an ultimate heat-sink 341, 342 (e.g., the atmosphere, the Earth, a body of water, Space, etc.), cooling the facility coolant. The installed heat-transfer system 300 subsequently conveys the now-cooled facility coolant back to the coolant distribution unit 320 to provide further cooling to the server-cooling loop.

[0075] More particularly, in the illustrated embodiment, the coolant-distribution unit 320 receives heated server/rack coolant from a collection manifold 322 and delivers cool server/rack coolant to a distribution manifold 324 after rejecting heat from the server/rack coolant to the facility coolant. The coolant-distribution unit 320 has within, in this embodiment, a liquid-to-liquid heat exchanger (not shown) that rejects heat, Q, from the server/rack coolant received from the collection manifold 322 to the relatively cooler facility coolant received from the facility supply inlet 321. As the facility coolant passes through the heat exchanger, it absorbs the heat, Q, and increases in temperature, eventually exiting the heat exchanger through the facility return conduit 323. After rejecting the heat, Q, the now cooled server/rack coolant returns to the several nodes to be cooled, e.g., rack-nodes or server-nodes (or both).

[0076] In addition to providing cool facility coolant to the coolant-distribution unit 320, the system 300 can recover some waste heat carried by the facility return conduit 323. For example, the system 300 can include a waste-heat-recovery branch 351 fluidly coupled with the return conduit 323, e.g., with a fluid coupler 356. With such a branch 351, a portion (e.g., a fractional portion or a whole portion) of the heated facility coolant can be directed from the return conduit 323 into the waste- heat-recovery branch 351. The waste -heat-recovery branch 351 can convey the warm facility coolant to a waste -heat-recovery device 350. A valve 359a can be used to cause some or all of the facility coolant to be directed through the waste-heat-recovery branch 351. For example, when the valve 359a is partially closed, a portion (e.g., a minor fractional portion, a major fractional portion, or a whole portion) of the facility coolant can pass through the valve 359a while a balance of the flow of facility coolant will tend to pass into the branch 351.

[0077] A waste-heat-recovery device 350, in turn, can include a heat-exchanger (e.g., a liquid-to- liquid heat exchanger, a liquid-to-air heat exchanger, etc.) configured to facilitate heat transfer 355 from the heated facility coolant received from the waste -heat-recovery branch 351 to another, cooler working fluid. For example, the waste-heat-recovery device 350 can receive a flow of cool working fluid through an inlet conduit 352 and can provide a thermal coupling between the flow of warm facility coolant and the relatively cooler working fluid, allowing the working fluid to absorb the transferred heat 355. The waste-heat-recovery device 350 can convey the then-heated working fluid through an outlet conduit 354 for other heating purposes (e.g., floor heating, industrial heating, room heating, etc.)

[0078] In some embodiments, the working fluid can be room air from an office space and the waste- heat-recovery device 350 can be used to provide or to supplement heating of the room air. In other embodiments, the working fluid is a liquid, e.g., water or a mixture containing water. A liquid heated by the waste -heat-recovery device 350 can be used in a radiant heating system to supplement room heating in some embodiments or to supply heat to an industrial process in other embodiments.

[0079] After rejecting the heat 355 from the facility coolant, the waste-heat-recovery device 350 can return the now-cooled facility coolant to the main facility loop through the return line 353. In the embodiment in FIG. 5, a coupler 357 joins the return line 353 with the facility return conduit 323. As well, a shunt conduit extends from the return conduit 323 to the supply conduit 321, e.g., a coupler 358 couples the shunt conduit with the supply conduit 321.

[0080] A valve 359b can control whether facility coolant passes through the shunt conduit. For example, when a temperature of the facility coolant in the return line 353 is at or below a threshold temperature for the inlet to the facility-level cooler 340, the valve 359b can permit the relatively cool facility coolant from the return line 353 to directly flow into the supply conduit 321.

Alternatively, another portion (e.g., a minor fractional portion, a major fractional portion, or a whole portion) of the facility coolant returning from the waste-heat-recovery device 350 can be allowed to pass into the supply conduit 321 with the balance of the facility coolant returning from the waste- heat-recovery device 350 being allowed to pass to an inlet to the facility-level cooler 340. For example, when the valve 359b is partially closed, a portion (e.g., a minor fractional portion, a major fractional portion, or a whole portion) of the facility coolant can pass from the return line 353 through the valve 359b, while a balance of the flow of facility coolant from the return line will tend to pass into the facility return line 323 and ultimately into the facility-level cooler 340.

[0081] Although FIG. 5 shows only valves 359a and 359b, the system 300 can include one or more additional valves. For instance, an optional control valve (not shown) can be positioned within a branch of the facility-level loop having the facility-level cooler 340, e.g., in the return line 323 between the coupler 357 and an inlet to the facility-level cooler 340, in the supply line 321 between the facility-level cooler 340 and the coupler 358, or within a conduit of the facility-level cooler 340. The optional control valve can be closed when the valve 359b is opened. Such an arrangement will ensure that facility coolant passes through the shunt conduit (e.g., through the valve 359b) from the return line 323 to the supply line 321.

[0082] Whether cooled by one or both of the facility-level cooler 340 and the waste -heat-recovery device 350, the system 300 supplies cool facility coolant to the coolant distribution unit 320 through the supply line 321. [0083] Further, operation of the valve 359b with the optional control valve can be coordinated to selectively control the portion (e.g., a minor fractional portion, a major fractional portion, or a whole portion) of the facility coolant that passes through the shunt conduit from the return line 323 to the supply line 321 compared to the portion (e.g., a minor fractional portion, a major fractional portion, or a whole portion) of the facility coolant that passes through the facility-level cooler 340 from the return line 323 to the supply line 321. For example, as described more fully below, a controller can selectively adjust the valve 359b and selectively adjust the optional control valve. As a more particular, but non-exclusive example, the controller can adjust each of the valve 359b and the optional control valve to a desired opening within a range of opening arrangements extending from a fully closed arrangement to a fully open arrangement to achieve a desired apportionment of flow rate of the facility-level coolant through the shunt conduit and through the facility-level cooler 340.

[0084] As just one example, a controller can incorporate a processing unit. The processing unit can implement a known flow-network modelling technique using stored or retrievable head-loss information associated with the system 300 to select an opening position for each of the valve 359b and the optional control valve that suitably apportions the flow of facility coolant through the shunt conduit and the facility-level cooler. For example, a data store or other accessible memory can store head-loss information associated with valve 359b, the optional control valve and other portions of the system 300 (e.g., shunt-conduit, the branch of the system 300 that includes the facility-level cooler 340 and the remainder of the fluid-distribution loop in the system 300). The processing unit can incorporate and combine such head-loss information with known or modeled pump curves to assess and determine flow rates through one or more (or all) of the conduits of the system 300. As well, the controller can acquire sensor data and/or predict temperatures and pressures at one or more locations throughout the system 300. Responsive to a flow-rate, a pressure at a given location or a pressure-drop through a segment of the system 300, a temperature at a given location or a change in temperature through a segment of the system, or another observed or computed condition of the system 300, the controller can adjust operation of one or more flow-control devices within or among the system 300 (e.g., the valve 359b, the optional control valve, or the valve 359a, another system valve, a pump). Such adjustments can be made while commissioning a new system, starting up a newly-installed system, or during real-time operation of the system 300.

[0085] FIG. 5 also shows an optional second coolant distribution unit in dashed lines. As with the coolant-distribution unit 320, the coolant-distribution unit in dashed lines can receive cool facility coolant from the facility supply 321, e.g., through supply branch 321a, which can facilitate cooling of a second secondary cooling loop by absorbing heat in a manner as described in connection with the coolant-distribution unit 320. Similarly, the coolant-distribution unit in dashed lines can return heated facility coolant to the facility return 323, e.g., through the return branch 323a. In addition, though not illustrated in FIG. 5, one or more further coolant-distribution units analogous to the coolant-distribution unit 320 can be coupled with the facility supply 321 and the facility return 323 in a manner analogous to that shown and described in reference to the coolant-distribution unit in dashed lines. A flow rate of facility coolant through the facility supply 321 and facility return 323, as well as a cooling capacity provided by one or more waste -heat-recovery devices analogous to the waste -heat-recovery device 350 and the facility-level cooler 340, can be selected according to an anticipated demand for cooling imposed by a selected number and sizes of coolant-distribution units 340 to be cooled by the facility system 300.

[0086] The embodiment of the system 300 discussed above recovers waste -heat from a facility coolant. That being said, principles and aspects of heat-transfer systems described above in detail in connection with the system 300 can be applied analogously to other heat transfer systems described above in detail. Accordingly, a waste-heat recovery device can be configured to recover waste -heat from coolant circulating through any of the modular heat-transfer systems described above (e.g., from coolant circulating through a coolant-distribution unit and one or more corresponding component-cooling nodes).

VII. CONTROL LOGIC, PUMP CURVES AND SYSTEM OPERATING CURVES

[0087] As noted above, pressure loss through a given coolant loop or branch thereof as a function of mass-flow rate of coolant through the loop or branch can be adjusted by a controller 127/227. FIG.

5 schematically illustrates, for a given pump operating at a constant speed, how the system operating point (and resulting pump-efficiency) can change by adjusting, for example, one or more flowcontrol valves that impart a higher- or a lower-pressure loss through the coolant loop (or branch).

[0088] FIG. 5 shows a plot 300 of pressure head (or loss) as a function of flow rate for a pump and for a given coolant loop. The curve 310 depicts a nominal curve of pressure-loss through a coolant loop as a function of flow-rate through the coolant loop. As the curve 310 shows, pressure loss through the coolant loop (or branch thereof) increases (or decreases) roughly with the square of flow rate. The operating point (flow rate and pressure drop) 320 for the coolant loop (or branch) is found by the intersection of the system curve 310 with the pump curve 330. The pump curve 330 depicts how pressure head provided by the pump varies as a function of discharge flow rate provided by the pump. Overlaid on the plot 300 are estimates of different pump efficiencies (e.g., available fluid energy discharged by the pump divided by the power consumed by the pump at different operating points along the pump curve 330. For the pump whose operating curve 330 is depicted in FIG. 5, the point of most efficient operation lies between the tick-marks labeled hi.

[0089] As the operating point moves to higher discharge heads and lower flow rates from the upper tick-mark labeled hi or to lower discharge heads and higher discharge flow rates, pump efficiency drops, as indicated by the tick-marks labeled I12 and ha. Accordingly, by partially closing a flowcontrol valve or removing a server-cooling node, overall pressure loss increases across various flow rates through the coolant loop, as shown by the adjusted system curve 314. Perhaps counterintuitively, for this particular pump and system adjustment, overall pumping efficiency can increase, as indicated by movement from a region of relatively lower efficiency (the lower tick-mark labeled I12) to a region of relatively higher efficiency (toward the tick-mark labeled hi).

[0090] By contrast, by partially opening a flow-control valve or perhaps by adding a server-cooling node to a branch of the coolant loop, overall pressure loss can decrease across various flow rates through the coolant loop, as shown by the adjusted system curve 312. Again, perhaps counterintuitively, for this particular pump and system adjustment, overall pumping efficiency can decrease, as indicated by moving from a region of relatively higher pump efficiency to a region of relatively lower pump efficiency.

[0091] The foregoing discussion of operating point was based on a pump operating at a constant speed. However, as noted above, a controller 127/227 can control the speed of one or more pumps, adding to the degrees-of-freedom available to the controller to tailor flow-rate (and thus cooling capacity) among rack-, server-, and component-cooling nodes to achieve efficient overall operation. For example, FIG. 6 shows a family 400 of pump curves 402, 404, 406 and 408, each of which corresponds to a given pump speed different from the speed corresponding to the pump curve 330. Overlaid on the family 400 of pump curves are lines of constant pump efficiency (hi, I12, and hi), as well as lines indicating pumping power (Pi, P2, and P3). Combining the family 400 of pump curves shown in FIG. 6 with the system curve 310 and its variants 312, 314 arising from adjusting flowcontrol valves (e.g., valves llla-d/211a-d) can yield the operating point for the cooling systems 100, 200. [0092] As briefly explained above, these various pump curves and system curves can be stored in a retrievable memory (e.g., memory 92, storage 94, or storage 98b in FIG. 6). A controller (e.g., controller 127, 227 in FIGs. 3 and 4) can retrieve these pump curves and system curves to select a suitable pump speed and valve position among the flow-control valve based, for example, on a measure of IT workload to be cooled by each component-, server-, and/or rack-cooling node within the cooling system 100, 200.

[0093] For example, a given heat-transfer system installation can be calibrated as by determining an overall system curve for each in a plurality of setting combinations of the flow-control valves 112a- d, 212a-d. These calibrations can be stored in a memory and retrieved by the controller when it is called to assess and if appropriate to adjust one or more flow-control valve settings based on a given distribution of IT workload among the servers and components cooled by the heat-transfer system. Similarly, a family of pump curves (and the annotated power curves and efficiency curves as in FIG. 6) can be stored in a memory and retrieved by the controller. The controller (or an associated processor) can select a suitable combination of flow-control-valve settings and pump speeds to achieve a system operating point that provides an adequate degree of cooling to each rack-, server-, and/or component-cooling node of interest based on the observed distribution of IT workload.

[0094] Although a closed- form solution to such an optimization problem would rarely if ever be available, computational approaches suitable for such optimizations are well-known under the general category of machine learning. As but several exemplary approaches, a Bayesian search, a gradient descent method, a genetic algorithm and a spectral method can be used to identify a suitable combination of valve settings and pump speeds to provide a distribution of cooling among various rack-, server-, and component-cooling nodes tailored to a distribution of IT workload to be cooled.

[0095] In still a further embodiment, the distribution of IT workload can itself be adjustable to obtain efficient overall operation (e.g., based on a combination of power consumed by the IT workload and power consumed by the cooling system). For example, adjusting the distribution of IT workload (which here is being used as a proxy for the distribution of dissipated heat among the various components, servers and racks in a data center) can facilitate tailoring the distribution of cooling among various rack-, server-, and component-cooling nodes to the IT workload. For instance, if a given cooling node of a liquid-based cooling system receives a relatively high massflow rate of coolant (and thus has available a relatively high cooling capacity) across a variety of flow-control valve and pump settings, a data center installation that can selectively adjust the distribution of its IT workload stands to benefit from concentrating relatively heavier IT workloads among the components, servers, and/or racks cooled by that higher-capacity cooling node and distributing relatively lower IT workloads elsewhere to be cooled by the relatively lower-capacity cooling nodes.

VIII. TELEMETRY, ANOMALY DETECTION, AND A SYSTEM CONTROL GATEWAY [0096] Telemetry data from various sensors and flow devices described above can be tied into the Data Center Information Manager (DCIM) software and Building Management System (BMS) software to monitor the system performance, as a whole. For example, a system control gateway can provide a communication interface between control logic for a coolant-distribution unit (CDU) (e.g., the controller 127 in FIG. 3 or the controller 227 in FIG. 4) and a pre-existing DCIM / BMS.

[0097] In an embodiment, the gateway has a first communication connection with the control logic for the liquid-cooling system and a second communication connection with the DCIM / BMS system. Notably, a communication protocol over the first communication connection with the liquid-cooling system’s control logic can differ from a communication protocol over the second communication connection with the DCIM / BMS system. Further, the physical (or wireless) interfaces over which the communication connections exist can differ between the first communication connection and the second communication connection. For example, the first communication connection (e.g., to the liquid-cooling system’s control logic 127) can incorporate Cat6 Ethernet connections for SNMP, Modbus IP, RS-485, CANBus and other known communication protocols. And, the first communication connection can include one or more discrete I/O and analog input connections suitable for various types of sensors, including for example, leak-detection sensors. An advantage of digitized communication of sensor signals using such protocols, for example, is that the sensors need not be positioned close to the gateway device. Further, digitized sensor signals can be less susceptible to electro-magnetic interference and other sources of noise that can interfere with a sensor signal, providing a more reliable signal over longer distances than analog sensor signals can provide.

[0098] By contrast, DCIM and BMS systems commonly interface with a network over an ethernet connection using other communication protocols, such as, for example, BACNet, SNMP, Modbus and Redfish. Nonetheless, the gateway can serve as a middle-ware layer that translates information pertaining to the liquid-cooling system into information that the DCIM and BMS system can absorb For example, the secondary supply temperature in one embodiment can be read from address 30015 as a 16-bit signed integer with a scale of 0.1. In another embodiment, he secondary supply temperature can be read from address 30012 as a 32-bit signed integer with a scale factor of 0.1. In this example, the gateway can incorporate a look-up table or other data store that relates each in a plurality of cooling systems with a corresponding address, data length, and scale factor for the secondary supply temperature. Of course, a given liquid-cooling system can incorporate any number of sensors of various types, as described above. Accordingly, the data store can relate each liquid-cooling system (or component) with a selected plurality of sensors and sensor types. Further, the data store can relate each sensor or sensor type with an address, word length, scale factor and any other selected parameter suitable to obtain relevant information.

[0099] Additionally, the gateway can map each sensor or sensor type to a given variable or register in a DCIM or BMS data structure. Accordingly, when the gateway receives information from the liquid cooling system or its control logic, the gateway can map the received data to a suitable channel in the DCIM or BMS system, allowing the DCIM or BMS system to receive data from the liquid-cooling system. With such a standardized gateway, the amount of software required to interface to a new device can be significantly reduced.

[0100] Further, such a gateway can provide enhanced anomaly detection across a plurality of installed heat-transfer systems. For example, control logic for a given modular heat-transfer system, as described above, may receive information from a selected array of sensors and/or servers. Nonetheless, a given data-center installation may include a plurality of such modular heat-transfer systems, e.g., dozens, hundreds, or thousands of such systems. Accordingly, a gateway can access information from each modular heat-transfer system and not just from the servers or sensors associated with a single cooling loop. Accordingly, the gateway can implement a multi-variate anomaly-detection technique to assess whether operating anomalies (e.g., leaks, over-temperature components, failed or failing pumps, etc.) have occurred or are likely to occur within any one or more in the plurality of heat-transfer systems.

[0101] Similarly, the gateway can predict when maintenance or repairs should occur, rather than relying solely on pre-defined maintenance intervals or waiting for an outright failure to occur. For example, a gateway can monitor flow rate as a function of pump speed evaluate how that relationship might evolve over time. In this example, if flow-rate decreases over time at a given pump speed, then a pump may need to be serviced or a filter may need to be changed (e.g., a decrease in flow -rate can indicate a failed or failing pump or increased pressure -head in the system due to filter blockages).

[0102] Further, a gateway can receive an interrupt or other signal directly from a leak-detection sensor or via a controller (e.g., control logic 127, 227), providing a facility with a pathway for responding to a leak, in addition to the response provided by a given cooling node or cooling loop. For example, a DCIM or a BMS might adjust or re-route a supply of coolant to the branch in which a leak is detected, in addition to, for example, a cooling loop’s controller disconnecting or otherwise isolating coolant from or within the branch. As another example, input to a DCIM or a BMS can permit the DCIM or BMS to adjust a server workload or to transfer the server workload from a server affected or in the vicinity of a leak to another server remotely positioned from the leak and out of harm’s way from the leak. In such an operation, the DCIM or BMS can supplement or otherwise route coolant to the remotely positioned server.

[0103] As yet another advantage provided by a gateway that aggregates information across closed- loop or modular heat-transfer system, overall energy consumed by cooling an installation can be monitored, controlled or reduced. For example, a data-center installation can have a plurality of closed-loop or modular heat-transfer systems. Within each closed-loop system, control logic can determine a minimum flow rate that each rack (or node) needs to maintain a given component (e.g., a processor) at or below a threshold temperature, as described above.

[0104] A gateway that monitors a plurality of closed-loop heat-transfer systems, in turn, can aggregate control over the plurality of closed-loop systems and inform whether pumps in a given closed-loop system need to be operated at a higher flow rate (and thus consume more power) or whether they can operate a lower flow rate (and thus consume less power). For example, a gateway in communication with a DCIM / BMS system (e.g., over an IPMI bus) can receive information relating to a current or an anticipated workload on one or more servers. The gateway can respond to a current or anticipated workload for a group of servers in a given closed-loop system by tailoring flow rates within those servers to the workload.

[0105] Similarly, the gateway can assess a measure of cooling margin remaining in a given closed- loop system and communicate that assessment to a DCIM / BMS controller, which can in turn tailor a workload on the servers cooled by that system to correspond to the remaining margin. For example, one or more servers cooled by a given closed-loop heat-transfer system might have excess cooling applied to them under a given workload. The gateway can observe the operating conditions of those one or more servers, determine that the servers are available for increased workload, and communicate that availability to the DCIM / BMS system. In response, the DCIM / BMS system can schedule future workloads to be applied to the under-utilized servers, balancing workloads across the servers in a given installation while ensuring that power consumed by the cooling system is kept low or is minimized. Further, a smart manifold as described above can be used to tailor cooling across a plurality of nodes to similarly maintain lower power usage while providing adequate server cooling.

IX. COMPUTING ENVIRONMENTS

[0106] FIG. 7 illustrates a generalized example of a suitable computing environment 90 in which described methods, embodiments, techniques, and technologies relating, for example, to maintaining a temperature of a logic component and/or a power unit below a threshold temperature can be implemented. The computing environment 90 is not intended to suggest any limitation as to scope of use or functionality of the technologies disclosed herein, as each technology may be implemented in diverse general-purpose or special-purpose computing environments. For example, each disclosed technology may be implemented with other computer system configurations, including wearable and/or handheld devices (e.g., a mobile-communications device), multiprocessor systems, microprocessor-based or programmable consumer electronics, embedded platforms, network computers, minicomputers, mainframe computers, smartphones, tablet computers, data centers, servers and server appliances, and the like. Each disclosed technology may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications connection or network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0107] The computing environment 90 includes at least one central processing unit 91 and a memory 92. In FIG. 7, this most basic configuration 93 is included within a dashed line. The central processing unit 91 executes computer-executable instructions and may be a real or a virtual processor. In a multi-processing system, or in a multi-core central processing unit, multiple processing units execute computer-executable instructions (e.g., threads) to increase processing speed and as such, multiple processors can run simultaneously, despite the processing unit 91 being represented by a single functional block. A processing unit can include an application specific integrated circuit (ASIC), a general purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures arranged to process instructions.

[0108] The memory 92 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory 92 stores software 98a that can, for example, implement one or more of the technologies described herein, when executed by a processor.

[0109] A computing environment may have additional features. For example, the computing environment 90 includes storage 94, one or more input devices 95, one or more output devices 96, and one or more communication connections 97. An interconnection mechanism (not shown) such as a bus, a controller, or a network, interconnects the components of the computing environment 90. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment 90, and coordinates activities of the components of the computing environment 90.

[0110] The store 94 may be removable or non-removable, and can include selected forms of machine -readable media. In general machine-readable media includes magnetic disks, magnetic tapes or cassettes, non-volatile solid-state memory, CD-ROMs, CD-RWs, DVDs, magnetic tape, optical data storage devices, and carrier waves, or any other machine-readable medium which can be used to store information and which can be accessed within the computing environment 90. The storage 94 can store instructions for the software 98b, which can implement technologies described herein.

[0111] The store 94 can also be distributed over a network so that software instructions are stored and executed in a distributed fashion. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0112] The input device(s) 95 may be any one or more of the following: a touch input device, such as a keyboard, keypad, mouse, pen, touchscreen, touch pad, or trackball; a voice input device, such as a microphone transducer, speech-recognition software and processors; a scanning device; or another device, that provides input to the computing environment 90. For audio, the input device(s) 95 may include a microphone or other transducer (e.g., a sound card or similar device that accepts audio input in analog or digital form), or a computer-readable media reader that provides audio samples to the computing environment 90.

[0113] The output device(s) 96 may be any one or more of a display, printer, loudspeaker transducer, DVD-writer, or another device that provides output from the computing environment 90.

[0114] The communication connection(s) 97 enable communication over or through a communication medium (e.g., a connecting network) to another computing entity. A communication connection can include a transmitter and a receiver suitable for communicating over a local area network (LAN), a wide area network (WAN) connection, or both. LAN and WAN connections can be facilitated by a wired connection or a wireless connection. If a LAN or a WAN connection is wireless, the communication connection can include one or more antennas or antenna arrays. The communication medium conveys information such as computer-executable instructions, compressed graphics information, processed signal information (including processed audio signals), or other data in a modulated data signal. Examples of communication media for so-called wired connections include fiber-optic cables and copper wires. Communication media for wireless communications can include electromagnetic radiation within one or more selected frequency bands.

[0115] Machine -readable media are any available media that can be accessed within a computing environment 90. By way of example, and not limitation, with the computing environment 90, machine -readable media include memory 92, storage 94, communication media (not shown), and combinations of any of the above. Tangible machine-readable (or computer-readable) media exclude transitory signals.

[0116] As explained above, some disclosed principles can be embodied in a tangible, non-transitory machine -readable medium (such as microelectronic memory) having stored thereon instructions. The instructions can program one or more data processing components (generically referred to here as a “processor”) to perform a processing operations described above, including estimating, computing, calculating, measuring, adjusting, sensing, measuring, filtering, addition, subtraction, inversion, comparisons, and decision making (such as by the control unit 52). In other embodiments, some of these operations (of a machine process) might be performed by specific electronic hardware components that contain hardwired logic (e.g., dedicated digital filter blocks). Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components. [0117] For sake of brevity throughout this disclosure, computing-environment components, processors, interconnections, features, devices, and media are generally referred to herein, individually, as a “logic component.”

X. WORKING EMBODIMENT

[0118] In a working embodiment, a pressure independent balancing and temperature control valve having two temperature and two pressure sensors was used to adjust / control flow through portions of a working cooling system. The pressure sensors are mounted on both sides of the valve allowing for flow measurement, and the temperature sensors are mounted on the supply and return manifold and are used to measure the differential temperature across the IT gear. Maximizing the AT can reduce the amount of pumping power required to keep the IT gear operating below its maximum temperature.

[0119] Once configured, this valve can operate autonomously, or can be connected to the BMS to report telemetry data including a BTU meter and flow meter. The valves can be configured to fully open, fully close or remain in a given position if a power or control communication failure is experienced.

[0120] As discussed above, benefits of a “smart” manifold and/or a “smart” cooling system include efficiency increases. By controlling flow at a per-rack level, the required pumping power can be reduced, the number of racks serviced by one CDU can be increased and there can be better resolution in assessments of how a given cluster is performing.

[0121] The amount of energy saved by disclosed principles can vary according to how IT workload is managed by the cluster. Substantial savings can be achieved when the rack is at a low IT workload. This can occur when, for example, the rack is idle, or when a given workload is IO intensive, but not computationally intensive, for example.

[0122] For this example, consider a system that has the following configuration (or operating conditions: Tprimary = 45C; Qprimary = 300LPM; 15 racks, with 42 dual socket Skylake servers with a max TDP of 205W; and Tease, max=80C. The system can be designed to cool the maximum TDP of 205W with a 90% flow rate of the CHx750, which can deliver 0.74LPM. This gives an expected Tease for the first socket of 75.2C and for the second socket, 79.2C, as FIG. 8 shows. [0123] Such a scenario can arise from IO intensive workloads where the CPUs are only operating at about 100W of power consumption. In this case, the CDU pump speed can be reduced to 20%, for a flow rate of just 0.16LPM per node. In this configuration under this load, CPUs will be at 68.7C and 77.9C respectively.

[0124] The CDU pumps have been reduced by 70%, and the CDU power consumption drops from 4.1kW to the minimum of 2.5kW, a savings of approximately 40%. See FIG. 9.

[0125] As an alternative to electrical power savings, the number of racks supported by one CDU can be increased, which can reduce capital expenditures for a given installation. For example, using the same system parameters as above, the system can be designed to operate a pump at 100% of its rated speed to meet an upper threshold power dissipation (e.g., 100% thermal load), despite that the system is rarely being utilized to 100% of the upper threshold power dissipation. Instead of cooling a given system under such a high workload, an embodiment of a disclosed cooling system can adjust the coolant mass-flow rate according to the return temperature of coolant from each rack, diverting flow to the racks that need higher coolant mass-flow rate for higher cooling capacity.

[0126] As shown in FIG. 10, an additional rack can be added while maintaining a component case temperature of about 76.3 and 80.2C. Although this temperature may exceed an upper threshold temperature, this situation it is unlikely to occur in the field as a flow of coolant can be diverted from other racks under less load, either by a gateway or by the control logic 127/227.

[0127] Similarly, a leak detection system can be tied into the same DCIM/BMS to which the Smart Manifold is tied. This can allow for rack-level isolation when a leak is detected. In some embodiments, a flow-control valve (e.g., 112a-d, 212a-d) can incorporate a leak detection and/or an automatic server or rack isolation function. For example, the flow-control valve can automatically close on detection of a leak.

[0128] Co-location data centers can have different rack configurations. As these facilities adopt direct liquid cooling, they may find tailoring coolant mass-flow to a given server’s IT workload to be desirable. Technologies disclosed herein can avoid the use of manual, mechanical flow setter valves that require an operator to manually adjust settings for each rack. XI. OTHER EMBODIMENTS

[0129] The examples described above generally concern apparatus, methods, and related systems to tailor a cooling system’s cooling capacity to an observed or predicted distribution of IT workload. Nonetheless, the previous description is provided to enable a person skilled in the art to make or use the disclosed principles. Embodiments other than those described above in detail are contemplated based on the principles disclosed herein, together with any attendant changes in configurations of the respective apparatus or changes in order of method acts described herein, without departing from the spirit or scope of this disclosure. Various modifications to the examples described herein will be readily apparent to those skilled in the art.

[0130] Directions and other relative references (e.g., up, down, top, bottom, left, right, rearward, forward, etc.) may be used to facilitate discussion of the drawings and principles herein, but are not intended to be limiting. For example, certain terms may be used such as "up," "down,", "upper," "lower," "horizontal," "vertical," "left," "right," and the like. Such terms are used, where applicable, to provide some clarity of description when dealing with relative relationships, particularly with respect to the illustrated embodiments. Such terms are not, however, intended to imply absolute relationships, positions, and/or orientations. For example, with respect to an object, an "upper" surface can become a "lower" surface simply by turning the object over. Nevertheless, it is still the same surface and the object remains the same. As used herein, “and/or” means “and” or "or", as well as “and” and “or.” Moreover, all patent and non-patent literature cited herein is hereby incorporated by reference in its entirety for all purposes.

[0131] And, those of ordinary skill in the art will appreciate that the exemplary embodiments disclosed herein can be adapted to various configurations and/or uses without departing from the disclosed principles. Applying the principles disclosed herein, it is possible to provide a wide variety of cooling nodes, and related methods and systems to tailor a cooling system’s distribution of cooling capacity to an estimated or observed distribution of IT workload (or power dissipation). For example, the principles described above in connection with any particular example can be combined with the principles described in connection with another example described herein. Thus, all structural and functional equivalents to the features and method acts of the various embodiments described throughout the disclosure that are known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the principles described and the features and acts claimed herein. Accordingly, neither the claims nor this detailed description shall be construed in a limiting sense, and following a review of this disclosure, those of ordinary skill in the art will appreciate the wide variety of cooling nodes, and related methods and systems to tailor a cooling system’s distribution of cooling capacity to an estimated or observed distribution of IT workload (or power dissipation) that can be devised using the various concepts described herein. [0132] Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim feature is to be construed under the provisions of 35 USC 112(f), unless the feature is expressly recited using the phrase "means for" or "step for".

[0133] The appended claims are not intended to be limited to the embodiments shown herein, but are to be accorded the full scope consistent with the language of the claims, wherein reference to a feature in the singular, such as by use of the article "a" or "an" is not intended to mean "one and only one" unless specifically so stated, but rather "one or more". Further, in view of the many possible embodiments to which the disclosed principles can be applied, we reserve the right to claim any and all combinations of features and technologies described herein as understood by a person of ordinary skill in the art, including the right to claim, for example, all that comes within the scope and spirit of the foregoing description, as well as the combinations recited, literally and equivalently, in any claims presented anytime throughout prosecution of this application or any application claiming benefit of or priority from this application, and more particularly but not exclusively in the claims appended hereto.