MULTIPLE MODES OF OPERATION

BACKGROUND

Field

[0001] The present invention relates generally to providing resources for computing. More particularly, the present disclosure relates to systems and methods for operating and cooling computing systems using liquid coolants (e.g., oil-cooled systems).

Description of the Related Art

[0002] A data center typically includes a group of computing devices at a common physical location. Data centers are often housed in conventional building structures and use air conditioning systems to remove heat generated by electronic components (chips, hard drives, cards, etc.)

[0003] Many commercially-available servers used in data centers are designed for air cooling. Such servers usually comprise one or more printed circuit boards having a plurality of electrically coupled devices mounted thereto. These printed circuit boards are commonly housed in an enclosure having vents that allow external air to flow into the enclosure, as well as out of the enclosure after being routed through the enclosure for cooling purposes. In many instances, one or more fans are located within the enclosure to facilitate this airflow.

[0004] Data centers housing such servers and racks of servers typically distribute air among the servers using a centralized fan (or blower). Air within the data center usually passes through a heat exchanger for cooling the air (e.g., an evaporator of a vapor-compression cycle refrigeration cooling system (or "vapor-cycle" refrigeration), or a chilled water coil) before entering a server. In some data centers, the heat exchanger has been mounted to the rack to provide "rack-level" cooling of air before the air enters a server. In other data centers, the air is cooled before entering the data center.

[0005] In general, electronic components of higher performing servers dissipate correspondingly more power. However, power dissipation for each of the various hardware components (e.g., chips, hard drives, cards) within a server can be constrained by the power being dissipated by adjacent heating generating components, the airflow speed and airflow path through the server and the packaging of each respective component, as well as a maximum allowable operating temperature of the respective component and a temperature of the cooling air entering the server as from a data center housing the server. The temperature of an air stream entering the server from the data center, in turn, can be influenced by the power dissipation and proximity of adjacent servers, the airflow speed and the airflow path through a region surrounding the server, as well as the temperature of the air entering the data center (or, conversely, the rate at which heat is being extracted from the air within the data center).

[0006] Data centers often have large, unpredictable variations in computing demand. The computing demand may change from day to day, hour to hour or even more frequently. For example, a data center for that performing processing for retails sales may have high demand on "Black Friday" and, by contrast, low demand on a typical late night. As computational demand rising and falls, the demand for resources such as electrical power and cooling also rises and falls.

[0007] CPUs for some servers can operate in enhanced computational modes. For example, some Intel processors include a "Turbo" mode that allows the processor to operate at a power level that is higher than its rated upper power limit for short durations to maximize performance. These processors may implement algorithms that manage current, power and temperature. Thermal conditions in which the processor resides may, however, limit the processor's ability to maximum performance and/or efficiency.

SUMMARY

[0008] Embodiments of systems and methods for operating and cooling computing systems using a multi-mode controller are described herein. In some embodiments, a controller performs an optimization to enable processors in the servers to operate in an enhanced computational output mode (for example, Turbo mode). In one embodiment, a controller for liquid-cooled servers receives information about one or more servers operating in one or more liquid-cooled racks. The controller optimizes for an enhanced computational output mode (for example, Turbo mode) for server processors operating in the liquid-cooled racks. A liquid cooling system is operated by the controller to cool the servers to allow the server processors to operate in the enhanced computational output mode.

[0009] In an embodiment, a computing system includes one or more racks. The racks include one or more servers, one or more tanks that hold liquid coolant for the servers, a liquid cooling system, and a controller that controls the liquid cooling system. Liquid coolant held in the tanks removes heat from the servers. The controller optimizes for an enhanced computational output mode for server processors operating in the liquid-cooled racks. The liquid cooling systems are operated to cool the servers based on the optimization to allow the server processors to operate in an enhanced computational output mode.

[0010] In an embodiment, a method of controlling liquid-cooling of rack of servers includes receiving information about one or more servers operating in the liquid-cooled racks, optimizing for an enhanced computational output mode for the server processors operating in the liquid- cooled racks, operating the liquid cooling system to cool the servers based on the optimization to allow the server processors to operate in an enhanced computational output mode.

[0011] In an embodiment, a computing system includes one or more racks. The racks include one or more servers, one or more tanks that hold liquid coolant for the servers, a liquid cooling system, and a multi-mode controller that controls the liquid cooling system. Liquid coolant held in the tanks removes heat from the servers. The multi-mode controller selectively operates the liquid cooling systems in a computational performance mode or a resource efficiency mode. The computational performance mode prioritizes computational performance over resource efficiency. The resource efficiency mode prioritizes resource efficiency over computational performance.

[0012] In an embodiment, a method of controlling liquid-cooling of rack of servers includes receiving information about one or more servers operating in the liquid-cooled racks, optimizing for an enhanced computational output mode for the server processors operating in the liquid- cooled racks, operating the liquid cooling system to cool the servers based on the optimization to allow the server processors to operate in an enhanced computational output mode. The system is switched between a computational performance mode and a resource efficiency mode. The computational performance mode prioritizes computational performance over resource efficiency. The resource efficiency mode prioritizes resource efficiency over computational performance.

[0013] In an embodiment, a multi-mode controller operates liquid coolant systems in a computational performance mode or a resource efficiency mode. The computational performance mode optimizes for computational performance (for example, enabling an enhanced processor mode). The resource efficiency mode optimizes for efficiency of at least one resource (for example, to minimize power consumption). The mode of operation may be switchable by a user.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1A illustrates an embodiment of computing system including enclosed cooling module for the racks. [0015] FIG. IB illustrates one embodiment of an exemplary system for efficiently cooling a plurality of independently operable data processing modules.

[0016] FIG. 1C illustrates an alternative embodiment of an exemplary system for efficiently cooling a plurality of independently operable data processing modules.

[0017] FIG. 2 illustrates a rack that can be used in a computing module in various embodiments.

[0018] FIG. 3 illustrates one embodiment of a computing module including racks with liquid- cooled servers, power distribution components, and an under-floor cooling module.

[0019] FIG. 4 is a schematic view illustrating circulation at a section of a rack in one embodiment.

[0020] FIG. 5 is a schematic diagram illustrating a data center with a multi-mode controller with a cooling mode that allows increased computational output.

[0021] FIG. 6 illustrates optimizing liquid cooling for an enhanced computational output mode for servers operating in a rack.

FIG. 7 illustrates a method of liquid cooling servers in a rack that includes switching between computational performance mode and a resource efficiency mode.

[0022] FIG. 8 illustrates a method of liquid-cooling servers in a rack that includes adjusting cooling parameters to allow a processor to operate in an enhanced computational performance mode.

[0023] While the invention is described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that the invention is not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words "include", "including", and "includes" mean including, but not limited to.

DETAILED DESCRIPTION OF EMBODIMENTS

[0024] In various embodiments, a controller for a liquid-cooled rack (e.g., oil-cooled rack) operates in different modes of operation, including one mode optimized for overall power consumption, and another mode optimized for computational output. In certain embodiments, the controller optimizes to allow an Intel processor to operate in a Turbo mode.

Computing System with Cooling Module

[0025] In various embodiments, a plurality of servers is mounted vertically in a rack. The servers may be vertically removed and replaced from a rack with an open top. The servers may be mounted in an array, arranged horizontally. Each server may be removed without affecting the functionality of other servers in the rack or in the data center. Each server may operate independently of each of the servers.

[0026] In some embodiments, spacing and structure allow for accommodation of many different form factors, including but not limited to conventional rack mount servers normally used for air cooling. Servers may be mounted adjacently to each other to minimize upward flow around the servers or motherboards.

[0027] In some embodiments, dielectric fluid is pumped out of a rack, cooled by flowing through a heat exchanger, and pumped back into the rack. In other embodiments, the heat exchanger is located inside the rack. A secondary liquid circuit flows into the rack with dielectric fluid, and through a heat exchanger, cooling the dielectric fluid.

[0028] In one embodiment, a dielectric coolant flows out of a rack at an elevated temperature. A circuit may include a pump, heat exchanger, and measurement devices. A secondary circuit flows through the above heat exchanger, cooling the dielectric fluid, then flows outside and dissipates heat external to the room housing the dielectric-filled rack.

[0029] Systems and methods of cooling and operating electronic devices using liquid coolant- filled racks may be as described in US Patent Publication No. 2011/0132579 (the "'579 Publication"), by Best et al., published June 9, 2011, which is incorporated by reference in its entirety as if fully set forth herein.

[0030] In some embodiments, a computing system includes a row of liquid-cooled racks on either side of an aisle. A cooling module is included under a floor of the aisle.

[0031] FIG. 1A illustrates one embodiment of a computing system with a cooling module between two rows of racks. Computing system 1 10 includes racks 1 12. A cooling system 1 14 is provided for each rack. Each of racks 1 12 is supplied power through a power distribution unit 1 16. Cable trays 1 18 hold cables that supply power or enable exchange of data between servers in racks 112 and external systems.

[0032] Each of racks 112 includes tank 122. In the embodiment shown in FIG. 1A, a plurality of tanks 122 are provided, each tank 122 containing vertically mounted, independently removable and replaceable data processing modules. As shown in FIG. 1A, in this embodiment, tanks 122 are arranged in two banks adjacent an aisle 124. The tanks may be arranged in other configurations, however. For example, a single bank of tanks 122 may be installed in the center of the unit with aisles on either side of tanks 122. Or a single bank of tanks 122 may be installed against a wall of an enclosure (e.g., a shipping container housing racks and the cooling module.)

[0033] Cooling system 1 14 includes pumps 130, one or more heat exchangers, and associated pipes and control systems. Pumps 130 and the heat exchanger(s) are included in the form of pump module 135. Pump module 135 includes enclosure 136. Enclosure 136 houses the pumps and other components of pump module 135.

[0034] The pipes in between the rack and the pump module may be made of steel or another temperature resistance material such that if a fire breaks out, the oil carrying pipes will not melt and will contain the fluid. In addition, the enclosure for the pump module and the rack housing may be made of a fire-retardant material, such as steel.

[0035] In some embodiments, the floor of aisle 124 (which, in the embodiment shown in FIG. 1, is the top of the enclosure for the cooling module) is permeable such that coolant (for example, oil) can drip through the floor into the cooling module. Allowing coolant on the floor to seep into a porous material may extinguish or suppress fire (for example, by isolating the oil in the pores of the material from air).

[0036] The top of a pump module and any removable service panels for a pump module may be made of flame resistant material. The top of the pump module may have a permeable surface such that any coolant (for example, oil) that has spilled into the aisle drains drips through the top and passes into pump module. In certain embodiments, the system includes one or more trays or pans that collect spilled fluid around the rack or pump module and drain it into the pump module.

[0037] As shown in FIGS. 1A and IB, module 135, which may include at least elements as shown in FIGS. IB, for example, provided cooling for tanks 122, according to one or more embodiments. That is, according to one or more embodiments, each pump module 135 may include primary and secondary pumps 130 (and associated pump motors) connected to filter 160 and liquid coolant heat exchanger 140 of at least one bank of tanks 122 via fluid circuit 170 such that primary and secondary pumps 130 may function independently of one another for backup purposes, with electrically isolated pump 130 motors. According to one or more embodiments, primary pump 130 motor is controlled by variable speed controller 180 for regulating temperature of coolant loop 170 by varying liquid coolant flow, whereas secondary pump motors may be fixed-speed and controlled by on-off control. [0038] Module 135 for evaporative cooling apparatus 150, according to one or more embodiments, includes a controller for controlling a pump motor in loop 175, which may be on- off control or variable speed control, according to one or more embodiments, and includes a controller for one or more fans, motor , which may be like controller 180 of Figure IB, for example, but for regulating fans speed of evaporative cooling apparatus 150 in order to control temperature of cooling loop 175 by varying air flow over evaporative final heat exchanger 152. A pump, motor controller and cooling water loop may also be provided to run water over the exterior of a heat exchanger external to module 135 for additional cooling.

[0039] Controllers 180 may be interfaced via a network with a master controller for which a single dashboard is provided, according to one or more embodiments, which is for displaying and controlling water flow in one or more loops through one or more cooling towers, fan power for air flow across the one or more heat exchangers, one or more cooling towers, and liquid coolant flow in one or more loops for tanks 122.

[0040] A control module may connect into a larger monitoring system, such as a building management system, data center management system, stand alone or other manner of operations.

In some embodiments, the control module controls and monitors external components, such as building water pumps, cooling towers, remote battery backup components, other modules, remote power generators, security, room PDUs, rack PDUs, and other systems.

[0041] Power distribution units 116 mount near the back of the racks 112, under the lid area. In this embodiment, the only in/out cable in the rack may be the main power feed(s) for the PDU.

This arrangement may minimize the amount of space necessary for users to service the back of the rack.

[0042] A single or multiple power feed(s) may feed electronic systems in the module. The power feed may be at voltages such as 208, 240, 277, 480 VAC, or DC voltages in the case of systems using DC battery backup or distribution system. A single feed may go into a power distribution center, or multiple power distribution centers or subpanels, to connect all required loads and/or if redundancy is required.

[0043] The power distribution system may include a transformer that adjusts (and, if required, isolates) the input AC voltage to the required load voltages. The power may be distributed into multiple ports. Each or a group may include a breaker. Each port may connect, for example, to a rack-level power rack distribution device (e.g. outlet strip) attached to the racks and/or the liquid- filled rack system. Alternatively, loads may be wired into the distribution system directly or other manner of connection. In some embodiments, power distribution components may adjust DC power from a distribution voltage to a load voltage.

[0044] Referring now to FIG. IB, computing system 1 10 may include a cooling system 185 for transferring heat from data processing modules 310. The liquid coolant heated by data processing modules 310 is fluidly coupled through suitable piping or lines to a pump 130, which pumps the heated liquid coolant through suitable piping or lines to a heat exchanger 140 associated with a heat-rejection or cooling apparatus 150. In some embodiments, heat exchanger 140 is remotely or distally located from tank 122 and/or computing system 110. Heat exchanger 140 rejects the heat from the incoming heated liquid coolant and fluidly couples the cooled liquid coolant through a return fluid line or piping 170 back into the tank 122. Thus, at least a portion of the liquid coolant completes a fluid circuit through the data processing modules 310 in tank 122, pump 130, heat exchanger 140, and back into tank 122. The heat rejected from the heated liquid coolant through the heat exchanger 140 may then be selectively used by alternative heat rejection or cooling apparatus 150 to dissipate, recover, or beneficially use the rejected heat depending on the different environmental conditions or data processing modules 310 operating conditions to which the system is subject.

[0045] Referring now to FIG. 1C, an embodiment of an alternative cooling system 195 is illustrated for cooling data processing modules 310. Unlike the cooling system 185, heated liquid coolant does not flow outside the tank 122. Instead, one fluid circuit 260 of the flowing liquid coolant is completely internal to the tank 122. A thermal coupling device 280, such as a heat exchanger, is mounted within the tank 122 within the fluid circuit through the data processing modules 310, so that at least a portion of the heated liquid coolant flow exiting the data processing modules flows through the thermal coupling device 280. Cooled liquid coolant exits the coupling device 280 and at least a portion of the cooled dielectric coolant circulates in the internal fluid circuit 260 back through the data processing modules 310.

[0046] Cooling systems 185 (FIG. IB) and 195 (FIG. 1C) include a computer controller 180 with suitable applications software for implementing various embodiments. A detailed description of controller 180 is included in international published patent application WO 2010019517 which is incorporated by reference as if fully set forth herein. In some embodiments, temperatures of operation may be established and maintained as set forth in the WO 2010019517 application.

[0047] Referring now to Figure IB, cooling apparatus 150, which may provides an evaporative final heat exchanger and a motor driven fan for forcing air flow through the final heat exchanger, is located sufficiently far away from tanks 122 to enable adequate heat dissipation at the heat exchanger to cool the heated liquid in loop 175. The resulting heat may be vented to the ambient outside environment. Alternately, the resulting heat may be beneficially used, as described in PCT patent application WO 2013022805, which is incorporated by reference as if fully set forth herein. The cooled liquid is then recirculated through the return pipe in loop 175 to cool the liquid coolant in loop 170 which, in turn, cools the data processing modules 310 in tanks 122. In some embodiments, cooling apparatus 150 is mounted on the exterior top of a container for the computing system.

[0048] Although one cooling apparatus 150 is shown, more than one may be provided in various embodiments. For example, one cooling apparatus 150 may be provided for each bank of tanks 122. Further, cooling loops (for example, cooling loop 175) may be arranged, and each cooling apparatus 150 may be sized, so that a plurality of cooling apparatus 150 may provide backup cooling for one other. Cooling apparatus 150 need not be attached to the shipping container.

[0049] In some embodiments, two or more racks are mounted next to each other, and cable trays extend across two or more racks. The individual racks and cable tray may combine to form one rack. For example, as shown in FIG. 1A, cable tray 1 18 may extend across two of racks 1 12.

[0050] In one embodiment, a single high voltage power feed goes into one or more power distribution units (PDU). A transformer reduces the power from high voltage to a lower voltage (for example, 480 VAC to 208 VAC in the US). The power distribution may include protection against power spike and/or line noise, such as transient voltage surge suppression. The reduced voltage lines (e.g., 208 VAC) may be distributed into multiple different feeds.

[0051] Each feed may be connected to a cabinet distribution unit (CDU) at the rack. Each feed may include a breaker and/or current monitoring. In certain embodiments, one or more of these components are separated out (transformer, transient voltage surge suppression ("TVSS"), and others), rather than housed inside of a power distribution unit.

[0052] In some embodiments, CDUs are included and mounted to each rack. The CDUs may be passive, active, or combination thereof. An active CDU may include power level measurement of voltage and current for each plug (normally where the server plugs into) and the capability to turn each port on or off. Monitoring may be performed by control system for the module. Monitoring may include values for CDU, PDU, or both.

[0053] FIG. 2 illustrates a rack that can be used in a computing module in various embodiments. Rack 222 includes power distribution units 223, liquid coolant inlet port 224, liquid coolant exit port 225, and rack control module 226. [0054] Rack 222 includes a cable management system. Cable management system includes cable management rails 227. Each of cable management rails 227 includes a series of cable guides. The cable guides may be used to guide and support power cables or data cables for each of the servers in rack 232. In some embodiments, cables and cable connector receptacles remain above the surface level of the coolant in the rack.

[0055] FIG. 3 illustrates one embodiment of a computing module including racks with liquid- cooled servers, power distribution components, and an under-floor cooling module. In some embodiments, liquid-cooled servers, power distribution components, and liquid cooling components may ship together as an assembly. Power distribution components and liquid cooling components may be sized for the servers to be operated in the computing module.

[0056] Computing module 230 includes racks 232, cooling component 239, and power distribution components (power distribution components may be under the floor of the aisle). Each of racks 232 includes servers 240. Servers 240 are vertically arranged in racks 232. In some embodiments, any of servers 240 may be removed from rack 232 without removing or disturbing operation of the other servers when the top is removed or folded back. Power distribution units may feed power to cabinet power distribution units on each of racks 232.

[0057] A computing module of several liquid submersion cooling racks and one pump module (such as computing module 230) may maximize efficiency and data center floor space. Each block of four racks may have its own control system that optimizes coolant flow in real-time for the given heat load while monitoring the cooling system across more than twenty-five parameters.

[0058] In some embodiments, OEM and ODM servers are installed vertically into rack 232. Racks 232 may support servers from any manufacturer, in, for example, Standard 19" or Open Compute Standard form factors. Servers may be lowered vertically into the liquid-filled rack. PDUs may be mounted to the front or the back of the rack.

[0059] Cooling component 239 may include coolant pumps, filters, and coolant-to-water heat exchangers. The pump module is responsible for circulating the liquid coolant and drawing heated coolant through heat exchangers to remove server heat from the racks. The pump module may then filter the coolant and return the reduced temperature coolant to the liquid submersion cooling rack. The pump module may establish a stable and uniform cooling environment for servers 240 that is controlled to ±1° C throughout each of racks 232.

[0060] Cooling component 239 may receive power connectivity and water connectivity from the facility. Cooling component 239 may be configured to use virtually any form of water available in a data center facility. In some embodiments, a pump module includes an independent secondary system for backup. If the primary pump should fail, the secondary kicks on instantaneously and cooling will continue undisturbed. Although pump module 239 is shown for illustrative purposes at the end of rows of racks 232, depending on the space requirements of the facility, the pump module may be installed adjacent to the racks, under the floor, or in the data center periphery.

[0061] Coolant may flow in and out of racks 232 at the ends of racks 232 (coolant lines for the rearmost pair of racks may also be coupled to cooling component 239, however, they are omitted from FIG. 3 for clarity.) In one embodiment, the overall path of coolant flow is as shown in FIG. 3. Coolant coming into rack 232 may be distributed across the length of rack 232 in high pressure manifold 245 and then channeled downward (for example, via ducts, barriers, or baffles) to the bottom of rack 232. In some embodiments, high pressure manifold 245 includes a series of nozzles pointing downwardly in the rack. The nozzles may cause circulation through or across each of servers 240.

[0062] Nozzles may be spaced across the length of manifolds 245 and 247, such that a similar circulation is achieved across the length of the manifold (for example, the flow at each server position may be similar). In certain embodiments, a nozzle is provided corresponding to each server location. Nozzles may nevertheless, be spaced at any interval, regular or irregular, to achieve desired flow characteristics (e.g., closer together or farther apart than one nozzle per rack position).

[0063] From the bottom of rack 232, coolant may flow up between or through servers 240 until it reaches the top of servers 240 (and near the surface of liquid coolant), increasing in temperature as heat is transferred from heat producing components on the servers. Some of the coolant may be drawn back toward the high-pressure manifold side of the rack. A portion of the coolant near the surface may be drawn out of rack 232 through suction manifold 247 and returned to a heat exchanger in cooling component 239.

[0064] Downward flow exiting the nozzles under the manifold may draw liquid at the surface of the bath downward (for example, in the direction of arrows shown in FIG. 4). In some embodiments, suction at the surface of the liquid resulting from the high pressure nozzles is such that the level of liquid on the high pressure manifold side is lower than the level of the liquid on the suction side. [0065] In various embodiments, augmentation devices (such as nozzles, pumps, or fans) are provided at multiple locations in a rack. For example, in certain embodiments, nozzles or pumps may be provided at the bottom of each of servers 232.

[0066] In some embodiments, a flow pattern in the rack is such that the fluid flow goes from one side of the top of the rack to other (horizontally). This flow may be formed by fluid rising through the server, coming out of the server, travelling horizontally, and then traveling back downward outside of the server.

[0067] FIG. 4 is a schematic view illustrating circulation at a section of a rack in one embodiment. The pattern in FIG. 4 may be the same at each server position in the rack. Liquid coolant is introduced under pressure through nozzle 251 on the underside of high-pressure manifold 245. High pressure manifold 245 causes circulation of part of the coolant in a generally down-across-up-across pattern within the bath of liquid coolant. The arrows in FIG. 4 illustrate the general pattern of the flow.

[0068] Coolant from nozzle 251 may combine with coolant being drawn from the surface of the coolant and moved near the bottom of the tank, below the bottom of servers 240. Coolant in the bottom of the tank may be forced upwardly through or across servers 240, removing heat from heat-producing components (for example, processors, semiconductor devices) of the servers. Similar to the manner described relative to FIG. 3, near the surface of the coolant, some of the coolant may be drawn back toward the high-pressure manifold side of the rack. A portion of the coolant near the surface may be drawn out of rack 232 through suction manifold 247 and returned to a heat exchanger in cooling component 239.

[0069] Suction manifold is located against the wall of rack 232. All of the coolant at the surface is thus in a contiguous body between suction manifold 247 and the opposing wall of the rack. In some embodiments, the cooling system moves the liquid coolant such that substantially all of the liquid coolant at the surface is in motion (for example, either right toward the wall on the high-pressure manifold side or left toward the wall on the suction manifold side.)

Multi-mode Controller for Liquid-Cooled Rack with Enhanced Computational Output Mode

[0070] In various embodiments, a controller for a liquid-cooled rack (e.g., oil-cooled rack) operates a system in different modes of operation, including one mode optimized for overall power consumption and another mode optimized for computational output. In some embodiments, the controller performs an optimization to enable processors in the servers to operate in an enhanced computational output mode (for example, Turbo mode of certain Intel processors). The controller may enable an enhanced computational output in one or more of the following ways: controlling the coolant temperature at a point known to allow enhanced mode; reading the CPU temperatures through a network or from a signal; and/or reading the CPU state through a network or another signal.

[0071] In some embodiments, a controller optimizes for energy efficiency or for CPU performance. The multi-mode controller allows the end user to optimize the data center depending on the desired effect. For example, if the user is running a data center for retail sales, the demand on the data center at 2 am on Thanksgiving may be low. As such, the operator may optimize for power consumption and decrease the cost to run the datacenter. However, at noon on Black Friday, the data center load will be at peak, therefore at that time the operator may optimize for output. Switching between modes may be carried out automatically (for example, based on pre-determined criteria), manually (for example, by a user on command through a menu selection), or a combination of automatic and manual switching.

[0072] In some embodiments, a master controller optimizes all elements for minimum power consumption of the system while maintaining sufficient cooling. The network controller performs diagnostic testing of each element separately for functionality and reports the functionality back to a single user. This single management point makes the system more reliable and more efficient, since the master controller can obtain maximum efficiency for all components. In some embodiments, control is carried out as described in the '579 Publication.

[0073] In some embodiments, a controller provides diagnostics and controls for a computing module. The control system may optimize coolant flow to provide the most efficient coolant flow at all times for the given heat load. The control module may also initialize the backup system and provide alerts in the event of system downtown or failure. In various embodiments, the controller may provide temperature analysis, pressure and coolant level verification, power consumption, smart monitoring, and diagnostics. Controller outputs may include log files of the above parameters, e-mail and SNMP diagnostic alerts, and hourly status condition updates. This information may also be made available via a network as well as a secure internet portal.

[0074] FIG. 5 is a schematic diagram illustrating a data center with a multi-mode controller with a cooling mode that allows increased computational output. Data center 300 includes racks 302, pump module 304, cooling apparatus 306, and network 308. Racks 302, pump module 304, cooling apparatus 306, and the components thereof may be similar to those described above relative to FIGS. 1A through 1C. [0075] Pump module 304 includes pump 312, filter 314, heat exchanger 316 and multi-mode controller 318. Multi-mode controller 318 may control pump 312, cooling apparatus 306, or both based on information about, and/or received from, components in racks 302. Information may include, for example, CPU type, duty cycle, current CPU state, or current CPU thermal data.

[0076] Racks 302 a row of include servers 320 in a common volume of liquid coolant (e.g., oil). Each of servers 320 may be separately removable from the submerged liquid coolant through the top of a tank. Each of servers includes motherboard 322 and processor 324. Each processor 324 may be a CPU. In some embodiments, motherboard 322 and processor 324 are housed in a server chassis. Pump module 304 may be operated to circulate liquid coolant through each of servers 320 (for example, upwardly in the direction the arrows in racks 302). Racks 302 also include cooling system sensors 326. Cooling system sensors 326 may measure characteristics of liquid coolant in rack 302. Cooling sensors 326 may sense coolant temperature, component temperature, flow rate, pressure, or humidity.

[0077] Data is exchanged between servers and other components in rack 302 and pump module 304 by way of network 308. Network 308 may include any suitable data network or combination of networks that allow the exchange of information among devices in data center 300 or external to data center 300. For example, network 308 may include one or more Local Area Networks (LANs) such as Ethernet networks, as well as Wide Area Networks (WANs), Metropolitan Area Networks (MANs), or other data or telecommunication networks implemented over any suitable medium, such as electrical or optical cable, or via any suitable wireless standard such as IEEE 802.1 1 ("Wi-Fi"), IEEE 802.16 ("WiMax"), etc.

[0078] In some embodiments, messages are set over network 308 using Simple Network Management Protocol (SNMP) data. Any network protocol, however, may be used in various embodiments to send data from, or receive data into, controller 318 and devices and sensors in racks 302.

[0079] Multi-mode controller 318 may control cooling in racks 302 in response to information received from processors, sensors, and other components in racks 302. In some embodiments, one mode prioritizes computational output and another mode that prioritizes resource efficiency (e.g., power consumption, cooling system efficiency). In one embodiment, the controller has one mode optimized for overall power consumption, and another mode optimized for computational output.

[0080] Multi-mode controller 318 may optimize for an enhanced mode. The cooling system may be set to such that processors in the servers can run in an enhanced mode (e.g., Turbo mode for an Intel processor). Multi-mode controller 318 may use data from temperature sensors 326, state information from processor 324, and temperature information from processor 324.

[0081] A controller as described herein may operate an individual server level, a rack-level (for example, controlling cooling for the servers in a single rack), at a multi-server level (controlling cooling for servers in multiple racks), or a data center level. In some embodiments, optimization is performed for multiple servers in a rack. In certain embodiments, the controller adjusts cooling to allow an enhanced computational performance mode based on variations in data center requirements.

[0082] In some embodiments, a temperature of a liquid coolant may be monitored and/or controlled. Methods of monitoring and controlling temperature of the oil may be as described in the '579 Publication"), by Best et al, published June 9, 201 1, which is incorporated by reference in its entirety as if fully set forth herein.

[0083] In FIG. 5, data is as exchanged between racks 302 and the controller by way of network 308. Data may nevertheless in some embodiments be exchanged directly via wire or optical cable between any or all of the racks in a data center and one or more controllers.

[0084] In some embodiments, a controller for liquid-cooled rack of servers optimizes for enhanced computational output mode (e.g., optimizing for Turbo mode on Intel processor). FIG. 6 illustrates one embodiment of optimizing liquid cooling for an enhanced computational output mode for servers operating in a rack. At 340, information is received about one or more servers operating in one or more liquid-cooled racks. The information may include CPU state information (for example, which mode one or more of the server CPUs is operating in, or a level of processing activity). The information may also include thermal data associated with one or more of the CPUs (for example, a signal corresponding to the operating temperature of the CPU). The received in formation may also include characteristics relating to cooling the system. For example, a controller may monitor temperatures and/or flow rates of liquid coolant at one or more locations for a rack of servers.

[0085] At 342, an optimization is performed for an enhanced computational output mode for server processors operating in the liquid-cooled racks. Optimization may include, in various embodiments, linear programming, non-linear programming, finitely terminating algorithms, iterative methods, heuristics, or combinations thereof. In various embodiments, optimization may include determining an operating mode for a cooling system or computing device (actual or desired), determining a setting for one or more cooling parameters (for example, a temperature set point or flow rate), or combinations thereof. In certain embodiments, optimization is multimodal.

[0086] In some embodiments, each of the server processors can dynamically increase processor frequency. An enhanced computational output mode may include the processor operating an increased frequency over its rated frequency. In certain embodiments, optimization for a system is based on whether a server processor is in an enhanced computational output mode (e.g., Intel Turbo mode).

[0087] At 344, the liquid cooling system cools the servers to allow one or more of the server processors to operate in an enhanced computational output mode. For example, in one embodiment, cooling is increased to allow the server processors operate at an increased frequency that is above the rated upper limit for the processors. In one embodiment, the enhanced mode has a clock speed that is higher than the clock speed of one or more other modes for a processor. When conditions change (for example, computational output demand in the data center subsides), the controller may adjust the cooling system to operate in a mode that does not support enhanced computational output.

[0088] In some embodiments, a multi-mode controller switches between a computational performance mode and a resource efficiency mode. FIG. 7 illustrates liquid-cooling servers in a rack with switching between computational performance mode and a resource efficiency mode. At 360, information is received by the multi-mode controller about servers operating in a liquid- cooled rack. The information used by the controller may be similar to that described above relative to FIG. 6.

[0089] At 362, the controller switches between a computational performance mode and a resource efficiency mode. The computational performance mode prioritizes computational performance over resource efficiency. In certain embodiments, the computational performance m ode is an enhanced computational output mode (e.g., in which a processor operates at a higher clock speed). The resource efficiency mode prioritizes resource efficiency over computational performance. In some embodiments, the system performs an optimization for computational output in the computational performance mode, and performs an optimization for efficiency of one or more resources (for example, to minimize power consumption) in the resource efficiency mode.

[0090] In some embodiments, a user controls which mode the controller is in. For example, a user may make a selection of the mode of operation via a drop-down menu. [0091] At 364, servers in the liquid cooled racks are cooled in a selected mode. If computational demand is low, for example, the system may be placed in operation in a resource efficiency mode. If demand rises to above a pre-determined threshold, the system may be switched to a computational performance mode. Switching to the computational performance mode may include increasing cooling (for example, reducing a set point temperature for one or more of the CPUs) such that the servers can operate in the computational performance mode.

[0092] In some embodiments, cooling is controlled to enable processors in one or more servers to operate in an enhanced mode of performance. FIG. 8 illustrates a method of liquid-cooling servers in a rack that includes adjusting cooling parameters to allow a processor to operate in enhanced computational performance mode. At 380, a state of a processor is assessed. The state of the processor may include, for example, one or more enhanced modes of operation.

[0093] At 382, if the state information indicates that the processor is operating in the enhanced mode, cooling to the processor may be maintained at 384. The system may continue to assess the state of the processor (for example, on a periodic basis).

[0094] If the state information for the processor at 382 indicates that the processor is not in the enhanced mode, the system may determine an adjustment to the cooling system that would allow the processor to operate in the enhanced mode at 386.

[0095] At 388, liquid cooling parameters are adjusted to increase cooling. For example, the flow of liquid coolant in a primary loop, secondary loop, or both may be increased. The system may assess the state of the processor after the adjustment.

[0096] In some embodiments, mounting members of a rack are configured to mount the servers closely adjacent to one another in the server rack to restrict the flow of the dielectric liquid coolant between the vertically-oriented servers, such that the flow of the dielectric liquid coolant through the servers is enhanced.

[0097] In some embodiments, flow through the servers in augmented using augmentation, such as nozzles, fans, or pumps. A separate augmentation device may be included on each node, every other node, each row of nodes, or other frequency. The '579 Publication describes apparatus and methods using augmentation devices in various embodiments.

[0098] In some embodiments, liquid coolant may be removed through the top of the rack. Liquid coolant may be reintroduced after having been cooled (for example, by passing the liquid coolant through a heat exchanger outside of the rack. The '579 Publication describes apparatus and methods for removing liquid coolant from the top of a rack in various embodiments. [0099] In various embodiments described herein, computing modules are shown as having four racks. Computing modules may nevertheless in various embodiments have any number of racks. In one embodiment, a computing module has one rack.

[00100] In various embodiments described herein, a system has been described as holding motherboard assemblies in a submersed or partially submersed condition. A system may nevertheless in various embodiments hold other types of circuit board assemblies or components in a partially submersed condition.

[00101] Controllers for systems as described herein may be in a computer system.

Computer systems may, in various embodiments, include components such as a CPU with an associated memory medium such as Compact Disc Read-Only Memory (CD-ROM). The memory medium may store program instructions for computer programs. The program instructions may be executable by the CPU. Computer systems may further include a display device such as monitor, an alphanumeric input device such as keyboard, and a directional input device such as mouse. Computing systems may be operable to execute the computer programs to implement computer- implemented systems and methods. A computer system may allow access to users by way of any browser or operating system.

[00102] Embodiments of a subset or all (and portions or all) of the above may be implemented by program instructions stored in a memory medium or carrier medium and executed by a processor. A memory medium may include any of various types of memory devices or storage devices. The term "memory medium" is intended to include an installation medium, e.g., a Compact Disc Read Only Memory (CD-ROM), floppy disks, or tape device; a computer system memory or random access memory such as Dynamic Random Access Memory (DRAM), Double Data Rate Random Access Memory (DDR RAM), Static Random Access Memory (SRAM), Extended Data Out Random Access Memory (EDO RAM), Rambus Random Access Memory (RAM), etc.; or a non-volatile memory such as a magnetic media, e.g., a hard drive, or optical storage. The memory medium may comprise other types of memory as well, or combinations thereof. In addition, the memory medium may be located in a first computer in which the programs are executed, or may be located in a second different computer that connects to the first computer over a network, such as the Internet. In the latter instance, the second computer may provide program instructions to the first computer for execution. The term "memory medium" may include two or more memory mediums that may reside in different locations, e.g., in different computers that are connected over a network. In some embodiments, a computer system at a respective participant location may include a memory medium(s) on which one or more computer programs or software components according to one embodiment may be stored. For example, the memory medium may store one or more programs that are executable to perform the methods described herein. The memory medium may also store operating system software, as well as other software for operation of the computer system.

[00103] The memory medium may store a software program or programs operable to implement embodiments as described herein. The software program(s) may be implemented in various ways, including, but not limited to, procedure-based techniques, component-based techniques, and/or object-oriented techniques, among others. For example, the software programs may be implemented using ActiveX controls, C++ objects, JavaBeans, Microsoft Foundation Classes (MFC), browser-based applications (e.g., Java applets), traditional programs, or other technologies or methodologies, as desired. A CPU executing code and data from the memory medium may include a means for creating and executing the software program or programs according to the embodiments described herein. [00104] As used herein, the terms "or" is intended to cover a non-exclusive inclusion. That is, "or" includes both meanings of both "or" and "and/or."

[00105] As used herein, "state" means a condition or mode that a device or object is in at a point in time, or for a period of time. For example, a processor may have different states or modes of operation, or operate at different clock speeds at different times.

[00106] As used herein, an "enhanced" mode of operation means a mode in which one or more performance parameters are improved. Examples of operating a processor in an enhanced mode include operating the processor at a frequency higher than its rated frequency (e.g., overclocking). A processor's computational output may be increased when the processor is operating in an enhanced mode.

[00107] As used herein, "optimize" includes finding a best available value or selection of a function based on a set of inputs. In some cases, optimizing includes computing a maximum or minimum value of a function to achieve a desired result. In certain cases, optimization includes determining a mode from among a set of possible choices.

[00108] As used herein, "prioritize" includes treating one thing as more important than one or more other things. In some cases, prioritize means to give one or more operational characteristics of a system greater weight than one or more other operational characteristics of the system.

[00109] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[00110] As used herein, the term "data processing module" generally refers to one or more computing devices running software configured to receive requests, typically over a network. A data processing module may include one or more servers connected to a network and running software configured to receive requests from other computing devices on the network, which may include other servers, and desktop and mobile computing devices, including cellular phones. Such data processing modules typically include one or more processors, memory, input/output connections to a network and other electronic components, and may include specialized computing devices such as blade servers, network routers, data acquisition equipment, disc drive arrays, and other devices commonly associated with data centers.

[00111] As used herein, the term "node" refers to a computing device that can be configured to receive and respond to requests to perform computing operations. A node may have one processor or multiple processors. In some embodiments, a node includes one or more servers and/or one or more data processing modules.

[00112] As used herein, the term "tank" refers to a container with or without a lid, containing a liquid coolant into which one or more data processing modules may be installed.

[00113] As used herein, an "independently operable" device means capable of usefully functioning without regard to an operational status of an adjacent device. As used herein, an "independently operable data processing module" means a data processing module that is capable of usefully functioning to provide data processing services and without regard to an operational status of an adjacent data processing module. Operation of independently operable data processing modules can be influenced (e.g., heated) by one or more adjacent data processing modules, but as used herein, an independently operable data processing module generally functions regardless of whether an adjacent data processing module operates or is operable.

[00114] As used herein, the term "liquid coolant" may be any sufficiently nonconductive liquid such that electrical components (e.g., a motherboard, a memory board, and other electrical or electronic components designed for use in air) continue to reliably function while submerged without significant modification. A suitable liquid coolant is a dielectric liquid coolant, including without limitation vegetable oil, mineral oil, transformer oil, or any liquid coolant have similar features (e.g., a non-flammable, non-toxic liquid with dielectric strength better than or nearly as comparable as air).

[00115] As used herein, "fluid" means either a liquid or a gas, and "cooling fluid" means a gas or liquid coolant typically used for heat-rejection or cooling purposes. As used herein, a liquid coolant is a subset of the universe of cooling fluids, but a cooling fluid may be a dielectric or non-dielectric liquid or gas, such as, for example, a conventional air conditioning refrigerant.

[00116] The flowchart and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and program products, according to various embodiments of the present invention.

[00117] While this specification contains many specifics, these should not be construed as limitations on the scope of the invention or of what can be claimed, but rather as descriptions of features specific to particular implementations of the invention. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a sub combination or variation of a sub combination.

[00118] Similarly, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[00119] Further modifications and alternative embodiments of various aspects of the invention may be apparent to those skilled in the art in view of this description. Accordingly, this description is to be construed as illustrative only and is for the purpose of teaching those skilled in the art the general manner of carrying out the invention. It is to be understood that the forms of the invention shown and described herein are to be taken as embodiments. Elements and materials may be substituted for those illustrated and described herein, parts and processes may be reversed, and certain features of the invention may be utilized independently, all as would be apparent to one skilled in the art after having the benefit of this description of the invention. Methods may be implemented manually, in software, in hardware, or a combination thereof. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Changes may be made in the elements described herein without departing from the spirit and scope of the invention as described in the following claims.