Background

[0001] Advances in semiconductor processing and logic design have permitted an increase in the amount of logic that may be present on integrated circuit devices. As a result, computer system configurations have evolved from a single or multiple integrated circuits in a system to multiple hardware threads, multiple cores, multiple devices, and/or complete systems on individual integrated circuits. Additionally, as the density of integrated circuits has grown, the power requirements for computing systems (from embedded systems to servers) have also escalated. Furthermore, software inefficiencies, and its requirements of hardware, have also caused an increase in computing device energy consumption. In fact, some studies indicate that computing devices consume a sizeable percentage of the entire electricity supply for a country, such as the United States of America. As a result, there is a vital need for energy efficiency and conservation associated with integrated circuits. These needs will increase as servers, desktop computers, notebooks, ultrabooks, tablets, mobile phones, processors, embedded systems, etc. become even more prevalent (from inclusion in the typical computer, automobiles, and televisions to biotechnology).

[0002] In many computing environments, it is an established fact that for much of the time, computing systems such as servers are operating well below their peak performance level. During these periods of low utilization the focus is on saving as much power as possible in order to reduce the energy costs. Power management technologies can deliver significant power savings during periods of low utilization. However any power management technology involves a power/performance tradeoff.

[0003] Due to increasing integration, many processors can include power management technologies which can control up 2/3 rds of total platform power. In many cases these technologies are controlled by a power control unit (PCU) in the processor. Each power management feature is specifically tuned in design to achieve an optimal power/performance tradeoff. At the time of tuning, there is little knowledge of the actual workload and usage pattern for the system in the field. Given this lack of knowledge, the tuning process is conservative and is necessarily biased towards losing as little performance as possible. This approach prevents significant power savings for an end user who is willing to tolerate more performance loss in return for power savings.

[0004] Thus typically power management features are statically tuned to tolerate very little performance loss. This results in several negative downsides. First, at low utilizations where an end user can tolerate high performance loss, available power savings are not realized. Second, an end user typically has no choice regarding power/performance tradeoffs, other than default profiles provided by an operating system (OS). Given the complexity involved in tuning power management features, end users rarely venture into tuning individual features for their target usage, and thus the potential benefit of the features are often not realized.

Brief Description of the Drawings

[0005] FIG. 1 is a block diagram of an architecture of a tuning circuit for a tunable power performance loadline technique in accordance with one embodiment of the present invention.

[0006] FIG. 2 is a block diagram of a tuning table in accordance with an embodiment of the present invention.

[0007] FIG. 3 is a flow diagram of a method in accordance with an embodiment of the present invention.

[0008] FIG. 4 is a block diagram of a processor in accordance with an embodiment of the present invention.

[0009] FIG. 5 is a block diagram of a multi-domain processor in accordance with another embodiment of the present invention.

[0010] FIG. 6 is a block diagram of a system in accordance with an embodiment of the present invention.

[0011] FIG. 7 is a block diagram of a multiprocessor system with a point-to-point (PtP) interconnect in accordance with one embodiment of the present invention. [0012] FIG. 8 is a block diagram of a partially connected quad processor system in accordance with one embodiment of the present invention.

Detailed Description

[0013] Embodiments provide a so-called energy performance bias (EPB) as an architectural feature. Control of this parameter allows for a simple high level input from an end user to indicate a power/performance tradeoff preference from the end user. This input can be used to provide multiple tuning levels with different points of power and performance tradeoff. By associating this energy performance bias with direct user input, embodiments enable the end user to directly control

power/performance tradeoff in a simple manner. As used herein the terms "end user" or "user" are comprehended to include computer users of varying degrees, including technical and non-technical users, information technology (IT) personnel, data center personnel and so forth.

[0014] Thus instead of providing complete tuning flexibility for each power management technology and allowing an end user to tune each feature, a single input can be provided by the user to control these different features. The EPB value may thus correspond to a single input value to control a plurality of power management features. Furthermore, understand that the provision of the EPB value can be from a variety of external entities including but not limited to an operating system (OS), a basic input/output system (BIOS), an external embedded controller of a platform such as a baseboard management controller (BMC), a data center central management software and communicated via a network and a node manager device or so forth to a platform, among others, automatically or via a user. And in some embodiments, the end user may be prevented from such individual control. As such, the inherent difficulty in exposing all of a large number of power management features to the end user can be avoided, particularly as most end users have little or no knowledge as to how to tune such individual features. In practice, 90% of server users never change the default power management configuration for the server, and a very similar dilemma exists in client usage models as well.

[0015] Table 1 below shows a list of power management features available in different processor models and which can be controlled globally using one or more EPB inputs in accordance with an embodiment of the present invention. It can be seen from Table 1 that the number features is disparate and large. This list continues to grow with each generation of processor. As seen, power management features can be performed within a processor itself, a memory or memory interconnect, or other interconnect structures. Embodiments can provide for global control of a plurality of power management features of a processor including, in some embodiments, those shown in Table 1. However, understand that this listing is exemplary only, and other power management features can be controlled using an EPB value in accordance with an embodiment of the present invention.

Table 1

[0016] A power/performance loadline is a well accepted representation of power consumption of a system at different utilizations. This loadline represents the power consumed at each delivered performance level and takes into account the performance impact of a power management feature. Via an energy performance bias setting in accordance with an embodiment of the present invention, tuning of a power performance loadline can be realized. This setting or slider thus allows an end user to choose a range of choices between performance-oriented tuning settings and power saving-oriented tuning settings. Each EPB value can be mapped to a corresponding level of tradeoff between power and performance.

[0017] Although the following embodiments are described with reference to energy conservation and energy efficiency in specific integrated circuits, such as in computing platforms or processors, other embodiments are applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of embodiments described herein may be applied to other types of circuits or semiconductor devices that may also benefit from better energy efficiency and energy conservation. For example, the disclosed embodiments are not limited to any particular type of computer systems, and may be also used in other devices, such as handheld devices, systems on chip (SoCs), and embedded applications. Some examples of handheld devices include cellular phones, Internet protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically include a microcontroller, a digital signal processor (DSP), network computers (NetPC), set-top boxes, network hubs, wide area network (WAN) switches, or any other system that can perform the functions and operations taught below. Moreover, the apparatus', methods, and systems described herein are not limited to physical computing devices, but may also relate to software optimizations for energy conservation and efficiency. As will become readily apparent in the description below, the embodiments of methods, apparatus', and systems described herein (whether in reference to hardware, firmware, software, or a combination thereof) are vital to a 'green technology' future, such as for power conservation and energy efficiency in products that encompass a large portion of the US economy.

[0018] Referring now to FIG. 1 , shown is a block diagram of an architecture of a tuning circuit for a tunable power performance loadline technique in accordance with an embodiment of the present invention. In the embodiment of FIG. 1 , each logical thread can provide an input, namely an energy performance bias value, referred to herein as an EPB value. Note that this logical thread can correspond to a logical processor or other execution entity. The policy choice between performance and power can be provided per logical thread via a register or other storage, e.g., a 4-bit input register. In this example, 16 levels of power and performance tradeoff can be realized, in such an embodiment. According to one convention, a value of zero represents a very low (or even zero) tolerance for performance loss while saving power. On the other hand a value of 15 represents a high tolerance for performance loss and saving as much power as possible. Note that this EPB input can be provided by an operating system (OS) or by system basic input/output system (BIOS), or even by a user level application directly or indirectly.

[0019] In one usage model, where an operating system supports multiple power/performance profiles, the OS configures the energy performance bias register to an appropriate value based the power/performance profile chosen by an administrator which is one example of an end user. To this end, embodiments may provide a user interface to request and receive user selection of a given EPB value. This user interface functionality can be accommodated within BIOS or an OS or other custom system level software. In an advanced usage model, a data center administrator or other information technology personnel can provide the input based on time of day policies. For example, during times of peak usage, the administrator can choose to configure a value of 0 while during non-peak hours the administrator can choose to configure the server to a value of 15 to save as much power as possible.

[0020] In another usage model for advanced application level control that supports the option for application monitoring of the level of performance that the application expects to receive under a level of service agreement, the application may tune through an OS service the level of performance loss that is acceptable by the application under its current level of operation.

[0021] As seen in FIG. 1 , a power management tuning circuit 100 can be used to receive incoming EPB values from multiple cores of a multicore processor and use these values to determine an appropriate tradeoff between power and performance policies and provide settings to the agents responsible for controlling various power management features of a system including the processor. In one embodiment, circuit 100 can be implemented in logic of a power control unit (PCU) of a processor, although embodiments can be implemented in other hardware or software. As seen, circuit 100 includes a sampler 110 that can receive incoming EPB values from the various cores. Note that these values received can be from logical threads, in one embodiment, which provide a value based on its programming. Although described in the embodiment of FIG. 1 as being received from cores, in various embodiments, EPB values can be per core's thread, per core or per package that includes a number of cores, or other granularity. By these multiple inputs, thread specific optimizations can be made based on a workload being executed. Received EPB values can be stored in temporary storage, e.g., registers within EPB sampler 110 at a sampling point, which may be according to a given interval. In one embodiment, the EPB inputs can be sampled at a predetermined interval, e.g., once every 1 millisecond (ms). The samples obtained from the various cores can be provided to an EPB combiner 120 that can generate a single value from these multiple values. For power management optimizations that are not thread specific (for example, of a memory controller, interconnect, memory or so forth), a mechanism can be provided to resolve the inputs from the multiple threads. In one embodiment, this resolution can be via a policy of favoring performance. To this end, a min function can be performed on all thread's inputs and the resulting value represents the most performance favoring EPB input. This value can then be used to determine the processor power/performance policy level. However, understand the scope of the present invention is not limited in this way. As an example instead of a minimum function, an averaging between the individual thread values can be performed.

Another example is to make the minimum function per control power domain that may hold several sets of cores or threads.

[0022] Still referring to FIG. 1 , the single value corresponding to, e.g., the minimum value of the EPB value, can be provided to a bin generator 130. Note that in various embodiments this bin generator may be optional to provide for finer-grained control. That is, bin generator 130 may generalize the tuning by taking the received EPB value and placing it into one of multiple bins that correspond to multiple coarser- grained levels of control such that the EPB value thus determined can be bucketed into an internal value. As described above, with a 4-bit EPB value, 16 levels can be provided. However, some systems may not be sufficiently complex with regard to the number of power management features, and thus the determined EPB value can be corresponded to one of N bins less than 16 individual levels. In one embodiment, to simplify and minimize the amount of tuning for a processor, the EPB value can be allocated to one of N buckets or bins. N can equal four, in one embodiment. In this example, each bucket represents a continuous range that is 1/4th of total power management range. These buckets may directly map to power/performance profiles supported by certain operating systems. Table 2 shows the tuning criterion for four buckets in accordance with one embodiment of the present invention. However, understand that different tuning criteria can be used in other embodiments.

Table 2

[0023] Still referring to FIG. 1 , the bin value thus determined can be provided to a change detector 40 that can determine whether the value of the bin has changed since the last analysis interval. If not, no further operation of circuit 100 may be performed for that analysis interval. If instead the value has changed, this new value can be used to access a power/performance table 150. In various embodiments, this table can be stored in a processor, e.g., in non-volatile storage of the PCU. The table can be pre-tuned to provide user transparent tuning settings that correspond to a selected power/performance profile. In general, table 150 can include one or more entries for each power management feature. Each entry can further include a plurality of fields, each associated with one of the buckets (or indicated EPB values) to store a parameter for use in that feature or setting of a feature. These parameters may be values used for a certain power management feature, or can be an address to code that is executed for a given setting.

[0024] Referring now to FIG. 2, shown is a block diagram of a tuning table in accordance with an embodiment of the present invention. As shown in FIG. 2, table 150 includes multiple entries 151 ₀ -151 _n - As seen, each of these entries can be associated with a given power management feature setting. Some features may have more than one setting associated with them. For other power management features, only an entry for a single setting is provided. [0025] As further seen in table 150, each entry includes a plurality of fields. In the embodiment shown in FIG. 2, fields 152, 154, 156 and 158 are provided. Each field is associated with a given one of the multiple bins as discussed above, namely a given policy. In other embodiments, each field can be associated with a given EPB value. Thus for each power management feature setting, every bin can have a value in its corresponding field. In specific implementations, each field may include a value that can be used to control enablement of a power management feature, provide a configuration register value or so forth. For some features and settings, it is possible that multiple bins may have the same value, however the scope of the present invention is not limited in this aspect.

[0026] Based on the bin value, a respective column of the table can be accessed and all of the fields of the accessed column can be read out, e.g., as a single vector read or iteratively via a field of an entry per cycle. As will be discussed below, these values output from the table can be provided to a change detector 160. Although shown at this high level in the embodiment of FIG. 3, understand the scope of the present invention is not limited in this regard.

[0027] In some embodiments, in addition to the EPB input, a workload configuration input can be provided. To this end, the table can have 3 dimensions such that based on the workload configuration input, a different set of entries for the defined power management features can be accessed, as different values may be present in the table for different workload configurations. By this workload configuration input a vertical user having an understanding and control of the exact workloads running on their systems can benefit from well-tuned settings. As examples, a user can configure a workload input as non-uniform memory architecture (NUMA), uniform memory architecture (UMA), input/output (I/O) intensive, etc. This input allows for choosing tuning settings that favor a specific workload pattern. For example, if the workload is NUMA, aggressive settings can be applied to off-chip interconnects such as Intel® Quick Path Interconnect (QPI) links to save as much power possible while causing very little performance impact, as off-chip accesses can be expected to be low. Thus for embodiments in which a workload configuration input is provided, it can be used as an additional input to access the table. [0028] Once a new set of configurations is determined from access to the tuning table, the target features can be updated. There are two classes of configurations. For features like a power C-state auto demotion and turbo upside clipping, the configurations can be internal to power management code, such as present in firmware of the PCU. In this case the update can be effected by loading new values into an internal feature specific data structure such as one or more configuration registers. In a second class of configuration, the configuration value is specific to an entity that resides outside the PCU. For example, a clock related configuration can be implemented in the memory controller. In this case, the PCU can initiate a series of writes to target configuration space to update the settings. In order to minimize the number of writes, each configuration value can be compared to the previous value and a write is issued only if the actual configuration value changes.

[0029] Accordingly, with further reference back to FIG. 1 the identified field of the multiple entries of table 150 can be output to a change detector 160 to determine whether the values have changed from the previous output. Note that for certain power management features, settings may not change when a bin changes level. Thus change detector 160 acts as a filter to filter from sending messages for updates when in fact no update is needed. If a change in fact has occurred, it may be sent from change detector 160 to the indicated destination. As described, many power management features can be controlled using the PCU, and accordingly updated settings can be sent to a PCU updater 170 which thus may update the settings, e.g., by updating values in configuration registers, other storages, enabling or disabling certain power management features, changing variables used with regard to duration of power management features or so forth.

[0030] If instead the power management feature is for a non-PCU controlled feature, update messages can be sent to the destination, e.g., via one or more write messages including the information. Although shown with this particular

implementation in the embodiment of FIG. 1 , understand the scope of the present invention is not limited in this regard.

[0031] Referring now to FIG. 3, shown is a flow diagram of a method in accordance with an embodiment of the present invention. As seen, method 200 can be performed, e.g., using policy management logic of a PCU or other power controller. Method 200 begins by receiving an energy performance bias value from multiple threads (block 210). As discussed above, these threads can be logical threads or equivalently the EPB values can be received from the cores themselves. Next, a global EPB value can be determined from the received EPB values (block 220). For example, a min function can be performed or an average obtained from these multiple values. Then, this global EPB value can be placed into a bin of multiple performance level bins (block 230). Such binning provides a smoothing function, although it may not be present in some implementations where very fine-grained control of power and performance tradeoffs is desired.

[0032] Still referring to FIG. 3, control next passes to diamond 240 where it can be determined whether a bin change has occurred since the last evaluation interval. If not, method 200 concludes for that given evaluation interval which can be, e.g., on the order of approximately 1 ms.

[0033] If instead a bin change has occurred, control passes to block 250 where a power-performance table can be accessed based on the bin. This table access thus can be used to read out for the given bin, and for each of multiple power

management features, one or more settings or other values used to control the power management feature. This reading and the updating process can be performed iteratively for each feature. Thus it can be determined whether an additional power management features present in the table (diamond 255). If not, method 200 terminates for that evaluation interval. If another feature is present, it can be determined whether a change in the feature settings has occurred since the last read out from the table (diamond 260). If not control passes back to diamond 255. If instead a feature change has occurred, control passes to diamond 270 where it can be determined whether the feature is a PCU-controlled feature. If so, one or more settings for the feature can be updated in the PCU (block 290). Such updating can be via configuration register updates or so forth. Otherwise if the updates are for a power management feature controlled by an external agent, control passes to block 280 where one or more messages, e.g., write messages can be sent to the destination agent to update the settings accordingly. For example, writes can be issued to target using a given message channel (e.g., QPI, integrated memory controller, a Peripheral Component Interconnect Express (PCI Express™ (PCIe™)) link or so forth). Although shown with this particular implementation in the embodiment of FIG. 3, understand the scope of the present invention is not limited in this regard.

[0034] In one embodiment, a tuning methodology for generating to power- performance table may include the following. First, each individual power management feature is tuned separately while the other features are turned off. This tuning includes generating a power/performance profile across a range of workloads for each feature. As an example various benchmark workloads can be executed to generate the profile. Second, multiple features can be enabled to a profile whether the power/performance tradeoff met the tuning goals for overall power/performance profile. Then individual feature tunings can be adjusted to meet overall

power/performance profile goals. The above steps can be repeated until overall goals are reached.

[0035] By default the operating system may populate the EPB input with a profile chosen by the end user. This tuning can be achieved via an iterative process in which different EPB values are input and workloads run with each setting. As the EPB input is tuned towards power savings, a user may notice a continuous degradation of response time (performance). Once a desired response time is met, the EPB value may correspond to a desired maximum power savings for the user's specific usage. This in essence is the goal of the tunable power performance loadline, as each end user can tune the power/performance loadline to his specific usage.

[0036] Embodiments can be implemented in processors for various markets including server processors, desktop processors, mobile processors, and so forth. Referring now to FIG. 4, shown is a block diagram of a processor in accordance with an embodiment of the present invention. As shown in FIG. 4, processor 300 may be a multicore processor including a plurality of cores 310 _a - 310 _n . In one embodiment, each such core may be of an independent power domain and can be configured to operate at an independent voltage and/or frequency, and to enter turbo mode when available headroom exists. The various cores may be coupled via an interconnect 315 to a system agent or uncore 320 that includes various components. As seen, the uncore 320 may include a shared cache 330 which may be a last level cache. In addition, the uncore may include an integrated memory controller 340, various interfaces 350 and a power control unit 355.

[0037] In various embodiments, power control unit 355 may include a policy tuning logic 359, which may be a logic to perform dynamic control of power management settings based on an input EPB value. As further seen, a tuning table 357 can be present to store power management feature settings. This tuning table generated, e.g., during design of the processor, based on benchmark workload testing on the processor may be used to determine appropriate settings for the different power management policies that map to EPB values or bins.

[0038] With further reference to FIG. 4, processor 300 may communicate with a system memory 360, e.g., via a memory bus. In addition, by interfaces 350, connection can be made to various off-chip components such as peripheral devices, mass storage and so forth. While shown with this particular implementation in the embodiment of FIG. 4, the scope of the present invention is not limited in this regard.

[0039] Referring now to FIG. 5, shown is a block diagram of a multi-domain processor in accordance with another embodiment of the present invention. As shown in the embodiment of FIG. 5, processor 400 includes multiple domains.

Specifically, a core domain 410 can include a plurality of cores 410 ₀ -410n, a graphics domain 420 can include one or more graphics engines, and a system agent domain 450 may further be present. In various embodiments, system agent domain 450 may execute at a fixed frequency and may remain powered on at all times to handle power control events and power management such that domains 410 and 420 can be controlled to dynamically enter into and exit low power states. Each of domains 410 and 420 may operate at different voltage and/or power. Note that while only shown with three domains, understand the scope of the present invention is not limited in this regard and additional domains can be present in other embodiments. For example, multiple core domains may be present each including at least one core. [0040] In general, each core 410 may further include low level caches in addition to various execution units and additional processing elements. In turn, the various cores may be coupled to each other and to a shared cache memory formed of a plurality of units of a last level cache (LLC) 440 ₀ - 440 _n . In various embodiments, LLC 440 may be shared amongst the cores and the graphics engine, as well as various media processing circuitry. As seen, a ring interconnect 430 thus couples the cores together, and provides interconnection between the cores, graphics domain 420 and system agent circuitry 450. In one embodiment, interconnect 430 can be part of the core domain. However in other embodiments the ring interconnect can be of its own domain.

[0041] As further seen, system agent domain 450 may include a display controller 452 which may provide control of and an interface to an associated display. As further seen, system agent domain 450 may include a power control unit 455 which can include a policy tuning logic 459 in accordance with an embodiment of the present invention to dynamically control power management settings obtained from a tuning table 457 based on one or more EPB values. In various embodiments, this logic may execute the algorithm described above in FIG. 3.

[0042] As further seen in FIG. 5, processor 400 can further include an integrated memory controller (IMC) 470 that can provide for an interface to a system memory, such as a dynamic random access memory (DRAM). Multiple interfaces 480 ₀ - 480 _n may be present to enable interconnection between the processor and other circuitry. For example, in one embodiment at least one direct media interface (DMI) interface may be provided as well as one or more Peripheral Component Interconnect Express (PCI Express™ (PCIe™)) interfaces. Still further, to provide for

communications between other agents such as additional processors or other circuitry, one or more interfaces in accordance with a Intel® Quick Path Interconnect (QPI) protocol may also be provided. Although shown at this high level in the embodiment of FIG. 5, understand the scope of the present invention is not limited in this regard.

[0043] Embodiments may be implemented in many different system types.

Referring now to FIG. 6, shown is a block diagram of a system in accordance with an embodiment of the present invention. As shown in FIG. 6, multiprocessor system 500 is a point-to-point interconnect system, and includes a first processor 570 and a second processor 580 coupled via a point-to-point interconnect 550. As shown in FIG. 6, each of processors 570 and 580 may be multicore processors, including first and second processor cores (i.e., processor cores 574a and 574b and processor cores 584a and 584b), although potentially many more cores may be present in the processors. Each of the processors can include a PCU or other logic to perform dynamic control of power management settings based on a resolved power management policy, as described herein.

[0044] Still referring to FIG. 6, first processor 570 further includes a memory controller hub (MCH) 572 and point-to-point (P-P) interfaces 576 and 578. Similarly, second processor 580 includes a MCH 582 and P-P interfaces 586 and 588. As shown in FIG. 6, MCH's 572 and 582 couple the processors to respective memories, namely a memory 532 and a memory 534, which may be portions of system memory (e.g., DRAM) locally attached to the respective processors. First processor 570 and second processor 580 may be coupled to a chipset 590 via P-P interconnects 552 and 554, respectively. As shown in FIG. 6, chipset 590 includes P-P interfaces 594 and 598.

[0045] Furthermore, chipset 590 includes an interface 592 to couple chipset 590 with a high performance graphics engine 538, by a P-P interconnect 539. In turn, chipset 590 may be coupled to a first bus 516 via an interface 596. As shown in FIG. 6, various input/output (I/O) devices 514 may be coupled to first bus 516, along with a bus bridge 518 which couples first bus 516 to a second bus 520. Various devices may be coupled to second bus 520 including, for example, a keyboard/mouse 522, communication devices 526 and a data storage unit 528 such as a disk drive or other mass storage device which may include code 530, in one embodiment. Further, an audio I/O 524 may be coupled to second bus 520. Embodiments can be

incorporated into other types of systems including mobile devices such as a smart cellular telephone, tablet computer, netbook, ultrabook, or so forth.

[0046] FIG. 7 is a block diagram of a system 600 coupled with point-to-point (PtP) system interconnects in accordance with a given cache coherence protocol using QPI links as the system interconnect. In the embodiment shown, each processor 610 is coupled to two PtP links 625 and includes one instance of an integrated memory controller 615 that in turn is coupled to a corresponding local portion of a system memory 620. Each processor can perform power management techniques using optimized settings obtained from a tuning table in accordance with an embodiment of the present invention. The processors are connected to an input/output hub (IOH) 630 using one link and the remaining link is used to connect the two processors.

[0047] Referring now to FIG. 8, shown is a block diagram of a system in accordance with another embodiment of the present invention. As shown in FIG. 8, system 700 may be a partially connected quad processor system in which each processor 710 (each of which may be multicore multi-domain processors) is coupled to each other processor via a PtP link and is coupled to a local portion of memory (e.g., dynamic random access memory (DRAM)) 720 via a memory interconnect coupled to an integrated memory controller 715 of the corresponding processor. In the partially connected system of FIG. 8, note the presence of two lOHs 730 and 740 such that processors 710 ₀ and 710i are directly coupled to IOH 730 and similarly processors 710 ₂ and 710 ₃ are directly coupled to IOH 740.

[0048] Embodiments may be implemented in code and may be stored on a non- transitory storage medium having stored thereon instructions which can be used to program a system to perform the instructions. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, solid state drives (SSDs), compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs), static random access memories (SRAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions.

[0049] While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.