TECHNICAL FIELD

[0001] The disclosure herein relates to memory systems, and more specifically to buffered memory modules and methods.

BACKGROUND

[0002] Memory modules come in a variety of configurations depending on a given application and desired storage capacity. For high-capacity memory modules that utilize significant numbers of memory devices, a buffered architecture is often employed. The buffered approach buffers data transferred between the memory devices and a memory controller, thus reducing loading of a data bus to the relatively few buffer devices, rather than the high number of memory devices. Address and control busses are also typically buffered in these systems.

[0003] Conventional buffered memory systems employ primary data paths between the memory controller and buffer(s), and secondary data paths between the buffer(s) and the memory devices. The primary and secondary paths transfer data at the same rate. While this may work adequately for some applications, there may be unused bandwidth if both the primary and secondary buses are locked to the same transfer rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Embodiments of the disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0005] FIG. 1 illustrates one embodiment of a memory system that employs a memory controller and multiple memory modules.

[0006] FIG. 2 illustrates one embodiment of a buffered memory module that may be used with the memory system of FIG. 1.

[0007] FIG. 3 illustrates a timing diagram of data and command signals associated with the buffered memory module of FIG. 2.

[0008] FIG. 4 illustrates a flowchart of steps for one embodiment of a method of operating the buffered memory module of Fig. 2.

[0009] FIGs. 5 A - 5C illustrate various routing and module configurations for a point-to- point memory system topology employing the buffered memory module of Fig. 2.

[0010] FIG. 6 illustrates further detail for one specific embodiment of a buffered memory module configured for a point-to-point system topology.

[0011] FIG. 7 illustrates a timing diagram for data and command signals associated with the buffered memory module of FIG. 6. DETAILED DESCRIPTION

[0012] Embodiments of integrated circuits, memory modules and associated methods are disclosed herein. In one embodiment, a memory module includes a module substrate having first and second memory devices. Buffer circuitry disposed on the substrate couples to the first and second memory devices via respective first and second secondary interfaces. The buffer circuitry includes a primary signaling interface for coupling to a group of signaling links associated with a memory controller. The primary signaling interface operates at a primary signaling rate and the first and second secondary data interfaces operate at a secondary signaling rate. During a first mode of operation, the primary interface signaling rate is at least twice the secondary signaling rate. A first time interval associated with a transfer of first column data via the first secondary interface temporally overlaps a second time interval involving second column data transferred via the second secondary interface.

[0013] In a further embodiment, a memory system is disclosed. The memory system includes a system substrate, a first buffered memory module coupled to the system substrate and a second buffered memory module coupled to the system substrate. Each of the first and second buffered memory modules include a module substrate and first and second memory devices disposed on the module substrate. Buffer circuitry disposed on the substrate couples to the first and second memory devices via respective first and second secondary data interfaces. The buffer circuitry includes a primary data interface for coupling to a group of data links formed on the system substrate. The primary data interface operates at a primary data rate and the first and second secondary data interfaces operate at a secondary data rate. During a first mode of operation, the primary interface data rate is at least twice the secondary data rate, and first column data transferred via the first secondary interface overlaps second column data transferred via the second secondary interface.

[0014] In yet another embodiment, a memory controller is disclosed. The memory controller includes a controller primary signaling interface for coupling to at least one buffered memory module via a group of data links. Command generation circuitry dispatches command signals to the least one buffered memory module. During a first mode of operation, a signaling rate of the controller primary signaling interface is twice a rate of the first and second secondary interfaces.

[0015] Referring to Figure 1, one embodiment of a buffered memory architecture, generally designated 100, employs multiple memory modules 102 and 104 coupled to a primary bus 106 that transfers data, command and address signals. Memory control circuitry in the form of a memory controller 108 is also coupled to the primary bus 106 to direct data transfers to and from the modules 102 and 104. The memory control circuitry may include, e.g., a discrete memory controller separate from a requestor integrated circuit (IC). For one embodiment, the memory controller is an IC chip that controls dynamic random access memory (DRAM) devices disposed on the modules 102 and 104. In one embodiment, the memory controller employs EDC circuitry 105 to perform error detection and/or correction on write data and/or read data.

[0016] Further referring to Figure 1, the embodiment shown illustrates a multi-drop bus, where the primary bus 106 is shared by the modules 102 and 104. Point-to-point topologies are also envisioned, illustrated in FIGs 5-7, and described below. Each module includes buffer circuitry 120, described more fully below, and memory devices 124A - 124N and 126A - 126N. For one embodiment, the memory devices are DRAM devices. The buffer circuit 120 isolates the memory devices from the primary bus 106 for each module. Thus, instead of the multiple memory devices for each module loading the bus, only the buffer circuits load the bus. This minimizes capacitive loading on the shared primary bus, among other things. As explained more fully below, the buffer circuit 120 may also support error detection and correction (EDC) coding to detect and/or correct errors in data transferred to and from the modules.

[0017] Figure 2 illustrates additional detail for one embodiment of the memory architecture of FIG. 1, including memory modules 102, 104 and memory controller 108. Each module 102 and 104 includes a substrate 202 having a front face that mounts a plurality of memory devices 206. In some embodiments, the memory devices may organized as "ranks." Typically, a rank corresponds to a set of otherwise separately addressable memory devices used for a single memory access. Only a portion of a single rank is shown in FIG. 2 for each module.

[0018] Each memory device 206 may be realized as a single memory chip, or packaged to include plural memory die in a stacked configuration. In some embodiments, an additional set of memory devices may be mounted on the back face of the module, and organized as an additional rank. The memory devices each may include multiple "banks," and all of the banks organized within a given rank of devices. Each bank may be independently accessed via a given bank address.

[0019] Further referring to Figure 2, for one embodiment, a pair of memory devices (or packaged groups of devices) 210 couple to a data buffer 212 via corresponding secondary data busses 216 and 218 that transfer data at a secondary data rate. The data buffer 212, in turn, interfaces with the memory controller 108 through a pair of primary bus data paths 220 and 222 that transfer data at a primary data rate. Each data buffer 212, for one embodiment, manages buffer operations for two "nibbles" of data, with each nibble corresponding to four data paths or "DQs." For one embodiment, a differential strobe signal accompanies each nibble on two strobe paths. As is described more fully below, in one mode, the primary data rate may operate higher than the secondary data rate, and the memory architecture is configured to support the difference in rates. As a result, inexpensive memory devices that operate at slower speeds may be employed by the modules without impacting the performance of the memory system, resulting in reduced hardware costs.

[0020] With continued reference to FIG. 2, a command/address (C/A) buffer 223 interfaces with the memory controller 108 through a primary command bus 224, and routes command signals along a secondary command bus 226, to the memory devices 206 in, for example, a "fly-by" manner. A unidirectional buffer communications (BCOM) bus 228 links the C/A buffer circuit 223 to the data buffers 212. For one embodiment, each memory module employs a single C/A buffer 223 embodied as a first integrated circuit (IC) buffer chip, and plural data buffer IC chips 212. The plural data buffer chips may be configured to each support a x8 (byte or two nibble)) portion of the primary and secondary data busses, and corresponding to one or more dedicated groups 207 and 209 of the memory devices 206. While separate buffer circuits are shown in Figure 1 for the data and CA buffer circuits, the buffer circuitry may take many different forms, including, for example, a single buffer chip for each memory module that includes both the C/A and data buffer circuitry.

[0021] Further referring to FIG. 2, to support a mode where data transfers with the primary and secondary busses operate at different data rates, each data buffer 212 is configured to pass two nibbles of data simultaneously to two groups 207 and 209 of memory devices on a given module. As a result, data transfers for an entire module may be supported with half the primary data paths. Additionally, the memory controller 108 is configured to dispatch column data commands and data to at least two memory modules such that column data transactions overlap in time between the different modules 102 And 104.

[0022] For example, and with continued reference to FIG. 2, data transferred along a given nibble (4 paths), such as at 220, of the primary data bus is routed to a corresponding data buffer 212 for a given module 102. To support the increased data rate of the primary bus, the data from the nibble is split and transferred along the secondary bus paths 216 and 218 between groups of memory devices at a rate that is half the rate of the primary bus. Other transfer rates are possible, depending on the application. In this manner, data for separate transactions that are, for example, interleaved along the nibble, may be aggregated in a given group of memory devices at a data rate that is half the data rate of the primary bus. For some embodiments, the data associated with each nibble may not be when transferred along the primary bus.

[0023] In other embodiments, the number of modules may vary depending on the application. To support different data rates between the primary and the secondary interfaces, the transferred data may be organized in various ways for transfer along the primary data bus. This allows for effective steering of the data in the multiple modules to effect the reduced data rates along the secondary paths, and the concurrent accesses to the memory devices of the various modules.

[0024] FIG. 3 illustrates a timing chart associated with a set of write operations to more fully explain a mode of operation where the primary and secondary busses operate at different data rates. The waveform CK represents a timing reference for the transfer operations, where every four intervals of the clock corresponds to a column cycle time tCC. A tCC is a timing parameter that represents the minimum time necessary to carry out a column access. [0025] Further referring to FIG. 3, in transferring write data, the memory controller 108 dispatches respective column access commands CAX and CAY, at 302 and 304, to the modules 102 and 104. At 306, a first set of EDC-coded column write data is transferred along a 4-bit (nibble) set of the primary data paths (DQ-nibU) at a first data rate, such as 3.2 Gb/s. The first set of column write data includes first column data that is interleaved with second column data. Each of the first and second column data exhibit bit rates that are half the primary data rate. A second set of column write data begins a data transfer, at 308, along a second 4-bit (nibble) set of the primary data paths (DQ-nibV), at the 3.2Gb/s data rate.

[0026] With continued reference to FIG. 3, at 310, the column access command CAX is re- driven across the secondary CA bus 226 by the CA buffer 222. Following a buffer delay, the interleaved first set of column write data is de- interleaved by the data buffer 212, and the first and second column data transferred along respective secondary busses (DQ-modX-nibU),and (DQ-modX-nibV), at 312 and 314 for writing to separate groups 207 and 209 of memory devices 206 in the first memory module 102 as separate threads. This is carried out at a second data rate that is half the primary data rate, such as 1.6Gb/s. While the first memory module 102 is carrying out the first column data transfer, the CA buffer on the second memory module 104 re-drives the second column access command CAY, at 316, with the second set of column write data being transferred along the secondary busses of the second memory module 104, at 318 and 320. This is carried out at a data rate that is half the primary bus data rate. The respective first and second column data transfers overlap in time, over a duration tovERLAP, ^as shown at 322, such that at least a portion of the data transfer operations are carried out concurrently. [0027] The error encoding carried out on the transferred data may be generated in accordance with one of many acceptable EDC algorithms including, for example,

straightforward single-bit Hamming codes, to more sophisticated high-speed BCH (Bose, Ray- Chaudhuri and Hocquenghem) codes. Other EDC codes, such as "Chipkill" codes, Reed- Solomon codes, turbo codes, cyclic redundancy codes (CRC) and low density parity check (LDPC) codes may also be acceptable.

[0028] FIG. 4 illustrates a flowchart of steps that set out a method of operation consistent with the memory architecture described above. At 402, first and second memory devices on respective memory modules are accessed independently. First and second column data are transferred over respective primary interfaces to the corresponding modules at a first signaling rate, at 404. Following a buffer delay on each module, the first and second column data is then directed over secondary interfaces of the respective modules at a second signaling rate that is less than the first signaling rate, at 406. The concurrency in the multiple module operations results in an overlap of a first time interval associated with the transfer of the first column data with a second time interval associated with a transfer of the second column data, at 408.

[0029] The description provided above, relating to the architecture and methods of FIGs 1 - 4, relates to a mode of operation where a primary interface data rate is different than a secondary data interface rate. For some embodiments, a further mode of operation is supported that provides reverse compatibility for legacy systems that operate in a conventional manner. For such legacy modes, the primary interface rate may be locked with, or the same as the secondary interface data rate. Moreover, in such a mode, each module may be constrained to a single access at a time, and the secondary data transfer for a first memory module does not overlap a secondary data transfer for a second memory module. Selection of a given mode may be made at system initialization via a programmable interface.

[0030] While the memory architecture and methods described above are shown primarily in a multi-drop context, where multiple memory modules are coupled to all of the memory controller input/output (I/O) data paths, point-to-point topologies are also envisioned. FIGs. 5A - 5C illustrate several design considerations that allow for point-to-point connections between the memory controller and multiple modules. In such an architecture, a first set of I Os from the memory controller provides a first memory module with all of its data, while a second set of I/Os provides a second memory module with all of its data. More or less memory modules may be employed. One example of a dynamic point-to-point memory architecture is shown in

U.S. Serial No. , titled , filed , assigned to the Assignee of the present application, and expressly incorporated herein by reference in its entirety.

[0031] Further referring to FIG. 5A, a motherboard wiring pattern is shown, generally designated 500, that includes a first set of contact pads 502 representing groups of I/O pads associated with a memory controller. Six other sets of contact pads 504, 506, 508, 510, 512 and 514 are provided that represent groups of contacts for mounting respective memory modules. The memory modules are configured such that varying numbers of the modules may be used, while still providing point-to-point connections with the memory controller. For circumstances where not all of the memory module slots are intended to be used, a continuity module may be employed. A continuity module does not include memory devices, but merely passes connections from the memory controller on to another memory module. As can be seen from FIG. 5A, each memory controller I/O from the set of contact pads 502 terminates at a single contact pad from one of the sets of memory module contacts. As an example, contact pad group 520 is routed along path 522 in a point-to-point manner to contact group 524, which corresponds to the group of contacts 512. The other contacts of the memory controller are routed in a similar fashion. Other routings are also made, on the mother board itself, to provide for point-to-point connections in the event that continuity modules are used. For example, contact group 526, belonging to the first group of module contacts 504, is routed to contact group 528, which belongs to module group 514, along path 530.

[0032] FIG. 5B illustrates further detail of a point-to-point configuration where two memory modules 532 and 534 are installed on the motherboard wiring pattern of FIG. 5A, at module contact locations 506 and 514. Continuity modules 536, 538, 540 and 542 are installed in the remaining contact locations 504, 508, 510 and 512. This configuration provides for a point-to-point connection from the first memory controller I/O contact 520, along motherboard path 544, to continuity module 542, where the path is internally routed to contact 546, then connected via motherboard path 548 to module contact 550 of the last group of module contacts 514. The other memory controller I/Os are routed similarly, as shown in FIG. 5B. The end result of the point-to-point routing is to supply the data from three groups of I Os (half of the six shown) to each module.

[0033] While FIG. 5B shows a 2-module point-to-point topology, a six module topology is illustrated in FIG. 5C, where all of the contact slots are filled with memory modules. In a similar manner as that described above with respect to the 2-module configuration, a given subset of the memory controller I/O's are dedicated to supplying all of the data to a given memory module. Thus, in FIG. 5C, the single memory controller I/O provides all of the data to a single module via the routing configurations described above and shown in FIG. 5A. While not shown, the point-to-point routing scheme described above may apply to any number of I Os and modules, depending on the application.

[0034] In one embodiment, the buffered module data transfer methodology of FIGs 1 - 4, where a primary interface may transfer data at a rate higher than a secondary interface transfer rate, may be employed in a point-to-point topology, such as the examples shown in FIGs. 5 A - 5C. FIG. 6 illustrates a memory controller 602, and a memory module 604. For purposes of clarity, only one memory module is shown, with the understanding that multiple memory modules are connected to the memory controller 602 in a point-to-point configuration. Each memory module is formed similarly to those described above with respect to FIGs. 1 and 2, although the buffer circuitry includes logic to configure data distribution among the memory devices in the module.

[0035] Further referring to FIG. 6, one embodiment of the memory controller 602 includes multiple sets of I O contacts (each set corresponding to, for example, six paths) coupled to corresponding I/Os. Only one of the sets of I/Os, at 606, is connected to the memory module, at 608. All of the data transferred between the module 604 and the memory controller 602 is routed through the dedicated I/O set. For one embodiment, data buffers 610 on the module 604 form distributed buffer logic that coordinates the transfer of the data from the connection at 608 into appropriate nibbles for directed routing along secondary data paths to appropriate memory devices on the module. For one embodiment, the secondary data rate is half the primary data transfer rate.

[0036] For one embodiment, a given data transfer to a given module may involve concurrent accesses to a first group of memory devices on one side of the module, and a second group of memory devices on a second side of the module. Separate secondary data paths may be provided to the front and back devices to accomplish this. Additionally, the point-to-point data transfers between the memory controller 602 and modules 604 may be error coded. The error encoding may be generated in accordance with one of many acceptable EDC algorithms including, for example, straightforward single-bit Hamming codes, to more sophisticated high-speed BCH (Bose, Ray-Chaudhuri and Hocquenghem) codes. Other EDC codes, such as "Chipkill" codes, Reed-Solomon codes, turbo codes, cyclic redundancy codes (CRC) and low density parity check (LDPC) codes may also be acceptable.

[0037] FIG. 7 illustrates a timing diagram similar to the diagram of FIG. 3, for a point-to- point transfer of data from the memory controller 602 to a first module 604, with the understanding that similar point-to-point transfers occur simultaneously involving one or more other modules. Respective column commands CAy and CA _V are dispatched along a CA primary bus at a first bit transfer rate, at 702 and 704. First and second column data corresponding to the column commands are then transferred along a primary set of point-to- point links, at 706. The column access commands are then received by CA buffer circuitry, and re-transmitted along the secondary CA bus at a transfer rate that is half the primary CA transfer rate, at 710 and 712. The first and second data is then received by the data buffer circuitry, and the first data is directed to a first group of memory devices, at 714. The second data is directed to a second group of memory devices, at 716. The data transfers along the secondary data paths are at a rate that is half the primary data rate, and carried out at least partly concurrently, where an overlap in time tovERLAP exists between the data transfers. Other data rate ratios may be employed between the primary and secondary interfaces, depending on the application and system topology. [0038] Those skilled in the art will appreciate that the embodiments described above allow for optimum bandwidth utilization along respective primary and secondary signaling paths associated with buffered memory architectures. By transferring data along a primary signaling path at a rate higher than a signaling rate along a secondary signaling path, less costly memory devices may be employed, reducing the overall costs of the memory system. Further, by optimizing the respective primary and secondary signaling bandwidth efficiencies, performance improvements in the memory system may be realized.

[0039] When received within a computer system via one or more computer-readable media, such data and/or instruction-based expressions of the above described circuits may be processed by a processing entity (e.g., one or more processors) within the computer system in conjunction with execution of one or more other computer programs including, without limitation, net-list generation programs, place and route programs and the like, to generate a representation or image of a physical manifestation of such circuits. Such representation or image may thereafter be used in device fabrication, for example, by enabling generation of one or more masks that are used to form various components of the circuits in a device fabrication process.

[0040] In the foregoing description and in the accompanying drawings, specific terminology and drawing symbols have been set forth to provide a thorough understanding of the present invention. In some instances, the terminology and symbols may imply specific details that are not required to practice the invention. For example, any of the specific numbers of bits, signal path widths, signaling or operating frequencies, component circuits or devices and the like may be different from those described above in alternative embodiments. Also, the interconnection between circuit elements or circuit blocks shown or described as multi- conductor signal links may alternatively be single-conductor signal links, and single conductor signal links may alternatively be multi-conductor signal links. Signals and signaling paths shown or described as being single-ended may also be differential, and vice-versa. Similarly, signals described or depicted as having active-high or active-low logic levels may have opposite logic levels in alternative embodiments. Component circuitry within integrated circuit devices may be implemented using metal oxide semiconductor (MOS) technology, bipolar technology or any other technology in which logical and analog circuits may be implemented. With respect to terminology, a signal is said to be "asserted" when the signal is driven to a low or high logic state (or charged to a high logic state or discharged to a low logic state) to indicate a particular condition. Conversely, a signal is said to be "deasserted" to indicate that the signal is driven (or charged or discharged) to a state other than the asserted state (including a high or low logic state, or the floating state that may occur when the signal driving circuit is transitioned to a high impedance condition, such as an open drain or open collector condition). A signal driving circuit is said to "output" a signal to a signal receiving circuit when the signal driving circuit asserts (or deasserts, if explicitly stated or indicated by context) the signal on a signal line coupled between the signal driving and signal receiving circuits. A signal line is said to be "activated" when a signal is asserted on the signal line, and "deactivated" when the signal is deasserted. Additionally, the prefix symbol "/" attached to signal names indicates that the signal is an active low signal (i.e., the asserted state is a logic low state). A line over a signal name (e.g., ' < signal name > ') is also used to indicate an active low signal. The term "coupled" is used herein to express a direct connection as well as a connection through one or more intervening circuits or structures. Integrated circuit device "programming" may include, for example and without limitation, loading a control value into a register or other storage circuit within the device in response to a host instruction and thus controlling an operational aspect of the device, establishing a device configuration or controlling an operational aspect of the device through a one-time programming operation (e.g., blowing fuses within a configuration circuit during device production), and/or connecting one or more selected pins or other contact structures of the device to reference voltage lines (also referred to as strapping) to establish a particular device configuration or operation aspect of the device. The term "exemplary" is used to express an example, not a preference or requirement.

[0041] While the invention has been described with reference to specific embodiments thereof, it will be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, features or aspects of any of the embodiments may be applied, at least where practicable, in combination with any other of the embodiments or in place of counterpart features or aspects thereof. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.