FIELD OF TECHNOLOGY

[0001] This disclosure relates generally to an enterprise method, a technical field of software and/or hardware technology and, in one example embodiment, to storage controller data redistribution.

BACKGROUND

[0002] A storage controller may be used with an additional storage controller (e.g., dual controllers, mirrored controllers) to operate a storage device. A storage controller may obtain information (e.g., a data set, an I/O write, etc.) of a host system. The information may be written (e.g., stored) in a storage controller memory (e.g., a RAM, a DRAM, an SDRAM, etc.) and transferred to the additional storage controller. Transferring the information may include reading the information of the storage controller memory, communicating it to the additional storage controller, and storing it in an additional storage controller memory.

[0003] Reading the information of the storage controller may require a memory bandwidth use, a power use, and/or a system hardware of the storage controller. Reading the information may generate heat, which may reduce an electronic hardware lifespan and/or require additional cooling mechanisms (e.g., fans, heat sinks, heat pipes, water cooling systems, etc.). The additional cooling mechanisms may add an equipment cost and/or a power cost (e.g., electricity for a fan, a cost of electricity, etc.). Reading the information of the storage controller may also exhaust a period of time, which may delay a subsequent storage controller activity (e.g., receiving and/or processing additional information, communicating a write completion signal to the host system, etc.). Reading the information may therefore increase a system cost, cause delays, and/or other inefficiencies.

[0004] A method to improve a performance of the storage controller may include increasing a quantity and/or speed of the storage controller memory. Adding additional storage controller memory, however, may add a memory cost that may exceed an economic benefit. As a result, adding additional storage controller memory may be cost prohibitive. SUMMARY

[0005] A method, apparatus, and system of storage controller data redistribution, are disclosed.

[0006] In one aspect, a method includes acquiring a data set of a storage controller, generating one additional data set prior to storing the data set in a storage controller memory module, communicating the additional data set to an additional storage controller prior to storing the data set in the storage controller memory module, storing the data set in the storage controller memory module, and storing the additional data set in an additional storage controller memory module. [0007] The additional data set may be generated by creating a distinct memory write packet with an identical data payload. The distinct memory write packet may be created using a PCIe switch. The additional data set to the additional storage controller may avoid a memory read operation. The storage controller data redistribution method may include reducing a memory bandwidth load of the storage controller by avoiding a memory read operation. Any number of additional data sets may be generated.

[0008] The additional data set may be communicated to the additional storage controller using a transfer module. The transfer module may be coupled to the storage controller and the additional storage controller. The storage controller data redistribution method may include initiating a storing of the additional data set prior to completing a storing of the data set in the storage controller memory module. [0009] In addition, the method may also include generating a supplemental data set, communicating the supplemental data set to a supplemental storage controller, and storing the supplemental data set in a supplemental storage controller memory module. The method may include accelerating an early write signal to a host controller by communicating the additional data set to an additional storage controller prior to storing the data set in the storage controller memory module. The data sets (e.g., the data set and the additional data set) may be required to be stored in two or more storage controller memories before the accelerated early write signal is sent to the host controller. The early write signal may be accelerated by eliminating a read operation of a storage controller memory, which may otherwise postpone a subsequent write operation in an additional storage controller memory. The early write signal may also be accelerated by performing a memory write operation on more than one storage controller memory at a time, which may not be possible if an additional write operation can begin only after an initial write operation has been completed.

[0010] Accelerating the early write signal may allow a host system to operate more efficiently by reducing a latency of the write status. Reducing the latency of the write status may allow the host system to continue on with normal operation in less time than would otherwise be required.

[0011] The data set of the storage controller may be acquired by a redistribution module between a host module and a storage controller memory. The method may include a storage controller which may also include the redistribution module. [0012] In another aspect, a storage controller data redistribution system includes a redistribution module to acquire a data set of a storage controller and to generate an additional data set prior to storing the data set in a storage controller memory module, a communication module to communicate the additional data set to an additional storage controller prior to storing the data set in the storage controller memory module, the storage controller memory module to store the data set, and an additional storage controller memory module to store the additional data set. [0013] The additional data set may be generated by creating distinct memory write packet with an identical data payload. The distinct memory write packet may be created using a PCIe switch. A storage controller data redistribution system may include communicating the additional data set to an additional storage controller that may avoid a memory read operation. The system may include a memory read operation that may reduce a memory bandwidth load of the storage controller. The additional data set may be communicated to the additional storage controller using a bus. The bus may be coupled to the storage controller and the additional storage controller.

[0014] In yet another aspect, the method includes forming a redistribution module to acquire a data set of a storage controller and to generate the additional data set prior to storing the data set in a storage controller memory module, creating a communication module coupled to the redistribution module to communicate the additional data set to an additional storage controller prior to storing the data set in a storage controller memory module, placing a storage controller memory module coupled to the redistribution module in the storage controller to store the data set, and placing an additional storage controller memory module coupled to the communication module in the additional storage controller to store the additional data set.

[0015] The additional data set may be generated by creating at least one distinct memory write packet with an identical data payload. The distinct memory write packet may be created using a PCIe switch.

[0016] The methods, systems, and apparatuses disclosed herein may be implemented in any means for achieving various aspects, and may be executed in a form of a machine-readable medium embodying a set of instructions that, when executed by a machine, cause the machine to perform any of the operations disclosed herein. Other features will be apparent from the accompanying drawings and from the detailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Example embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

[0018] Figure 1 is a system view of a storage controller, a host module and multiple storage devices communicating with each other, according to one embodiment.

[0019] Figure 2 is an expanded view illustrating the storage controller, the host module and a communication module, according to one embodiment.

[0020] Figure 3 is a system view of multiple storage controllers and communication modules communicating with the host module and multiple storage device modules, according to one embodiment.

[0021] Figure 4A is a process flow of acquiring, generating and storing a data set in a storage controller memory module, according to one embodiment.

[0022] Figure 4B is a continuation of the process flow of Figure 4A illustrating additional operations of the additional data set, according to one embodiment.

[0023] Figure 5 is a process flow illustrating forming a redistribution module and other modules to acquire, generate, and store an additional data set in a storage controller memory module, according to one embodiment.

[0024] Other features of the present embodiments will be apparent from the accompanying drawings and from the detailed description that follows.

DETAILED DESCRIPTION

[0025] A method, apparatus, and system of storage controller data redistribution are disclosed. Although the present embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the various embodiments.

[0026] In one embodiment, an I/O write data may be mirrored to two storage controllers 106 A-B, which may eliminate the need for a memory read operation of a storage controller. A redistribution module 100 A-B may split the data set into two distinct memory write operations prior to the first memory write by redistributing the information between the a storage controller and an additional storage controller. The redistribution may be accomplished using multicasting (e.g., dual-casting, forking, etc). The multicasting may be performed by adding the PCIe switch 212 between the host I/O device (e.g., the host module 208) and the memory module 104A-B on each storage controller 106 A-B.

[0027] In another embodiment, two storage controllers 106 A-B may be connected to each other by means of a PCIe bus. The PCIe bus may connect two PCIe switches, each of which may be included in a storage controller. When an I/O write data passes though the PCIe switch 212 in the form of a PCIe memory write packet, the multicasting feature defined in the Multicasting ECN for PCI Express (PCIe) Base 2.0 specification may allow the packet to be split into two distinct PCIe memory write packets containing the same data payload. One packet may generate a memory write of the data payload to the storage controller memory module 104 A. The other packet may be routed to the storage controller 106B by means of the PCIe bus 214 connecting the two storage controllers 106A-B. The other packet may be written to storage controller memory module 104B. By redistributing (e.g., multicasting, mirroring, etc.) the memory write packet, it may be unnecessary to perform a read operation of the memory module 104 A to obtain information prior to writing the information to memory module 104B. As a result of avoiding the read operation, the memory bandwidth load on storage controllers 106 A-B may be reduced. [0028] In one embodiment, a method includes acquiring a data set (e.g., a memory write packet) of a storage controller (the storage controller 106A of Figure 1). The method further includes generating one or more additional data sets prior to storing the data set in a storage controller memory module (e.g., the memory module 104A of Figure 1). The method further includes communicating the additional data set (e.g., a distinct memory write packet with a payload identical to the original data packet) to an additional storage controller 106B prior to storing the data set in the storage controller memory module 104A and/or 104B. The method further includes storing the data set in the storage controller memory module 104A, and storing the additional data set in an additional storage controller memory module 104B. [0029] In another embodiment, a storage controller data redistribution system includes a redistribution module (e.g., the redistribution module 100A-B, 200) to acquire a data set of a storage controller (e.g., the storage controller of 106A-B, 206, 306A-N) and to generate an additional data set prior to storing the data set in a storage controller memory module (e.g., the memory module 104A-B, 204). The system further includes a communication module (e.g., the communication module 102, 202, 302A-B) to communicate the additional data set to an additional storage controller (e.g., the storage controller of 106A-B, 206, 306A-N) prior to storing the data sets in the storage controller memory module (e.g., the memory module 104A-B, 204). The system also includes the storage controller memory module (e.g., memory module 104A) to store the data set and an additional storage controller memory module (e.g., memory module 104B) to store the additional data set.

[0030] In another embodiment, the method includes forming a redistribution module (e.g., the redistribution module 100 A-B of Figure 1) to acquire a data set of a storage controller (e.g., the storage controller 106A of Figure 1) and to generate an additional data set prior to storing the data set in a storage controller memory module (e.g., the memory module 104A of Figure 1), creating a communication module 102 coupled to the redistribution module 100A-B to communicate the additional data set to an additional storage controller 106B prior to storing the data set in a storage controller memory module (e.g., the memory module 104B of Figure 1), placing the storage controller memory module 104A coupled to the redistribution module IOOA in the storage controller 106 A to store the data set, and placing an additional storage controller memory module 104B coupled to the communication module 102 in the additional storage controller 106B to store the additional data set. [0031] Figure 1 is a system view of a storage controller, a host module and an array of storage device communicating with each other, according to one embodiment. Particularly, Figure 1 illustrates a redistribution module 100A-B, a communication module 102, a memory module 104A-B, a storage controller 106A-B, a host module 108, a storage device module 110A-N, according to one embodiment. [0032] A data set (e.g., a memory write packet) may be a formatted unit of data carried by a packet mode computer network. A computer communications link that does not support packets, such as traditional point-to-point telecommunications links, may simply transmit data as a series of bytes, characters, or bits alone. The additional data set (e.g., an additional memory write packet) may include a distinct memory write packet and a data payload identical to a data payload of an original data set. The additional data set may be generated through multicasting, mirroring, splitting the memory write packet, and/or copying the memory write packet. The original data set may be transmitted and/or stored. Alternatively, two or more new memory write packets may be generated to be stored in the storage controllers (e.g., the storage controller 106A-B).

[0033] The redistribution module 100A-B may split a data set into two distinct memory writes operations using multicasting (e.g., dual-casting, forking, etc) to acquiring the data set of the storage controller 106A-B. The redistribution module may generate an additional data set prior to storing the data set in a storage controller memory module 104A-B. The redistribution module (e.g., the redistribution module 100A-B of Figure 1) may include a PCIe switch (e.g., the PCIe switch 212). The PCIe switch may split a memory write packet in accordance with a multicasting feature defined in the Multicasting ECN for PCI Express (PCIe) Base 2.0 specification. The PCIe switch may communicate with a PCIe bus, which may connect two or more storage controllers. The PCIe switch may communicate a multicast memory write packet to an additional storage controller. [0034] The communication module (e.g., the communication module 102, 202, 302A- B) may include a computer bus that may be used to attach peripheral devices to a processing unit (e.g., the host system, the computer motherboard, etc). The communication module 102 may communicate the additional data set to an additional storage controller before the data set is stored in the storage controller memory module 104A and/or 104B. The additional data set may be communicated prior to an initiation and/or a completion of a storage process of a memory write packet in the memory module 104 A and/or 104B. [0035] An early write signal to a host controller may be accelerated by communicating the additional data set to an additional storage controller prior to storing the data set in the storage controller memory module. The data sets (e.g., the data set and the additional data set) may be required to be stored in two or more storage controller memories before the accelerated early write signal is sent to the host controller. The early write signal may be accelerated by eliminating a read operation of a storage controller memory, which may otherwise postpone a subsequent write operation in an additional storage controller memory. The early write signal may also be accelerated by performing a memory write operation on more than one storage controller memory at a time, which may not be possible if an additional write operation can begin only after an initial write operation has been completed.

[0036] Accelerating the early write signal may allow a host system to operate more efficiently by reducing a latency of the write status. Reducing the latency of the write status may allow the host system to continue on with normal operation in less time than would otherwise be required. Reducing the latency of the write status may allow the host system to retry the operation and/or create a system fault in less time than would otherwise be required.

[0037] The memory module 104A-B may hold data in a magnetic medium (e.g., flash memory, a hard drive) and/or any other memory format (e.g., optical, mechanical, volatile memory, nonvolatile memory, etc.). An additional storage controller memory module may store the additional data set (e.g., the mirrored data, the multicast data, the data set generated by the redistribution module, etc.). The memory module 104 A-B may be coupled to the redistribution module 100A-B in the storage controller 106 A-B.

[0038] The storage controller (e.g., the storage controller of 106A-B, 206, 306A-N) may be used in pairs in a redundant configuration to prevent a loss of data. If one controller fails, the host system (e.g., the host module 108) associated with the storage controller may still have access to all of the storage devices (e.g., the storage device modules 110A-N, 210A-B, 310A-N). The host module 108 may include a processing unit that may communicate with various devices (e.g., the storage controllers 106A-B, storage device modules 11 OA-N) may communicate. The host module may download or upload programs that affect its operation with the storage controllers and/or storage device modules.

[0039] The storage device module 11 OA-N may hold and/or process information. The storage device module 11 OA-N may include a recording medium to hold information and data storage equipment to process information. The storage device module 110A-N may access a separate portable recording medium and/or a fixed device that may store and retrieve information.

[0040] In an example embodiment, the storage controllers 106A-B may include the redistribution modules 100A-B and the memory modules 104 A-B. The memory module 104A-B and the storage controllers 106A-B may communicate through the communication module 102. The host module 108 may communicate with the storage device modules 11 OA-B through storage controllers 106 A-B. [0041] Figure 2 is an expanded view of the storage controller that also illustrates the host module and a communication module, according to one embodiment. [0042] Particularly, Figure 2 illustrates a redistribution module 200, a communication module 202, a memory module 204, a storage controller 206, a host module 208, a storage device module 210A-B, a PCIe switch 212, a PCIe bus 214, an input I/O module 216, an output I/O module 218, a controller module 220, a processor module 222, according to one embodiment.

[0043] The redistribution module 200 may include a PCIe switch that may split the data set into two distinct memory write operations before the first memory write through the use of multicasting (e.g., dual casting, forking, etc). The multicasting may be accomplished by adding a PCIe switch (e.g., PCIe switch 212) between the host I/O device (e.g., input I/O module 216) and memory controller (e.g., controller module 220) on each storage controller (e.g., storage controller module 106A-B). [0044] The communication module 202 may include a PCIe bus 214 which may connect the two storage controllers 206A-B with two PCIe switches. The memory module 204 may be coupled to the redistribution module 200 in the storage controller 206, and the memory module 204 may store the data set. The storage controller 206 may provide input-output writes for the host system (e.g., the host module 208). The host module 208 may be a controlling processing unit in a redundantly configured system to store information in the storage device module 210A-B. The host module may include a server that controls a Redundant Array of Inexpensive Disks (RAID). Alternatively, the host module may include a microcomputer that controls a programmable logic controller (PLC) that operates with a set of hard disks and/or other electronic devices.

[0045] The storage device modules (e.g., the storage device modules 110A-N, 210A-B, 310A- N) may hold information. The storage device modules may communicate with other storage device modules. The storage device modules may communicate with storage controllers (e.g., the storage controllers 106A-B, 206, 306A-N). The storage device modules may communicate with an output I/O module (e.g., the output I/O module 218), the memory module 104A-B, and/or the host module (e.g., the host module 108, 208, and/or 308).

[0046] The PCIe switch 212 may create distinct memory write packets. When I/O write data passes though the PCIe switch in the form of PCIe memory write packets, the multicasting feature defined in the Multicasting ECN for PCI Express (PCIe) Base 2.0 specification may allow the packet to be split into two distinct PCIe memory write packets containing the same data payload. One of these packets may generate a memory write of the data payload to storage controller 206.

[0047] The PCIe bus 214 may be included in the communication module 202 and may connect the two storage controllers (e.g., the storage controller 106A-B, 206, 306A-N) and/or the PCIe switches of the storage controllers 106A-B. The PCIe bus 214 may communicate with the PCIe switch 212, the redistribution module 200, and/or the storage controller 206.

[0048] The input I/O module 216 may be a host I/O device which may reside in the storage controller 206. The input I/O module 216 may communicate information between the host module 208 and the PCIe switch in the redistribution module 200. The output I/O module 218 may reside in the storage controller 206, and it may communicate with the controller module 220 and the storage device module 21 OA-B. The controller module 220 may control memory write operations between the processor module 222 and the memory module 204. The processor module 222 may process the data set stored in the storage device module 210A-B. [0049] In an example embodiment, the storage controller 206 may include the input I/O module 216, the redistribution module 200, the memory module 204, the controller module 220, the processor module 222, and the output I/O module 218. The redistribution module 200 may include a PCIe switch 212. The communication module 202 may include a PCIe bus 214. The PCIe switch 212 may be connected to PCIe bus 214. The output I/O module 218 may be connected with the storage device module 21 OA-B. The modules in the embodiment may communicate with each other. [0050] Figure 3 is a system view of multiple storage controllers and communication modules communicating with the host module and multiple storage device modules, according to one embodiment. Particularly, Figure 3 illustrates a communication module 302A-N, a storage controller 306A-N, a host module 308 and a storage device module 31 OA-N, according to one embodiment. [0051] The communication module 302A-N may communicate information between the storage controller 306A-N. The communication module 302A-N may include the PCIe bus 214.

[0052] The storage controller 306A-N may be connected in an array. The storage controller 306A-N may communicate information between the host module 308 and the storage device module 310A-N. Additionally, the storage controllers 306A-N may be connected to each other by means of an additional PCIe bus which may also be connected to the PCIe switches. When I/O write data passes though PCIe switch in the form of a PCIe memory write packet, the multicasting feature defined in the Multicasting ECN for PCI Express (PCIe) Base 2.0 specification may allow the packet to be split into two distinct PCIe memory write packets containing the same data payload. One memory write packet may be sent to a storage controller memory, and a second memory write packet may be communicated to an additional storage controller.

[0053] The host module 308 may communicate information to be stored and/or retrieved using the storage device module 31 OA-N, which may hold the data set and/or the mirrored data. The storage controller 306A-N may process information to be stored in the storage device modules 31 OA-N, and the storage controller may redistribute (e.g., multicast, mirror) information to an additional storage controller regarding the information communicated by the host module 308. The storage controller 306A-N may communicate with the communication module 302A-N, the storage device module 31 OA-N and the host module 308.

[0054] Figure 4A is a process flow of acquiring, generating and storing a data set in a storage controller memory module, according to one embodiment. In operation 402, a data set of a storage controller (e.g., the storage controller 206 of Figure 2) may be acquired (e.g., using the redistribution module 200 of Figure 2). In operation 404, the additional data set may be generated (e.g., using the redistribution module 200 of Figure 2) prior to storing the data set in a storage controller memory module (e.g., the storage controller memory module 204 of Figure 2). In operation 406, the additional data set may be communicated (e.g., using the communication module 102 of Figure 1) to an additional storage controller prior to storing the data set in the storage controller memory module.

[0055] In operation 408, an early write signal to a host controller may be accelerated by communicating the additional data set to an additional storage controller (e.g., the storage controller 106 A-B of Figure 1) prior to storing the data set in the storage controller memory module 104A-B. The data sets (e.g., the data set and the additional data set) may be required to be stored in two or more storage controller memories (e.g., the storage controller memory module 104 A-B of Figure 1) before the accelerated early write signal is sent to the host controller. The early write signal may be accelerated by eliminating a read operation of a storage controller memory, which may otherwise postpone a subsequent write operation in an additional storage controller memory. The early write signal may also be accelerated by performing a memory write operation on more than one storage controller memory at a time, which may not be possible if an additional write operation can begin only after an initial write operation has been completed. In operation 410, the data set may be stored in the storage controller memory module 104 A-B.

[0056] Figure 4B is a continuation of the process flow of Figure 4A illustrating additional operations of the additional data set, according to one embodiment. In operation 412, the storage of the additional data set may be initiated prior to completion of storing of the data set in the storage controller memory module 104A- B. In operation 414, the additional data set may be stored in an additional storage controller memory module 104A-B. The additional data set may be generated (e.g., using the redistribution module 200 of Figure 2) by creating a distinct memory write packet with an identical data payload. The distinct memory write packet may be created using a PCIe switch 212. The additional data set may be communicated to the additional storage controller that may avoid a memory read operation. In operation 416, a memory bandwidth load of the storage controller 206 may be reduced by avoiding a memory read operation. The additional data set may be communicated (e.g., using the communication module 302A- N) to the additional storage controller 306 A-N using a transfer module. In addition, the transfer module may be coupled to the storage controller 306A-N and the additional storage controller 306A-N. [0057] In operation 418, a supplemental data set may be generated (e.g., using the redistribution module 100A-B of Figure 1) that may communicate the supplemental data set to the supplemental storage controller 106A-B, and store the supplemental data set in a supplemental storage controller memory module 106-B. The data set of the storage controller 106A-B may be acquired by a redistribution module (e.g., the redistribution module 100A-B of Figure 1) between a host module (e.g., the host module 108 of Figure 1) and a storage controller memory module (e.g., the storage controller memory module 104A-B of Figure 1). The storage controller 106A-B may include the redistribution module 100 A-B.

[0058] Figure 5 is a process flow illustrating forming a redistribution module and other modules to acquire, generate, and store an additional data set in a storage controller memory module, according to one embodiment. In operation 502, a redistribution module (e.g., the redistribution module 200 of Figure 2) may be formed that may acquire a data set of a storage controller and may generate an additional data set prior to storing the data set in a storage controller memory module (e.g., the memory module 204 of Figure 2). In operation 504, a communication module 202 may be created that may be coupled to the redistribution module 200 which may communicate with the additional data set to an additional storage controller prior to storing the data set in a storage controller memory module 204. [0059] In operation 506, a storage controller memory module 204 may be placed that may be coupled to the redistribution module 200 in the storage controller 206 which may store the data set. In operation 508, an additional storage controller memory module (e.g., the memory module 204 of Figure 2) may be placed which may be coupled to the communication module 302A-N in the additional storage controller 306 A-N which may store the additional data set. The additional data set may be generated by creating a distinct memory write packet with an identical data payload. The distinct memory write packet may be created using a PCIe switch. [0060] Although the present embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the various embodiments. For example, the various devices, modules, analyzers, generators, etc. described herein may be enabled and operated using hardware circuitry (e.g., CMOS based logic circuitry), firmware, software and/or any combination of hardware, firmware, and/or software (e.g., embodied in a machine readable medium). For example, the various electrical structure and methods may be embodied using transistors, logic gates, and electrical circuits (e.g., application specific integrated (ASIC) circuitry and/or in Digital Signal Processor (DSP) circuitry).

[0061] Particularly, the redistribution module 100 A-B, the communication module 102, the memory module 104A-B, the storage controller 106A-B, the host module 108, the storage device module 110A-N of Figure 1, the redistribution module 200, the communication module 202, the memory module 204, the storage controller 206, the host module 208, the storage device module 210A-B, the PCIe switch 212, the PCIe bus 214, the input I/O module 216, the output I/O module 218, the controller module 220 of Figure 2, and the communication module 302A-N, the storage controller 306A-N, the host module 308, the storage device module 310A-N of Figure 3 may be enabled using a redistribution circuit, a communication circuit, a memory circuit, a storage controller circuit, a host circuit, a storage device circuit, a PCIe switch circuit, a input I/O circuit, a output I/O circuit, a controller circuit and other circuit.

[0062] In addition, it will be appreciated that the various operations, processes, and methods disclosed herein may be embodied in a machine-readable medium and/or a machine accessible medium compatible with a data processing system (e.g., a computer system), and may be performed in any order (e.g., including using means for achieving the various operations). Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.