CONTROLLER

RELATED APPLICATIONS

[0001] This application hereby claims the benefit of and priority to U.S. Provisional Patent Application Number 62/136,336, titled "SEQUENTIAL WRITE/RANDOM READ STORAGE CONTROLLER", filed on March 20, 2015 and which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

[0002] Aspects of the disclosure are related to data storage and in particular to sequential write/random read storage controllers.

TECHNICAL BACKGROUND

[0003] Flash non-volatile storage devices are commonly used in computers as highspeed solid state storage devices. These devices require flash memory controllers that typically work as random write/random read, small data block devices. Their data block size is typically 8-16KB. Since these devices provide random write/random read data traffic, performance requirements limit the amount of error correction and data protection that may be used without reducing performance.

[0004] Further, these flash controllers require a small block flash translation layer (FTL) which is a mapping of host based small blocks into pages within the flash memory device. Flash translation layers are typically complex and create additional processes for the flash controller to manage such as garbage collection, management of a very large translation table, wear leveling, power fail recovery, and the like. Because of these requirements, flash translation layers are expensive in terms of area and power within the flash controller, and also reduce performance and increase wear.

OVERVIEW

[0005] In an embodiment, a storage controller for a storage system is provided. The storage controller includes a host interface, configured to receive host data for storage within the storage system, a storage interface, configured to transmit storage data to the storage system, and processing circuitry coupled with the host interface and the storage interface. The processing circuitry is configured to (a) receive the host data from a host, through the host interface, for storage within a partition in the storage system, and (b) determine a quantity of error correction code levels based on at least one partition parameter

corresponding to the partition.

[0006] The processing circuitry is further configured to (c) partition the host data into a plurality of payloads, (d) provide metadata for each of the payloads, (e) calculate error correction codes for each of the payloads and the metadata associated with each of the payloads, and to (f) create a plurality of codewords, each codeword comprising a payload and metadata and error correction codes corresponding to the payload. The processing circuitry is also configured to (g) based on the quantity of error correction code levels, either store the codewords in the storage system through the storage interface or partition the codewords into a plurality of further payloads, the further payloads iteratively processed according to steps (d) through (g) to provide the quantity of error correction code levels.

[0007] In another embodiment, a method of operating a storage controller is provided. The method includes (a) receiving host data from a host, through a host interface, for storage within a partition in the storage system, and (b) determining a quantity of error correction code levels based on at least one partition parameter corresponding to the partition. The method further includes (c) partitioning the host data into a plurality of payloads, (d) providing metadata for each of the payloads, (e) calculating error correction codes for each of the payloads and the metadata associated with each of the payloads, and (f) creating a plurality of codewords, each codeword comprising a payload and metadata and error correction codes corresponding to the payload. The method also includes (g) based on the quantity of error correction code levels, either storing the codewords in the storage system through the storage interface or partitioning the codewords into a plurality of further payloads, the further payloads iteratively processed according to steps (d) through (g) to provide the quantity of error correction code levels.

[0008] In a further embodiment, one or more non-transitory computer-readable media having stored thereon program instructions to operate a storage controller for a storage system are provided. The program instructions, when executed by processing circuitry, direct the processing circuitry to at least (a) receive host data from a host, through a host interface, for storage within a partition in the storage system, and to (b) determine a quantity of error correction code levels based on at least one partition parameter corresponding to the partition.

[0009] The programming instructions further direct the processing circuitry to at least (c) partition the host data into a plurality of payloads, (d) provide metadata for each of the payloads, (e) calculate error correction codes for each of the payloads and the metadata associated with each of the payloads, and to (f) create a plurality of codewords, each codeword comprising a payload and metadata and error correction codes corresponding to the payload. The programming instructions also direct the processing circuitry to at least (g) based on the quantity of error correction code levels, either store the codewords in the storage system through the storage interface or partition the codewords into a plurality of further payloads, the further payloads iteratively processed according to steps (d) through (g) to provide the quantity of error correction code levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Many aspects of the disclosure can be better understood with reference to the following drawings. While several implementations are described in connection with these drawings, the disclosure is not limited to the implementations disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents.

[0011] Figure 1 illustrates a computer host and data storage system.

[0012] Figure 2 illustrates a data storage system.

[0013] Figure 3 illustrates a method for operating a storage controller.

[0014] Figure 4 illustrates an example of memory storage cell organization.

[0015] Figure 5 illustrates a memory chip array.

[0016] Figure 6 illustrates a data storage system.

[0017] Figure 7 illustrates a data storage system including a media emulation layer.

[0018] Figure S illustrates an example of storage array organization.

10019] Figure 9 illustrates storage array partition examples.

[0020] Figure 10 illustrates an example data organization.

[0021] Figure 11 illustrates an example sequential writing process.

[0022] Figure 12 illustrates a storage controller.

DETAILED DESCRIPTION

[0023] Figure 1 illustrates computer host and data storage system 100. In this example embodiment, host system 110 sends data to, and receives data from, storage controller 120 for storage in storage system 130. In an example embodiment, storage system 130 comprises flash non-volatile memory, such as NAND memory. NAND memory is just one example, other embodiments of storage system 130 may comprise other types of storage. Storage controller 120 communicates with storage system over link 150, and performs the function of configuring data received from host system 110 into a format that efficiently uses the memory resources of storage system 130.

[0024] In this example, storage controller 120 provides data to storage system 130 using a sequential write/random read traffic flow. This traffic flow provides a number of advantages over the small data block random write/random read traffic flow typically used with flash non-volatile memory. In particular, the sequential write/random read traffic flow provides for efficient use of memory within the storage array and offers performance improvements including the availability of multiple, bi-directional data streams that are mapped to different partitions within storage system 130. These partitions may be configured individually with a desired partition size, data block size, storage density, physical memory attributes, protection attributes, security level, quality of service requirements, data type, data source, data stream attributes, and the like.

[0025] Partitions may be created or requested by different systems within computer host and data storage system 100. In some examples, host system 110 sends a request for a partition to storage controller 120 along with the parameters used to configure the partition. In other cases, a media emulation layer within storage controller 120 may create partitions for use in managing host data such as caching between partition, storing tables, performing garbage collection across partitions, and the like. Further, storage controller 120 itself may create partitions for system area data storage. In all cases, the partition creator owns the partition and defines the partition attributes (using the partition parameters).

[0026] Storage controller 120 provides translation between standard storage interfaces and command protocols used by host system 110 to a command protocol and the physical interface used by storage devices within storage system 130. Additionally, storage controller 120 may emulate different media types such as a hard disk drive, tape drive, or the like, allowing host system 110 to interface with storage system 130 as if it were a hard disk drive, tape drive, or the like. In such an emulation mode, storage controller 120 translates between host system 110 and storage system 130, configuring data traffic and translating commands between the format of the media type desired by host system 110 and the protocol used by storage system 130.

[0027] Within a single storage system 130 different partitions may be used to emulate different media types, with storage controller 120 managing the different data traffic streams and configuring the data, along with translating the commands in each stream as necessary.

[0028] Additionally, storage controller 120 implements error correction code (ECC) encode/decode functions, along with data encoding, data recovery, retry recovery methods, and other processes and methods to optimize data integrity. Storage controller 120 includes the ability to provide multi-level ECC correction over a wide range of data block sizes, allowing for the correction of data errors both large and small. ECC correction levels are determined on a per partition basis by the partition parameters used to create each partition.

[0029] Storage controller 120 may take any of a variety of configurations. In some examples, storage controller 120 may be a Field Programmable Gate Array (FPGA) with software, software with a memory buffer, an Application Specific Integrated Circuit (ASIC) designed to be included in a single module with storage system 130, a set of Hardware Description Language (HDL) commands, such as Verilog or System Verilog, used to create an ASIC, a separate module from storage system 130, built in to storage system 130, or any of many other possible configurations.

[0030] Host system 110 communicates with storage controller 120 over various communication links, such as communication link 140. These communication links may use the Internet or other global communication networks. Each communication link may comprise one or more wireless links that can each further include Long Term Evolution (LTE), Global System For Mobile Communications (GSM), Code Division Multiple Access (CDMA), IEEE 802.11 WiFi, Bluetooth, Personal Area Networks (PANs), Wide Area Networks, (WANs), Local Area Networks (LANs), or Wireless Local Area Networks (WLANs), including combinations, variations, and improvements thereof. These communication links can carry any communication protocol suitable for wireless communications, such as Internet Protocol (IP) or Ethernet.

[0031] Additionally, communication links can include one or more wired portions which can comprise synchronous optical networking (SONET), hybrid fiber-coax (HFC), Time Division Multiplex (TDM), asynchronous transfer mode (ATM), circuit-switched, communication signaling, or some other communication signaling, including combinations, variations or improvements thereof. Communication links can each use metal, glass, optical, air, space, or some other material as the transport media. Communication links may each be a direct link, or may include intermediate networks, systems, or devices, and may include a logical network link transported over multiple physical links.

[0032] Storage controller 120 communicates with storage system 130 over link 150. Link 150 may be any interface to a storage device or array. In one example, storage system 130 comprises NAND flash memory and link 150 may use the Open NAND Flash Interface (ONFI) command protocol, or the "Toggle" command protocol to communicate between storage controller 120 and storage system 130. Other embodiments may use other types of memory and other command protocols. Other common low level storage interfaces include DRAM memory bus, SRAM memory bus, and SPI.

[0033] Link 150 can also be a higher level storage interface such as SAS, SATA, PCIe, Ethernet, Fiber Channel, Infiniband, and the like. However - in these cases, storage controller 120 would reside in storage system 130 as it has its own controller. [0034] Figure 2 illustrates data storage system 200. This example system comprises storage controller 210 and storage system 220. Storage system 220, comprises storage array 230. Storage array 230 comprises memory chips 1-6 (231-236). In an example embodiment, each memory chip 231-236 is a NAND memory integrated circuit. Other embodiments may use other types of memory.

[0035] Storage controller 210 comprises a number of blocks or modules including host I/O interface 211, sequential write/random read (SWRR) controller 212, storage I/O port 0 213, and storage I/O port 1 214. Sequential write/random read controller 212 communicates with the other blocks over links 215, 216, and 217. Storage I/O port 0 213 communicates with storage system 220 over link 201 and storage I/O port 1 214 communicates with storage system 220 over link 202.

[0036] In some example embodiments, storage I/O ports 0 and 1 (213 and 214) may use the Open NAND Flash Interface (ONFI) command protocol, or the 'Toggle" command protocol to communicate with storage system 220 over links 201 and 201. The ONFI specification includes both the physical interface and the command protocol of ONFI ports 0 and 1. The interface includes an 8-bit bus (in links 201 and 202) and enables storage controller 210 to perform read, program, erase, and other associated operations to operate memory chips 1-6 (231-236) within storage array 230.

[0037] Multiple memory chips may share each ONFI bus, however individual memory chips may not share multiple ONFI buses. Chips on one bus may only communicate with that bus. For example, memory chips 1-3 (231-233) may reside on bus 201, and memory chips 4-6 (234-236) may reside on bus 202.

[0038] In this example, sequential write/random read controller 212 receives host data from a host through host I/O interface 211 over link 215. Sequential write/random read controller 212 configures the data as needed for storage in storage system 220 and transfers the data to storage I/O ports 0 and 1 (213 and 214) for transfer to storage system 220 over links 201 and 202.

[0039] Figure 3 illustrates a method for operating storage controller 210 as illustrated in Figure 2. In this example, storage controller 210 performs all of the following operations: Receiving host data from host system 110, through host I/O interface 211, for storage within storage system 220, (operation 300). Determining a quantity of error correction levels based on at least one partition parameter corresponding to the partition, (operation 302).

Partitioning the host data into a plurality of level (N) payloads, (operation 304).

[0040] Providing metadata for each of the payloads, (operation 306). Calculating error correction codes for each of the payloads and the metadata associated with each of the payloads, (operation 308). Creating a plurality of codewords, each codeword comprising a payload and metadata and error correction codes corresponding to the pay load, (operation 310).

[0041] In a decision step, determining if there are more error correction code levels remaining to be processed, (operation 312). If there are no more error correction code levels remaining to be processed, storing the codewords in storage system 220, through storage I/O ports 0 and 1 (213 and 214), (operation 314). If more error correction code levels remain to be processed, partitioning the plurality of codewords into a plurality of payloads, (operation 316) and continuing at operation 306.

[0042] Figure 4 illustrates an example of data storage cell organization. NAND flash non-volatile storage systems are organized as an array of memory cells 400 surrounded by control logic to allow it to be programmed, read, and erased. The cells in a typical flash array are organized in pages 410 for program and read operations. Multiple pages 410 are in a data (NAND) block 420 and usually must be written sequentially within a data block 420. Erase operations are done on a data block 420 basis. [0043] Figure 5 illustrates memory chip array 500. Memory chip array 500 within a die (chip/integrated circuit) comprises multiple data (NAND) blocks 500-522. The page 410, data block 420 and array 500 sizes vary by flash die design. Typical sizes are currently 16KB pages, 512 pages per data block, and 1024 data blocks per die. These sizes continue to grow as memory die increase capacity.

[0044] The memory chip array 500, when using NAND flash memory data cells, is controlled externally by the industry standard ONFI (Open NAND Flash Interface) specification or the 'Toggle" interface specification. The ONFI specification includes both the physical interface and the command protocol. The interface has an 8-bit bus and enables a controller to perform program, read, erase, and associated operations to operate the NAND die. Multiple die can share an ONFI bus.

[0045] Most NAND memory systems are built out of an array of NAND die to increase performance and capacity. Multiple NAND die can be packaged together, multiple packages can be assembled together on assemblies, and multiple assemblies can exist within a NAND memory system. The entire set of NAND memory being controlled is referred to as the storage array.

[0046] In order to use NAND memory in a system, a storage controller is required. The controller has many functions mcluding, but not limited to: converting between ONFI and standard storage interfaces and command protocols, organizing data across the storage array, adding metadata and/or tables to keep track of the data written and where it is mapped in the array, implementing ECC encode/decode, data encoding, recovery, retry recovery algorithms, and other processes and algorithms to optimize data integrity, and NAND and array management functions to optimize storage usage, capacity, performance, life, and power. [0047] Figure 6 illustrates data storage system 600. This example includes network or host system 610, storage controller 615, and storage array 630. Host system 610 and storage controller 615 exchange data over link 611. Storage controller 615 and storage array 630 exchange data and commands over ONFI buses 622 and 623. Storage controller 615 includes ONFI port 0620 which exchanges data and commands with storage array 630 over ONFI bus 622 and ONFI port 1 621 which exchanges data and commands with storage array 630 over ONFI bus 623.

[0048] Storage array 630 comprises a plurality of packages 640-642 each containing two NAND dies (DIE-LUN (Logical Unit Number) 0 and 1). In this example, ONFI port 0 620 sends data and commands to all of the memory dies having a logical unit number of "0". This includes DIE-LUN 0 650 in package 0640, DIE-LUN 0660 in package 1 641, and DIE- LUN 0 in package N 642. ONFI port 1 621 sends data and commands to all of the memory dies having a logical unit number of "1". This includes DIE-LUN 1 651 in package 0640, DIE-LUN 1 661 in package 1 641, and DIE-LUN 1 in package N 642. The ONFI ports may operate simultaneously to provide enhanced bandwidth.

[0049] This example is a storage controller 615 that is optimized to operate in a sequential write/random read mode. The example description focuses on the implementation with NAND flash, but the example is not intended to be limited to NAND flash storage array control. The controller of this example can be applied to any non-volatile storage technology that requires an integrated control function packaged with the medium in order to operate. Storage controller 615 is also referred to as "SWRR controller".

[0050] The SWRR controller 615 of this example implements a method of organizing and managing the data of the array and is optimized for the type of traffic that best suits a NAND storage array or other storage devices with similar characteristics. The traffic definition and behavior includes: multiple, bi-directional data streams that are mapped to different partitions, sequential writing is done using sequentially numbered data blocks.

[0051] In order for SWRR controller 615 to perform sequential write/random read traffic flows with storage array 630, rewrites of a data block in a partition without an intervening erase is not allowed, and data blocks within a partition may not be written out of order. Storage controller 615 supports random reads using numbered data blocks. Also, data block sizes are programmable per partition within storage array 630.

[0052] Figure 7 illustrates data storage system 700 including a media emulation layer (also referred to as a virtualization layer) 711. In this example the ingress interface to SWRR controller 710 includes media emulation layer 711. Media emulation layer 711 includes all of the command translation, data mapping, management, and tracking needed to adapt SWRR controller 710 to work in many modes and with many interfaces. Media emulation layer 711 also performs any functions required to shape the data traffic to match the SWRR traffic model.

[0053] The data traffic at emulation to SWRR interface 712 shown in Figure 7 matches the sequential write/random read traffic model as described above. The egress interface to storage array 730 is a multi-channel ONFI interface (ONFI ports 0 and 1 (720 and

721)) in the case of a NAND flash array. However, the ONFI interface can be adapted or replaced to directly interface to many different types of storage technologies.

[0054] Figure 8 illustrates an example of storage array 800 organization. Storage array 800 is made up of a large number of data blocks that are contained in multiple die, packages, and, possibly, assemblies. For a fresh storage array that has yet to have any data stored in it, it is seen by a storage controller as a large pool of free data blocks.

[0055] The SWRR controller uses programmable data blocks instead of fixed, 512 or

4K Byte blocks. The data block size can be optimized for the system and, for NAND, are typically at least the size of a page. In addition, the large, programmable blocks allow the system to be optimized to fit the storage array and the host system during configuration time. Another advantage of large data blocks is that it makes the controller scale to larger array sizes much more easily.

[0056] The data blocks are organized into data block stripes. A data block stripe contains 1 or more data blocks and is programmable. The highest level structure that controls how the array is accessed is a partition. All data is stored in partitions. A partition consists of any integer number of data block stripes. Each partition has a specific definition and size. Prior to storing any data, one or more partitions need to be created.

[0057] In this example, storage array 800 comprises 14 partitions. Partition 1 801 includes 4 data blocks. Partition 2 802 includes 16 data blocks. Partitions 3-12 (803-812) each include one data block, and partitions 13 and 14 (813 and 814) each include 20 data blocks.

[0058] All data is written in partitions. Each partition has its own definition, comprising one or more partition parameters.

[0059] Partitions can be static, extendable, or linked. They can also be created or destroyed at any time. When a partition is created, it must have at least one block from the free pool assigned. A partition may be destroyed if it has no valid data in it meaning it hasn't been written to yet or it has been erased.

[0060] Static partitions are defined as a fixed set of blocks and they cannot change unless they are destroyed.

[0061] Extendable partitions can grow as needed. Additional blocks from the free pool may be assigned as needed to keep it from filling up prior to filling the entire storage array. Extendable partitions are useful so that the array doesn't have to be fully allocated without prior knowledge of how much space is needed in each partition. [0062] Linked partitions can be either static or dynamic. Linked partitions are treated logically as a single partition. This feature is useful when a large sequential write session has filled a partition, but there is additional data to append later. Another use of linked partitions is for mixing densities within a data stream or to provide a caching function. Linked partitions need to have compatible attributes - but not identical. The primary areas of compatibility are data type, security level, and Quality of Service (QoS). Partition definitions can only change when they are created or erased.

[0063] There are many partition parameters to allow the controller to be optimized to many storage arrays and hosts. The categories of parameters are:

- Data block size.

- Physical storage cell density - e.g. SLC (Single Level Cell), MLC (Multi Level Cell), TLC (Triple Level Cell), etc.

- Protection - number of codeword levels, codeword sizes, ECC type and overhead, etc.

- Security - type of encryption, key and password handling, access rights, and digital rights management attributes.

- Quality of Service - different priorities of traffic, support of isochronous streams, etc.

- Data type and source - e.g. system, host, local array, network, etc.

- Physical - number of data block stripes, number of data blocks per data block stripe, location of data blocks in the partition.

- Data stream definition - interface type (e.g. fixed blocks, stream, bulk USB, etc.), file/data format (e.g. host block definition, file system awareness, etc.), interface source - for controllers with multiple active interfaces, etc.

[0064] As these parameters are set on a per partition basis, it enables straightforward implementation of multiple features such as SLC caching, mixing data security levels, support of multiple types of virtual devices and volumes simultaneously, and supporting different levels of protection based on data type or source as just a few examples.

[0065] Figure 9 illustrates storage array partition examples. This example illustrates two, more detailed examples of partitions. Partition 0 900 is the smallest possible with only one data block within one data block stripe 910. It may be used for internal system data such as configuration, table storage, NAND optimization parameters, manufacturing data, firmware upgrades, and the like. In this case, partition 0900 is a "system" partition meaning that it is used only for data internally generated by the controller.

[0066] Partition 1 910 is very large and consists of M by N data blocks arranged in multiple integer data block stripes 911-913. A data block stripe is a group of data blocks that are written in parallel to increase performance and optimize data protection. The data block members of a data block stripe can be split across dies, packages, or cards.

[0067] Once defined, a partition is filled sequentially with layered codewords. The partition is full when the remaining space is smaller than the highest level/largest size of codeword. A partition can only be filled once. Once it is full, it is read only.

[0068] A partition can be erased in block stripes, or as a full partition unit. Once a partition has been erased, the data blocks from that partition are returned to the free pool.

The partition can then be re-used, re-configured, or destroyed. New partitions can be created at any time with any size using data blocks from the free pool that are available.

[0069] When a block stripe is deleted, tables and high level metadata generated in the media emulation layer manage the deletion. A higher level process - normally in the media emulation layer or the host itself, will take care of copying all of the valid data from the victim block stripe to new data blocks appended to the end of the partition prior to releasing the block stripe for erase and return to the free pool. [0070] Figure 10 illustrates an example data organization. The SWRR controller organizes data into partitions. Each partition consists of one or more layers of data encoded with ECC. The number of data layers, their size, and ECC redundancy is programmable during partition creation. This layered, configurable data protection scheme is extremely flexible, resulting in more powerful error correction with less overhead.

[0071] The SWRR controller can have one or more levels of codewords. The example in Figure 10 shows three levels, but there can be more or fewer levels depending on the configuration of the partition. The different levels allow optimizations and trade-offs to be made between performance, overhead, and data reliability. Oftentimes, the codewords at different levels correspond to different physical structures that could fail. The levels allow data redundancy to be designed to recover from these failures.

[0072] As an example, the level 1 codeword may be the size of a page and protect from bit cell errors. The level 2 codeword may be the size of a page stripe and protect against data block failures. The level 3 codeword may be much larger and interleaved across die, packages, or card assemblies to protect against die, package, and card assembly failures. There is no requirement that codewords match any physical data boundary.

[0073] Each codeword has 3 primary components: payload, metadata, and ECC data.

[0074] The payload is the data received from the layer above. For the top layer, the payload is input data (either host or system data).

[0075] The second part is the metadata. The metadata describes the layout of the payload data, its addressing, boundaries, mapping, and its characteristics. The metadata field may not always be located at the end of the payload. It can be at the start, after ECC, or interspersed within the payload. The metadata is always covered by ECC.

[0076] The ECC field of a codeword contains parity or redundancy data to help correct for read errors. Any type of ECC may be used and, typically, different types are used at different codeword levels. The amount of ECC per codeword is also configurable. In general - especially at levels 2 and above, the ECC code used is systematic. A systematic ECC codeword does not encode the payload data and just adds calculated ECC. Some types of ECC that are supported include BCH, LDPC (Low-Density Parity-Check), turbo codes, and Reed-Solomon. Others can be used as well. Systematic codewords allow the payload to be written to flash in parallel with ECC calculation.

[0077] The partition parameters allow the controller to determine the codeword structure. The codeword structure changes by partition definition. The partition definition along with its physical mapping is embedded into the partition as metadata at the partition start and at intervals within the partition.

[0078] Codewords do not have to map in integer numbers into the payload of the next higher layer. The metadata in the next higher layer allows the controller to pack/unpack codewords across payload boundaries - as shown in Figure 10. Additionally, higher layer codewords can be shortened in order to fit into a page, page stripe, or partition boundary.

[0079] In this example embodiment, two data blocks 1010 and 1011 from a host system are to be stored using three levels of codewords. In this example, data blocks 1010 and 1011 are the same size as level 3 payloads 1020 and 1023, however there is no requirement for any such size mapping.

[0080] Based on level 3 payload 1020, the storage controller creates metadata 1021 and computes ECC 1022. Similarly, based on level 3 payload 1023, the storage controller creates metadata 1024 and computes ECC 1025. The combination of a level 3 payload with metadata and ECC data comprises a level 3 codeword 1026.

[0081] Each level 3 codeword is then partitioned into level 2 payloads. In this example, a portion of level 3 payload 1020 becomes level 2 payload 1030, the remaining portion of level 3 payload 1020 and metadata 1021 becomes level 2 payload 1033, and ECC 1022 and a portion of level 3 payload 1023 becomes level 2 payload 1036. This process continues until all of the level 3 codewords have been partitioned into level 2 payloads.

[0082] Based on level 2 payload 1030, the storage controller creates metadata 1031 and computes ECC 1032. Similarly, based on level 2 payload 1033, the storage controller creates metadata 1034 and computes ECC 1035, and based on level 2 payload 1036, the storage controller creates metadata 1037 and ECC 1038. The combination of a level 2 payload with metadata and ECC data comprises a level 2 codeword 1039.

[0083] Each level 2 codeword is then partitioned into level 1 payloads. In this example, a portion of level 2 payload 1030 becomes level 1 payload 1040, the remaining portion of level 2 payload 1030 becomes level 1 payload 1043, metadata 1031 and ECC 1032 become level 1 payload 1046, and a portion of level 2 payload 1033 becomes level 1 payload 1049. This process continues until all of the level 2 codewords have been partitioned into level 1 payloads.

[0084] Based on level 1 payload 1040, the storage controller creates metadata 1041 and computes ECC 1042. Similarly, based on level 1 payload 1043, the storage controller creates metadata 1044 and computes ECC 1045, based on level 1 payload 1046, the storage controller creates metadata 1047 and ECC 104, and based on level 1 payload 1049, the storage controller creates metadata 1050 and ECC 1051. The combination of a level 1 payload with metadata and ECC data comprises a level 1 codeword 1052.

[0085] The larger data block sizes and layered codeword design allow customized ECC protection schemes that better fit an array, its traffic model, and potential points of failure. These longer, layered codewords provide stronger data protection with less overhead. The schemes can also be tailored to protect against and recover from physical failures such as die, package, and assembly failures. [0086] The ECC scheme can be beefed up as much as needed to reduce the UBER (Uncorrectable Bit Error Rate) below the theoretical UBER of any competing storage technology or system. The improved ECC scheme also allows the use of lower grade, less expensive NAND in high reliability applications.

[0087] Beyond managing the partition configuration, memory array usage, and codeword encode/decode described in the previous sections, the memory controller requires other functionality and algorithms to provide a full controller solution.

[0088] The memory system needs to store non-user data in order to boot, configure, and operate. This data is referred to as system data and is normally not available through normal write / read commands via the host interface.

[0089] Examples of system data include internal firmware, manufacturing data, translation and management tables, statistics, array information, and the like. System data is generated internally and written to partitions that are created and configured specifically for system data. There can be as many system partitions as needed.

[0090] Figure 11 illustrates an example sequential writing process. Write requests to the storage array are mapped to a partition. Write data can be steered to different partitions based on data type, host ID, or data source.

[0091] The storage controller monitors traffic from all active partitions. Any writes competing for shared resources (such as a channel or die), are prioritized based on partition attributes queuing it for write to the storage array. Within a partition, the data is mapped into the payloads of the different levels of codewords.

[0092] Each time enough data is available from codeword generation to write a page, the data is transferred to the correct die and a page program command is sent. Each page in the storage array may include some metadata at a fixed position in the page to help locate codeword boundaries for read-back. [0093] In this example, the level 1 codewords from Figure 10 are to be stored in block stripe 1100. Block stripe 1100 includes a N by M array of pages. It includes page stripes 0- N (1111-1113). Each page stripe comprises pages 0-M.

[0094] The storage controller partitions the level 1 code words into page-sized chunks of data and sequentially stores these chunks of data into the pages of a page stripe. Once the page strip is full, it moves on and begins storing chunks of data into the pages of the next sequential page stripe.

[0095] In this example, a portion of level 1 payload 1040 is stored in page 0 of page stripe 0 1111, the remainder of level 1 payload 1040 and metadata 1042 is stored in page 1 of page stripe 0 1111, and the remaining level 1 codewords are partitioned and sequentially stored in the pages within page stripe 0 1111 until reaching page M of page stripe 0 1111. At that point the storage controller begins storing the level 1 code words in page stripe 1 1112, and continues to fill block stripe 1100 in this manner.

[0096] For systematic codewords, the ECC for each level can be accumulated in memory as payload data is written to the array in pages. As the metadata and ECC is generated, it is also written.

[0097] For non-systematic codewords, the entire codeword has to be assembled prior to writing it to the array. Codeword assembly includes all levels.

[0098] The partition is filled in page stripes across the set of data blocks designated as a data block stripe within the partition.

[0099] The write process incorporates a flush operation. There are two types of flush operations: mini and full.

[00100] A mini-flush is used to protect buffered data. It updates the current open level 0 codeword metadata to describe the flush and pad out operation. It then pads out the level 0 codeword and then pads out the buffered data to the next legal write unit (a page) and writes it to the array. This is used for temporary interruptions in the data stream or in cases where only a short amount of time is available to write out the data (e.g. imminent power fail).

[00101] A full flush updates the metadata and pads out all levels of codewords. It may shorten some of the higher layers of codewords to save capacity. It then pads out the write data to the next desired boundary (page, page stripe, data block stripe, partition). This operation is normally used to close a partition or provide a cleaner array write when more time is available to clean up after an error condition that has triggered the flush.

[00102] Flush operations are triggered for multiple reasons. Examples include:

1. The host or emulation layer may request a flush at any point in the data stream either as part of the write command or as a separate command.

2. A programmable timer based on the partition attributes can be set to force a flush.

3. An imminent power fail signal or other hardware unsafe signal can also trigger a flush.

[00103] The write process provides an acknowledgement to the upper layers or host each time a specific amount of data has been safely written to the array. The units and size of the acknowledgement is configurable based on the partition. Some example options are every 10 level 0 codewords every 100 data blocks, every data block stripe, every write command, and the like.

[00104] The write process keeps track of data block addressing per partition during writes. Any duplicate or out of order data block writes are rejected. This tracking is cleared after an erase.

[00105] Both random and sequential reads are supported by the SWRR controller. The read requests are based on the data block address. Sequential reads are supported with a partition plus host data block address accompanied by a requested number of data blocks. [00106] Minimum sized physical reads are in pages at the memory array. Minimum sized SWRR reads are in level 0 codewords. Since codeword sizes are variable and can span pages, a minimum data block read may require more than one page to be read.

[00107] When a specific data block is requested, the SWRR controller determines the physical location of the data block and reads the physical memory page or pages and level 0 codeword(s) to recover it. The ECC is checked and if the data is good, it is returned. If it is not good, recovery operations are initiated.

[00108] The SWRR controller can support multiple read requests across multiple partitions at a time. The requests are mapped and queued to the array using priority rules from the partition definitions and optimized for access.

[00109] Erases are performed on a block stripe or partition basis. When a partition is erased, it can be removed, reconfigured, move physically, or remain as an empty partition. A secure erase may also be done to remove the data that was stored in the partition.

[00110] Data blocks wear as they are erased. Current NAND technology supports thousands or tens of thousands of erases before they can no longer retain data. The differences are dependent primarily on the process technology and data densities.

[00111] Wear leveling is important for NAND controllers to ensure that the array wears uniformly. Wear leveling is performed as part of the partition creation and free block pool management process within the SWRR controller. Erase counts for each data block or partition are maintained for the array and used during partition erasure, creation, and destruction to keep the data block wear relatively uniform across the array. Wear leveling is done via partition management for the SWRR controller instead of on a block basis. Wear leveling is accomplished by the memory controller by selecting data blocks for use in a partition at least in part on their erase counts. [00112] The media emulation layer above the SWFF controller adapts the controller to different operating modes, traffic models, command sets, and interface protocols.

[00113] For each media emulation type, the controller ensures that system level behaviors such as power loss, data reliability, capacity, and latency are adhered to.

[00114] For media emulation modes that support host traffic that doesn't match the SWRR model, the media emulation layer performs "traffic shaping" to adapt the traffic to fit the SWRR mode.

[00115] The major types of operating and traffic models supported include: Sequential Write/Random Read, Fixed, Numbered blocks emulating NAND flash, Optical - CD, Blu- Ray, BD-XL, and the like; Sequential Write/Sequential Read, Fixed, un-numbered blocks emulating tape media; Sequential Write/Random Read, Large variable blocks emulating object based, large data stores; and Random Write/Random Read, Small fixed blocks emulating hard drives and solid state drives.

[00116] For all modes, the media emulation layer manages the configuration and setup of partitions it uses. It also must manage any system level configuration associated with the media emulation layer and host interface. This management may also include creation and usage of system partitions.

[00117] If needed, the media emulation layer handles some configuration and status commands and/or translates commands and forwards them to the SWRR controller.

[00118] There may be more than one emulation object present in a controller at a time. Thus, a system can support multiple, mixed virtual devices and volumes at a time.

[00119] For those emulation modes that require data translation, mapping information as well as host based addressing and attributes need to be recorded with the data. The media emulation layers define their own metadata fields embedded into the emulation data blocks to perform the mapping. If needed or desired for performance, translation data tables that are stored separately in system partitions can be created to reduce translation times. However, the metadata translation is always defined such that the translation table can be lost and be reconstructed using the metadata.

[00120] Sequential Write / Random Read - Fixed Blocks

[00121 ] Traffic Shaping: Depending on partition configuration, the host data block size may not match the emulation data block size. If this is the case, the media emulation layer maps host blocks into emulation data blocks in both directions.

[00122] Data Translation: As needed, metadata and tables are used to allow the media emulation layer to translate between host block addresses and emulation data block addresses.

[00123] Interface Management and Command Translation: In SWRR mode, the media emulation layer does no command or data translation. The primary function of the media emulation layer is to provide the physical interface and link layer control to ensure an error free command and data transport.

[00124] Sequential Write / Sequential Read - Fixed Blocks

[00125] The primary media emulation targets fitting this emulation model include tape and optical devices.

[00126] Traffic Shaping: Depending on the partition configuration, the host data blocks may need to be translated to emulation data block sizes. Otherwise, there is very little traffic management. If desired, the media emulation layer may implement read ahead in anticipation of a sequential read stream.

[00127] Data Translation: Most sequential write/read devices use relative block locations on the media and not absolute block numbers. The media emulation layer in this case keeps track of relative location and assigns block numbers to the emulation data block read/write requests sent on to the controller. [00128] In addition, tape drives have media movement commands that move the relative medium position such as rewind, locate, and search. Along with those commands are filemark and tape mark writes to mark relative medium positions. The media emulation layer takes care of virtualizing these tape features.

[00129] The data translation function of the media emulation layer also supports multiple virtual volumes which are mapped to separate partitions.

[00130] Interface Management and Command Translation: Most tape drives use the

T10 streaming command set. Most optical disks use the T10 Multi-media command set.

Common physical interfaces are SAS and FC for tape and SATA and USB for optical Blu-

Ray.

[00131] Sequential Write, Random Read, Large Variable Blocks

[00132] Currently, no storage technologies fit this type of use model, but this is the model for very large data stores. Object based storage model is an example. Using this emulation mode, the SWRR controller can have a direct, higher level interface that hides the storage device type and characteristics from the upper layers.

[00133] Traffic Shaping: The native traffic mode of the SWRR controller matches this model. There is very little to do for traffic shaping other than to provide some protection of very large blocks being written from errors and power loss.

[00134] Data Translation: The media emulation layer breaks up the large, variable blocks and maps them into emulation data blocks. It uses metadata and system data tables to perform the mapping.

[00135] Interface Management and Command Translation: The command interface is very simple for object based storage and related protocols. This requires the media emulation layer to handle a lot of the configuration and management of the overall device

characteristics. [00136] Random Write / Random Read - Small Fixed Blocks

[00137] This mode fits hard disk drives and SSDs (Solid State Drives).

[00138] Traffic Shaping: For writes, the media emulation layer collects the small block random writes into a larger sequential write stream broken up into emulation data block sizes.

For reads, the emulation data blocks containing the target data are requested.

[00139] Data Translation: Converting the small block random write traffic is exactly what the FTL (Flash Translation Layer) function in a SSD does. Thus, the media emulation layer requires an FTL function to map the small block writes into emulation data blocks.

Along with the FTL comes all of the other associated functionality including garbage collection, wear leveling, write coherency, small block rewrite support, and power loss management.

[00140] Interface Management and Command Translation: HDDs and SSDs use the T10 block command set. Primary physical interfaces are iSCSL SAS, NVMe, SATA, and FC. As with the other modes, multiple drives may be emulated,

[00141] Since the SWRR controller spans multiple storage models and technology, this section provides a brief overview of other controllers. Available storage controllers include HDD, SSD, USB flash, eMMC (embedded Multi-Media Card) flash and UFS (Universal Flash Storage) flash. All of these controllers fit the random write/read, small fixed block mode. All of them support 512 byte blocks. Some support 4 KByte blocks.

[00142] HDD Controllers

[00143] This overview only refers to the datapath management portion of HDD controllers. HDD controllers also include servo and magnetic head control functions not referenced here.

[00144] Hard disk drive controllers are optimized for the magnetic spinning disk media that they control. The small, fixed size blocks are based on disk sectors that were defined decades ago. The sector size was based on the maximum time that servo and PLL systems could go without fixed patterns for synchronization as the media passes by the head.

[00145] When HDD systems were integrated into computers and memory systems, the command set was designed with the assumption that the sector size would never change. This assumption proliferated through driver software, file systems, and even data transfer protocols to the point where it has become difficult - if not impossible for the sector size to change due to the vast infrastructure.

[00146] If the HDD model could/did change to allow larger sectors, the advantages would include increased capacity due to less inter-sector overhead (a few %), increased capacity due to larger ECC codewords (a few %), less command overhead due to smaller numbers of blocks (slightly improved latency), and easier to scale up systems to larger sizes as addresses apply to more data.

[00147] The disadvantages of larger sector sizes include wasted space and

inefficiencies when writing and reading data that is smaller than a block size.

[00148] Small blocks are becoming much less common traffic in HDD workloads as flash devices take over the bus based and first tier storage positions. As HDD moves farther back into the storage hierarchy, the traffic profile shifts towards large block sequential.

[00149] Shingled HDD

[00150] A recent innovation in for HDD is shingled mode recording. In shingled recording, the write tracks overlap each other to increase the density by 20-30%. The cost of doing this is that the tracks must be written in order within a band of tracks. Reading is still random access.

[00151] This recording mode essentially turns a HDD into a sequential write, random read device which fits the controller of this example better than a classic HDD controller.

[00152] NAND Flash Controllers [00153] NAND flash controllers for USB, SD Card, eMMC, and all work as random write/random read, small block devices. For this example, small block is any block size that is smaller than a native page size. For NAND, this is 8-16KB.

[00154] These controllers all require what is commonly referred to as a small block FTL (Flash Translation Layer). The FTL is simply a mapping of the host based small blocks into the NAND pages. Implementation of a FTL adds a lot of complexity and requires additional processes to manage. These additional processes include the creation,

management, searching, and nonvolatile update of a very large translation table.

[00155] Garbage collection is another additional process. As random writes proliferate, the entire array becomes littered with valid and invalid host data blocks. Eventually, the array is fully written, but only portions of it contain valid data. In order to deal with this, valid data must be collected and moved to fresh pages and data blocks so that pages with a lot of invalid data can be erased and re-used. In effect, garbage collection requires data to be written to the array more than 1 time. This effect is called "write amplification".

[00156] Wear leveling is also required. Due to the write amplification caused by garbage collection, erases are required much more frequently to the point that block erase is the primary failure mechanism. Wear leveling of the blocks within the array is more important when using a FTL to extend the life.

[00157] Power fail recovery must also be provided for. With garbage collection operations in the background moving user data and updating the FTL in volatile memory, there is risk of loss of previously written data if there is power failure.

[00158] Power fail recovery with the SWRR controller is much simpler and more robust due to the elimination of garbage collection, addition of a write acknowledge function, and large FTL tables. The translation tables within the SWRR controller are all redundant with metadata and interlocked with write acknowledgement handshakes. [00159] Data protection is also an issue with a FTL. The random access performance requirements for these small block systems limit the levels and sizes of codewords allowable for ECC and data protection. Level 1 codeword protection is fixed to a page size and level 2 is fixed to a block stripe. These controllers do not employ any layers higher than level 2.

[00160] The SWRR controller does not require a random, small block FTL. Due to minimization of data block erases and the elimination of the large FTL table, the life of the storage array is increased in every case. The life increase can be very substantial depending on the traffic type. The complexity associated with a FTL was described previously.

[00161] Figure 12 illustrates storage controller 1200. As discussed above, storage controller 1200 may take on any of a wide variety of configurations. Here, an example configuration is provided for a storage controller implemented as an ASIC. However, in other examples, storage controller 1200 may be built into a storage system or storage array, or into a host system.

[00162] In this example embodiment, storage controller 1200 comprises host interface 1210, processing circuitry 1220, storage interface 1230, and internal storage system 1240. Host interface 1210 comprises circuitry configured to receive data and commands from an external host system and to send data to the host system. In some embodiments, host interface 1210 or processing circuitry 1220 may include a media emulation layer.

[00163] Storage interface 1230 comprises circuitry configured to send data and commands to an external storage system and to receive data from the storage system. In some embodiments storage interface 1230 may include ONFI ports for communicating with the storage system.

[00164] Processing circuitry 1220 comprises electronic circuitry configured to perform the tasks of a SWRR controller as described above. Processing circuitry 1220 may comprise microprocessors and other circuitry that retrieves and executes software 1260. Processing circuitry 1220 may be embedded in a storage system in some embodiments. Examples of processing circuitry 1220 include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof. Processing circuitry 1220 can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions.

[00165] Internal storage system 1240 can comprise any non-transitory computer readable storage media capable of storing software 1260 that is executable by processing circuitry 1220. Internal storage system 1220 can also include various data structures 1250 which comprise one or more databases, tables, lists, or other data structures. Storage system 1240 can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.

[00166] Storage system 1240 can be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system 1240 can comprise additional elements, such as a controller, capable of communicating with processing circuitry 1220. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and that can be accessed by an instruction execution system, as well as any combination or variation thereof.

[00167] Software 1260 can be implemented in program instructions and among other functions can, when executed by storage controller 1200 in general or processing circuitry 1220 in particular, direct storage controller 1200, or processing circuitry 1220, to operate as described herein for a storage controller. Software 1260 can include additional processes, programs, or components, such as operating system software, database software, or application software. Software 1260 can also comprise firmware or some other form of machine-readable processing instructions executable by elements of processing circuitry 1220.

[00168] In at least one implementation, the program instructions can include partitioning module 1262, media emulation module 1264, ONFI translation module 1266, error correction module 1268, SWRR translation module 1270, and host communication module 1272.

[00169] Partitioning module 1262 includes instructions directing processing circuitry 1220 to create, utilize, and maintain partitions within a storage system. Media emulation module 1264 provides a media emulation layer for processing circuitry 1220 and storage controller 1200. ONFI translation module 1266 translates SWRR commands into ONFI commands for use by a storage system. Error correction module 1268 computes error correction codes for storage with a payload in a codeword in a storage system. SWRR translation module 1070 translates host data and commands into data and commands usable by a storage system. Host communication module 1272 interfaces with a host system to provide host data and commands to storage controller 1200 for conversion into storage data and commands usable by a storage system.

[00170] In general, software 1260 can, when loaded into processing circuitry 1220 and executed, transform processing circuitry 1220 overall from a general-purpose computing system into a special-purpose computing system customized to operate as described herein for a storage controller, among other operations. Encoding software 1260 on internal storage system 1240 can transform the physical structure of internal storage system 1240. The specific transformation of the physical structure can depend on various factors in different implementations of this description. Examples of such factors can include, but are not limited to the technology used to implement the storage media of internal storage system 1240 and whether the computer-storage media are characterized as primary or secondary storage.

[00171 ] For example, if the computer-storage media are implemented as

semiconductor-based memory, software 1260 can transform the physical state of the semiconductor memory when the program is encoded therein. For example, software 1260 can transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation can occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate this discussion.

[00172] The included descriptions and figures depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the invention. Those skilled in the art will also appreciate that the features described above may be combined in various ways to form multiple embodiments. As a result, the invention is not limited to the specific embodiments described above, but only by the claims and their equivalents.