FIELD QF THE INVENTION

[0001] The invention relates generally to the field of storage systems in storage environments and more particularly, to enclosures which may employ a redundant array of independent disks (RAID) architecture.

DISCUSSION OF RRT ATED ART

[0002] Storage systemsin computing environments have become increasingly important, as the demand for storage space has grown exponentially with the arrival of the Internet and microprocessors capable of handling graphics, video and animation. In addition, demand for reliable storage resources to provide backup and availability in case of individual node failure has become increasing important. [0003] Figures 1 A-IB illustrate a front and back view of a prior art storage enclosure 100. Figure IA is a front view 101 of prior art storage enclosure 100. Single disk carriers 150 are shown on front of storage enclosure 100. Each one of these single disk carriers 150 contain one nonvolatile disk drive (not shown). Each single disk carrier is cascaded together to form a logical volume in storage enclosure 100. A series of individual nonvolatile storage devices ("members") are cascaded to form a single logical volume. A single logical volume is created when one or more nonvolatile storage devices are coupled together. According to the prior art enclosure 100, only one nonvolatile disk drive can be stored in each one of single disk carriers 150. If a user has filled up each single disk earner slot on storage enclosure 100 with one nonvolatile disk drive, then they must purchase anodier storage enclosure 100 to add capacity to their storage system. [0004] Each individual one of the single disk carriers 150 has a pair of indicators, a status indicator (e.g., status indicator 121) and an activity indicator (e g., activity indicator 122) showing a read/wnte/seek operation on a nonvolatile disk drive). These indicators correspond to their one nonvolatile disk drive within their particular single disk carrier 150 (e g.} single disk earner 1503 for indicators 121, 122). The status indicator indicates that its nonvolatile disk drive is turned on. The activity indicator indicates that a read, write, or seek command is being processed by its nonvolatile disk drive. If a nonvolatile disk dnve within a single disk earner fails, a user must disengage the single disk earner by pressing eject button 120. The single disk earner corresponding to the failed nonvolatile disk dnve is pulled outward from storage enclosure 100 after the eject button 120 is pressed. [0005] Figure IB is a rear view 102 of the pnor art storage enclosure 100. Typically, a storage enclosure has one or more power supply units 103, fans 104, and modules 105A and 105B. The power supply units 103 are redundant (e.g., more than one power supply per storage enclosure 100) to protect the storage enclosure 100 if one of the power supplies fail. Similarly, the fans 104 are redundant to protect against overheating if one of the fans 104 fail. Illustrated in Figure IB are four fans, however, typically there can be more or less fans and power supplies within a storage enclosure according to the prior art. Furthermore, fans 104 may be located within the power supplies. Modules 105A and 105B typically contain semiconductor chips (e.g., microprocessor and memory) for controlling the input/output (I/O) instructions to and from storage enclosure 100. The modules 105A and 105B are accessible from the rear of storage enclosure 100 according to the prior art. If one of the modules 105A or 105B fails, a user may have to disconnect storage enclosure 100 and replace the modules 105A and 105B from the back of storage enclosure 100. SUMMARY [0006] An enclosure which comprises a set of removable canisters, a set of capsules within each canister, and at least one nonvolatile storage device diat resides within each canister. A removable top on the enclosure provides individual access to the capsules, in one exemplary embodiment. Canisters may also be removed from the front of the enclosure, so as to allow dual access to a nonvolatile storage device. BRIEF DESCRIPTION OF THE DRAWINGS [0007] The present embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements. [0008] Figures 1A-1B illustrate a front and back view of an exemplary prior art storage enclosure 100; [0009] Figures 2A-2B illustrate a front and back view of an exemplary storage enclosure 200; [OOIO] Figures 3A-3B illustrate cross-sectional views of an exemplary enclosures 200 and 300, and placement of modules; [0011] Figures 4A-4B illustrate an exemplary canister 25Oi that can store four nonvolatile storage devices. [0012] Figures 5A-5B illustrate an exemplary canister 550 that can store up to 16 nonvolatile storage devices. [0013] Figures 6 illustrates an exemplary three dimensional view of a storage enclosure 600 in which a canister 550 is removed from the front of storage enclosure 600; and [0014] Figures 7 illustrates a three dimensional view of an exemplary storage enclosure 700 in which a canister 550 is removed from the front of storage enclosure 700, and a capsule 504 within a canister 550 is removed {torn the top of storage enclosure 700. DETAILED DESCRIPTION [0015] A storage enclosure, canister, and flexible connector line for providing singularly removable, multiple nonvolatile storage devices within a single canister are described. Note that in this description, references to "one embodiment" or "an embodiment" mean that the feature being referred to can be included in at least one embodiment of the teachings described herein. Further, separate references to "one embodiment" in this description do not necessarily refer to the same embodiment; however, neither are such embodiments mutually exclusive, unless so stated and except as will be readily apparent to those skilled in the art. [0016] Figures 2A-2B illustrate a front and back view of a storage enclosure embodiment 200. Figure 2A shows a front view 201 of a storage enclosure 200. The storage enclosure 200 includes a group of canisters 250, each containing a plurality of nonvolatile storage devices (a cross sectional view of the nonvolatile storage devices 401 within an exemplary canister having four nonvolatile storage devices 401 is illustrated in Figure 4A). In one embodiment, the nonvolatile storage devices are serial ATA (SATA) hard drives. In another embodiment, the nonvolatile storage devices are parallel ATA (PATA) hard drives. The overall total cost of ownership should be lower as compared to the prior art enclosure because the cost of the power supplies, fans 204, and modules 205A and 205B are amortized over a larger number of nonvolatile storage devices in some cases (e.g., a user doesn't have to buy as many enclosure systems to support a required number of hard drives because one canister can support a plurality of hard drives). In one embodiment, the power supplies are 400W to 700W assuming that three power supplies are used in storage enclosure 200. Furthermore, there should be a reduced cost per slot (e.g., where a slot includes a connection point to a hard drive) in the embodiment shown in Figure 2 because more individual nonvolatile storage devices can be accommodated within a single storage enclosure 200 in some cases. In the exemplary embodiment of Figure 2A, each canister includes a unique status indicator for each hard drive and a unique activity indicator for each hard drive (e.g., canister 2504 has four status indicators 221A-D and four unique activity indicators 222A-D). (Here, note that Figure 2A would therefore correspond to a canister having four hard drives as depicted in Figure 4A.) [0017] Figure 2B is a rear view 202 of die storage enclosure 200 of Figure 2A. The storage enclosure 200 comprises one or more power supply units 203, fans 204, and modules 205A and 205B. The power supply units 203 are redundant (e.g., more than one power supply per storage enclosure 200) to protect the storage enclosure 200 if one of the power supplies fail. Similarly, the fans 204 are redundant to protect against overheating if one of the fans 204 fail. Illustrated in Figure 2B are four fans, however, just as in Figure IB, typically there can be more or less fans within a storage enclosure. Modules 205A and 205B typically contain semiconductor chips (e.g., a microprocessor, memory, ASICs, etc.) for controlling the input/ output (I/O) instructions to and from storage enclosure 200. [0018] In one embodiment, the modules 205A and 205B are redundant, and each module separately can control operations of all die canisters 250. In another embodiment, die modules 2O5A and 205B each control half the canisters 250 within each storage enclosure 200. The modules 205A and 205B are accessible from the rear of storage enclosure 200, as illustrated in Figure 2B, and/or from the front of storage enclosure 200 (front access of modules will be discussed in greater detail later in Figure 3B). [0019] In one embodiment, if one of die modules 205A or 205B fail, a user may be able to remove and replace the modules 205A or 205B without bringing down the storage enclosure 200 (e.g., the modules 205A and 205B may be hot-swappable). In another embodiment, the power supplies, fans, and canisters may also be hot-swappable (e.g., can be removed without powering down the device ot causing significant disruptions in quality of service). [0020] Figures 3A-3B illustrate cross-sectional views of two different exemplary enclosures 200 and 300. Figure 3A illustrates a cross-sectional view of a storage enclosure 200 in which modules 205A and 205B are located at the back 202 of die storage enclosure 200. As a result of modules 205A and 205B being in the back 202 of die storage enclosure 200, a number of individual canisters 250 (fourteen in particular) are shown that encompass the front 201 of the storage enclosure 200 (see Figure 2A for a front view of die storage enclosure 200). In one embodiment, height of 4U (IU = 1.75 inches) (a U is a standard unit of height in die telecommunications industry) is used for the storage enclosure 200. In another embodiment, the storage enclosure 200 is 19 inches wide (e.g., length of 201) and 26 inches deep. [0021] The cross-sectional view shown in Figure 3A illustrates four individual capsules 304 within each canister 250i. Each capsule 304 contains a nonvolatile storage device 401 as will be later discussed in Figure 4A and Figure 7. In one embodiment, each capsule provides support/ capability for high capacity and highRPM (rotations per minute) drives. A data tub is a very large storage device tihat may provide terabytes of storage space for users. A data tub can be provided in Figure 3A by filling each canister 250i with high capacity hard drives. High rpm drive capacity can be provided by filling the capsules 304 within each canister 25Ot with hard drives that operate at a fast rpm (e.g., currendy 10,000 to 15,000 rpm for a drive is considered fast, however, in the future this rpm range may increase). In anodier embodiment, a storage enclosure 200 can accommodate canisters optimized for drive RPM and for drive capacity. [0022] In one embodiment, each canister is removable from the front 201 of storage enclosure 200 (e.g., a user can press an eject button 220 on a canister and physically pull outward the canister from storage enclosure 200). When the eject button for a particular canister (e.g., eject button 220 for canister 2504) is pressed by a user, the individual nonvolatile storage devices 401 within each capsule 304 of the canister are removed from power, and die activity (read/write/seek) on each nonvolatile storage device 401 is terminated. [0023] In a further embodiment, when a canister is removed, current activity is completed or quiesced (e.g., finished active input/ output) before the canister is ejected (e.g., if a read or write operation is currently being performed on one of the nonvolatile storage devices 401 within a canister, that operation is completed before the canister is ejected). The canister may be removed by operation of an electromechanical switch that ejects the canister only when the activity on a nonvolatile storage device 401 within the canister is terminated, and when it is safe to remove a nonvolatile storage device 401 without risk of interrupting active I/O or losing data. [0024] The backplane 305 may be a printed circuit board onto which each of the canisters 250 connect. In one embodiment, a perpendicular arm 402 (see Figure 4A for a graphical view of an embodiment of this perpendicular arm 402) widiin each canister connects its canister to the backplane 305. The perpendicular arm may be a printed circuit board (PCB) in one embodiment. The connector 405 is designed to support the appropriate data throughput. In one embodiment, a connector 405 through which high speed serial data may flow (e.g., 6 gigabit per second serial ATA) on a perpendicular arm 402 is used to connect the canister to the backplane 305. In die case of eight or sixteen canister embodiments, as will later be discussed in Figure 5A, a capsule 504 may also connect onto a perpendicular arm 502 through a serial ATA connector 505 as shown later in Figure 5A. In further embodiments, each capsule 304 is individually removable from nonvolatile storage device 200 without removing die canister (e.g., a capsule 304 is removed from the top of nonvolatile storage device 200 as illustrated by the capsule 504 eject in Figure 7). [0025] In one embodiment, die modules 205A and 205B contain microprocessors with software that determine the data I/O management and storage within each nonvolatile storage device 401 wiriiin the storage enclosure 200 (e.g., a microprocessor and software witiiin module 205A and/or 205B determines which sector within a particular nonvolatile storage device 401 to store a particular block of data received from the network). In alternative embodiments, other types of semiconductor circuits may be used such as an ASIC or FPGA, or hardwired standard offering. Figure 3A also illustrates redundant power supply units 203 and fans 204, as previously discussed with reference to Figure 1 and Figure 2A. [0026] Figure 3B illustrates a cross-sectional view of a storage enclosure 300 in which modules 305A and 305B are observable from the front 301 of the storage enclosure 300. In one embodiment, storage enclosure 300 is similar to storage enclosure 200 in Figure 3A except that die modules 305A and 305B are observable from the front 301 of storage enclosure 300 radier than from die back 202 as illustrated in Figure 3A. In one embodiment, modules 305A and 305B are similar to modules 205A and 205B in terms of functionality, but with a different form factor (e.g., a different physical si2e to account for the difference in the lengdi of the storage enclosure 300 near the power supply units 203 and behind die backplane 305 verses in front of die backplane parallel to die canisters 250). In one embodiment, similar to Figure 2A, die standard telecommunications industry height of 4U (IU = 1.75 inches, a U is a standard unit of height in the telecommunications industry) is used for die storage enclosure 300 (and the canisters 250). [0027] Unlike Figure 2A5 the storage enclosure 300 can be shorter in depth because the modules 305A and 305B are at the front. As such, in one embodiment the enclosure 300 illustrated in Figure 3B is 19 inches wide (e.g., width of 301) and 22 inches deep as compared to 19 inches wide and 26 inches deep in the embodiment shown in Figure 3A. For rack spaces that prefer a smaller depth, the enclosure 300 should be useful because it is shorter in depth. In the embodiment shown in Figure 3B, if module 305A or 305B fail, a user may eject the module outward from the storage enclosure 300 by pressing a module eject button (not shown) on the front 301 of storage enclosure 300. Furthermore, indicators on the modules (e.g., showing activity, memory usage, power status, etc.) can be viewed from the front of storage enclosure 300, rather than from the back. In one embodiment, the module eject button is similar to the canister eject button 220. [0028] Signal integrity should be well maintained for both Figure 3A and Figure 3B (e.g., noise should be reduced). A connector access board 306 allows connections extending from module 305A and 3O5B to pass through storage enclosure 300 to external connections in the network (not shown). Odier embodiments of storage enclosure 300 are similar to as described with reference to Figure 3A. [0029] Figures 4A-4B illustrate a canister that can store four nonvolatile storage devices 401. In one embodiment, the four nonvolatile storage devices 401 are 3.5" hard drives. Each of the four nonvolatile storage devices are shown as placed within a canister 250. The four nonvolatile storage devices 401 may be any capacity nonvolatile storage device. In one embodiment, at least two nonvolatile storage devices 401 are logically grouped together to form a single member within a RAID system (e.g., multiple nonvolatile storage devices 401 appear to be one member onto which blocks of data can be stored by computers coupled to storage enclosure 200). Semiconductor circuitry and/or software configuration within a module may make a determination of how the individual nonvolatile storage devices 401 are grouped together in a RAID system. [0030] In Figure 4A, the four nonvolatile storage devices 401 connect to arm 402 through a nonvolatile storage device connector 403 on each nonvolatile storage device 401. Figure 4A also illustrates that the perpendicular arm 402 extends from the canister outward into backplane 305. In one embodiment, the perpendicular arm 402 does not extend outward into backplane 305, but radier is flush to the backplane 305. [0031] The perpendicular arm 402 may contain logic circuitry for translating parallel nonvolatile storage device 401 to/from a serial data stream that flows through a serial connector 405. Connector 405 connects uie perpendicular arm 402 to the backplane 305. In one embodiment, the connector 405 is a high speed serial ATA connector as illustrated in Figure 4B. The high speed serial ATA connector may operate at a transfer rate of 6 gigabits per second or higher in one embodiment. [0032] Figures 5A-5B illustrate a canister 550 that can store 16 nonvolatile storage devices. Canister 550 in Figure 5A is similar to the canister shown in Figure 4A except that canister 550 is designed for accommodating sixteen nonvolatile storage devices. In an embodiment, the sixteen nonvolatile storage devices may be 2.5" hard drives. The sixteen nonvolatile storage devices may be any capacity nonvolatile storage device that fit with the appropriate form factor. In another embodiment, there may be only four nonvolatile storage devices which may be 2.5" hard drives. In another embodiment, one row of eight drives may be within each canister. There also exists a vertical beam for each capsule (i.e., each capsule has more than one hard drive). Here each of four vertical beams 506 connect four nonvolatile storage devices to perpendicular arm 502 (e.g., so that more than one nonvolatile storage device can be removed from a storage enclosure 700, as shown in Figure 7, by vertically detaching a capsule 504 from the perpendicular arm 502). [0033] Each vertical beam 506 includes a connector 505 that connects the vertical beam 506 to the perpendicular arm 502, In another embodiment, the connector 505 is a high speed serial ATA connector as illustrated in Figure 5B. The high speed serial ATA connector may operate at a transfer rate of 6 gigabits per second or higher in one embodiment. Furthermore, capsule 504 in Figure 5A is similar to capsule 404, as described in Figure 4A, except that capsule 504 contains multiple nonvolatile storage devices in Figure 5A rather than one. As such, when a user removes a capsule 504 from a storage enclosure, multiple nonvolatile storage devices are removed. Other embodiments of the canister 550 are similar to the canister 25Oi as described in Figure 4A. [0034] In one embodiment, at least two nonvolatile storage device 501 are logically grouped together to form a single member within a RAID system. Semiconductor circuitry and/ or software configuration within a module may make a determination of how the individual nonvolatile storage devices are grouped together in a RAID system in another embodiment. The modules may be customized for the appropriate number of nonvolatile storage devices per canister (e.g., a different module for sixteen hard drives per canister embodiment than for a four hard drive per canister embodiment). Other embodiments are similar to those described in Figure 4A and 4B. [0035] Figure 6 illustrates a three dimensional view of a storage enclosure 600 in which a canister 550 is removed from the front of storage enclosure 600. The front of storage enclosure 600 is shown as 601 and the back is shown as 602. In the embodiment illustrated in Figure 6, the nonvolatile storage device 501 within each canister 550 can only be removed from the front of storage enclosure 600. As such, a user must remove a set of eight nonvolatile storage devices 501 within the canister 550 in order to remove or replace a defective nonvolatile storage device 501 within a canister 550. An eject button, similar to the eject button 220 shown in Figure 2A, may be pressed by a user to deactivate a particular canister that a user wishes to deactivate. The power supplies, fans 204, and modules 205A and 205B are similar to those as discussed with reference to Figure IB and Figure 2B. [0036] Figure 7 illustrates a three dimensional view of a storage enclosure 700 in which a canister 550 can be removed from die front of storage enclosure 700 and a capsule 504 widiin a canister 550 can be removed from the top of storage enclosure 700. A set of rails 730 are placed on either side of storage enclosure 700 so that the storage enclosure 700 can easily slide outward from a rack of storage or networking equipment. Unlike the storage enclosure 600 as illustrated in Figure 6, the storage enclosure in Figure 7 can open from the top 735, and each individual capsule 504 within a canister 550 can be individually removed. [0037] A user can sEde out a storage enclosure 700 from a rack space by pulling the storage enclosure 700 forward and sliding forward on rails 730. After the storage enclosure 700 is slid forward, a user can manually open the top 735 of storage enclosure 700. A user can then pull on a capsule from the top of the storage enclosure 700 to lift a particular set of nonvolatile storage device 501 and its associated vertical beam 506 out of storage enclosure 700. [0038] If there is only one nonvolatile storage device per capsule (as illustrated in Figure 4A), then a user may directly pull out a nonvolatile storage device since no vertical beam is present. In another embodiment, a set of handles (not shown) located at the top of a capsule help a user to remove the nonvolatile storage devices (e,g., a user can remove a capsule by holding a handle and pulling the handle to Eft a capsule). As such, a user does not have to disconnect all nonvolatile storage devices within a canister if they wish to only remove a single capsule of hard drives. [0039] In one embodiment, the top 735 slides backward as illustrated in Figure 7. In another embodiment, the top 735 opens from a pivot at the back of storage enclosure 700 similarly to the hood of a car. [0040] Each canister 550 includes a plurality of status indicators 721A-G and a plurality activity indicators 722A-G (e.g., illustrated in Figure 7 are eight status indicators and eight activity indicators). In the exemplary view in Figure 1, there are fourteen canisters 550 in an enclosure 700 (as illustrated in Figure T). Each nonvolatile storage device 501 within a canister 550 is associated with one of the status indicators 721A-G and one of the activity indicators 722A-G (e.g.5 the canister 550 shown in the exemplary view in Figure 7 includes eight sets of the indicators because a canister 550 may contain eight nonvolatile disk drives). Other embodiments of Figure 7 are similar to those described in Figure 5 and 6. [0041] It should be noted that various embodiments as described are not intended to limit the invention in any way. Numerous other embodiments are possible that are not explicitly defined in this specification but are within the scope of the present invention. For example, the storage enclosure may be applied to a variety of storage mediums and networking devices including, but not limited to, storage systems, RAID enclosures, high-performance external storage controllers, and Serial ATA storage subsystems, according to various embodiments of the present invention. In addition, canisters may have alternate physical configurations, and canisters may contain a different number of individual nonvolatile storage devices than what may be illustrated herein. As such, the figures and accompanying description is intended to be merely illustrative and not limiting onto the scope of die present invention.