SYSTEM AND APPARATUS FOR REAL-TIME PROCESSING OF MEDICAL IMAGING RAW DATA USING CLOUD COMPUTING

[0001] The computing process (image formation from raw data) in medical imaging has become increasingly complex with high computational demands. The computational equipment (in general a single computer) deployed with clinical imaging systems is often inadequate for achieving clinically practical image reconstruction times. The hardware has to be specified and tested years before the imaging devices are deployed and the hardware is consequently obsolete before it is deployed. After the imaging data is acquired, one or more processing steps are performed to reconstruct an image using the imaging data. Conventional processing systems are located proximate to the image acquisition hardware (e.g., ultrasonic scanner, magnetic resonance imaging (MRI), and computed tomography (CT)). This processing raw image data may be quite intensive in nature, and may require significant processing capability.

[0002] Aspects of the present system and apparatus eliminate the need to deploy data processing hardware locally by deploying the image reconstruction hardware and software in a (remote) cloud computing system. The powerful computational resources available in commercial cloud systems can be leveraged directly in modem medical imaging devices and by doing so, the image processing hardware is replaced by a flexible software component, which can be scaled on-the-fly to match modem algorithms and which can be serviced and deployed remotely. For the end-user of imaging device, e.g. the physician or technician operating the scanner, they will feel no differences than if the computation was performed locally. We have demonstrated that this is possible with clinical MRI systems and using this paradigm, image reconstruction can be sped up by an order of magnitude for some applications.

[0003] Additional detail and description of the functionality of the system and apparatus for managing and/or processing of medical imaging raw, according to aspects of the present invention, is provided in the attached Appendices.











Magnetic Resonance in Medicine

Apprendix 2

Cloud Computing

SCHOLA ONe ^™ Ma uscri ts mUc Resonance in M&dM Page 1 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3

4

5

6

7 Distributed MRI Reconstruction using

9

10

11 Gadgetron based Cloud Computing

12

13

14

Hui Xue ¹ , Souheil Inati ² , Thomas Sangild Serensen ³ , Peter Kellman ⁴ , Michael S. Hansen ¹

17

18 Magnetic Resonance Te iiiulogs Program, National Heart, Lung and Blood Institute, National 19 Institutes of Health, Bethesda, D, USA

20

21 National Institute of Mental I lealth. National Institutes of Health, Bethesda, MD, USA

22

23 3 Department of Computer Science and Department of Clinical Medicine, Aarhus University, Aarhus,

Denmark

Medical Signal and Image Processing Program, National Heart, Lung and Blood Institute, National

Institutes of Health, Bethesda, MD, USA

Corresponding author:

Hui Xue

Magnetic Resonance Technology Program

National Heart, Lung and Blood Institute

41 National Institutes of Health

42

10 Center Drive, Bethesda

43

44 MD 20814

45 USA

46

47 Phone: +1 (301) 496-3052

48

49 Cell: +1 (609) 712-3398

50 Fax: +1 (301) 496-2389

⁵ Email: hui.xue(¾nih.gov

52 ^ ^fa —

53

54 Word Count: 5,970

55

56

57

58

59

60

, . 1 . _x , .. ..

fctegneue Resonance in Meamm Magnetic Resonance in Medicine Page 2 of 38

Apprendix 2

Abstract

Purpose:

To expand the Open Source Gadgetron reconstruction framework to support distributed

computing and to demonstrate that a multi-node version of the Gadgetron can be used to

provide non-linear reconstruction with clinically acceptable latency.

Methods:

The Gadgetron framewoi was extended with new software components that enable an

arbitrary number of Gadgetron instances to collaborate on a reconstruction task. This cloud- enabled version of the Gadgetron was deployed on three different distributed computing

platforms ranging from a heterogeneous collection of commodity computers

commercial Amazon Elastic Compute Cloud. The Gadgetron cloud was used to provide nonlinear, compressed sensing, reconstruction on a clinical scanner with low reconstruction

latency for example cardiac and neuro imaging applications.

Results:

The proposed setup was able to handle acquisition and 11-SPIRiT reconstruction of nine high

temporal resolution real-time, cardiac short axis cine acquisitions, covering the ventricles for

functional evaluation, in under one minute. A three-dimensional high-resolution brain

6

. . . . 3 .

acquisition with 1 mm isotropic pixel size was acquired and reconstructed with non-linear

reconstruction in less than five minutes.

Conclusion:

, . 2 . _x , .. ..

fctegneue Resonance in Meamm Page 3 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3 A distributed computing enabled Gadgetron provides a scalable way to improve

4

5

6 reconstruction performance using commodity cluster computing. Non-linear, compressed

7

8 sensing reconstruction can be deployed clinically with low image reconstruction latency.

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

Magm c Res©sar¾ce in Uedlc ^' ni® Magnetic Resonance in Medicine Page 4 of 38

Apprendix 2

Key words:

Gadgetron

Distributed computing

Non-linear MRI reconstruction

Open-source sofr¾are _:

Running title:

Gadgetron Cloud Computing

MagrseUc Resonance in Page 5 of 38 Magnetic Resonance in Medicine

Apprendix 2

Introduction

reconstruction algorithms are an essential part of modern medical imaging devices. The

9^ complexity of the reconstruction software is increasing due to the competing demands of

12 improved image quality and shortened acquisition time. In the field of MR imaging, in 13

1 particular, the image reconstruction has advanced well beyond simple fast Fourier transforms 15

^ to include parallel imaging (1 -5), non- linear reconstruction (6,7) and real-time reconstruction

19 (8,9). The increased interest in 3D acquisitions and non-Cartesian sampling has increased the 20

21 computational demands further. For applications where lengthy acquisition and reconstruction ₂₂

23 time is prohibited by physiological motion, biological processes, or limited patient 24

25

26 cooperation, the reconstruction system is under further pressure to deliver images with low

27

28 latency in order to keep up with the ongoing study.

29

30

31

32

33 While the need for fast image reconstruction is growing, most published reconstruction 34

35 algorithms, especially those relying on iterative solvers, such as image domain compressive 36

37

3 _Q sensing (6,10-12) and A-space SPIRiT and its regularized versions (7,13,14), do not come with

39

40 the efficient reference implementations that would enable clinical use. In many cases, the

41

42 algorithms are not implemented for online use on clinical scanners. I- ven if the developers 43

44

45 would like to integrate their reconstruction algorithms for online use, the vendor provided

46

47 hardware and software platform may have inadequate specifications for a demanding

48

49

j- _Q reconstruction or the available programming window may be unsuited for integration of new

51

52 reconstruction schemes. Consequently, there is a gap between the number of new algonthms

53

54 being developed and published and the clinical testing and validation of these algorithms.

55

56

57

58

59

60 Magnetic Resonance in Medicine Page 6 of 38

Apprendix 2

Undoubtedly, this is having an impact on the clinical adoption of novel non-linear

reconstruction approaches (e.g. compressed sensing).

We have previously introduced an open-source platform for medical imaging reconstruction

algorithms called the Gadgetron (15), which aims to partial address the above-mentioned

concerns. This platform is freely available to the research community and industry partners. It

is platform independent and flexible for both prototyping and commercial development.

Moreover, interfaces to several commercial MR platforms have been developed and are being

shared in the research community. This simplifies the online integration of new reconstruction

algorithms significantly and the new algorithms in research papers can be tested in clinical

settings with less implementation effort;! ^■ iks a result, some groups have used Gadgetron for

online implementation and evaluation of thei r reconstruction methods (16-18). Since the

publication of the first version of the Gadgetron, the framework has adopted a vendor

independent raw data format, the IS VI KM Raw Data ( ISMR VIk : ) ) format (19). This further

enables sharing of reconstruction algorithms.

While this concept of an open-source platform and a unified ISMRMRD format shows great

potential, the original Gadgetron framework did not support distributed computing across

multiple computational nodes. Although the Gadgetron was designed for high performance

(using multiple cores or GPU processors), it was originally implemented to operate within a

single node or process. Distributed computing was not integral to the design. As

reconstruction algorithms increase in complexity they may need computational power that

would not be economical to assemble in a single node. The same considerations have led to

B

the development of commodity computing clusters where a group of relatively modest fctegneue Resonance in ¾8e<Sieso© Magnetic Resonance in Medicine

Apprendix 2 computers are assembled to form a powerful computing cluster. An example of such a cluster system is the National Institutes of Health Biowulf Cluster (htt -Jib iowulf. n ih . gov) . Recently commercial cloud based services also offer the ability to configure such commodity computing clusters on demand and rent them by the hour. The Amazon Elastic Compute Cloud (EC2) is an example of such a service (http ;//aws ,^

In this paper, we propose to extend Gadgetron framework to enable cloud computing on multiple nodes. With this extension (named "Gadgetron Plus or GT-Plus"), any number of Gadgetron processes can be started at multiple computers (referred to as 'nodes') and a dedicated inter-process controlling scheme has been implemented to coordinate the Gadgetron processes to run on multiple nodes. A large MRI reconstruction task can be split and run in parallel across these nodes. This extension to distributed computing maintains the original advantages of Gadgetron framework. It is freel available and remains platform independent. As demonstrated in this paper, the nodes can even run different operating-systems (e.g. Windows or different distribution of Linux) and have di ferent hardware configurations.

The implemented architecture allows the user to set up a GT-Plus cloud in a number of different ways. Specifically, it does not require a dedicated professional cloud computing platform. The GT-Plus cloud can be deployed on setups ranging from an arbitrary collection of networked computers in a laboratory (we refer to this as a "casual cloud") to high end commercial cloud systems, as demonstrated in the following. In this paper, we demonstrate the GT-Plus cloud set up in three different scenarios and demonstrate its flexibility to cope with variable computational environments. The first cloud setup is a "casual cloud", consisting of seven personal computers situated in our laboratory at the National Institutes of Magnetic Resonance in Medicine Page 8 of 38

Apprendix 2

1

2

3 Health. The second setup uses NIH's Biowulf Cluster (20). The last configuration is deployed

4

5

g on the commercial Amazon Elastic Compute Cloud (Amazon EC2) (21).

7

8

9

10

11 To demonstrate the benefits of this extension, we used the cloud enabled version of the

12

13 Gadgetron to implement nonlinear 11 -SPIRiT (7) for 2D time resolved (2D+t) and 3D imaging

14

15 applications. We demonstrate that cardiac cine imaging with 9-slices covering the entire left

16

17

18 ventricle (32 channels, acquired temporal resolution 50ms, matrix size 192x 100, acceleration

19

20 factor 5, ~1.5s per slice) can be reconstructed with 11-SPIRiT with a reconstruction time

21

22 (latency <30s) that is compatible with clinical workflow. Similarly we demonstrate high

23

24

25 resolution 3D isotropic brain scans (20 channels, matrix size 256 ^x 256 ^x 192, acceleration

26

27 factor 3x2, 1mm isotropic acquired resolution), can be reconstructed with non- linear

28

29

reconstruction with clinically acceptable latency (<2.5mins). For both cases, significant 30

31

32 improvement of image quality is achieved with non-linear reconstruction compared to the

33

34 linear GRAPPA results.

35

36

37

38

39 In the following sections, details of GT-Plus design and implementation are provided The

40

⁴¹ referral to specific source code components such as C++ classes, variables, and functions is

43

44 indicated with monospaced font, e.g. GadgetCloudController .

45

46

47

48

49 Methods

50

51 Architecture and Implementation

52

53

54

⁵⁵ In the following sections, we will first briefly review the Gadgetron architecture and the

56

57

_5g extensions that have been made to this architecture to enable cloud computing. Subsequently,

59

60

fctegneue Resonance in Meamm Magnetic Resonance in Medicine

Apprendix 2 we will describe two specific types of MRI reconstructions (2D time resolved imaging and 3D imaging) that have been deployed on this architecture.

Gadgetron framework

The Gadgetron framework is described in detail in (15). Here we briefly review the dataflow for comparison to the cloud based dataflow introduced below. As shown in Fig. 1, A Gadgetron reconstruction process consists of three components: Readers, Writers and Gadgets. A Reader receives and depenalizes the incoming data sent from the client (e.g. a client can be the MR scanner). A Writer serializes the reconstruction results and sends the data packages to the client. The Gadgets are connected to each other in a streaming framework as processing chains.

The Gadgetron maintains the communication with clients using a TCP/IP socket based connection. The typical communication protocol of Gadgetron process is the following: a) The client issues the connection request to the Gadgetron server at a specific network port.

b) The Gadgetron accepts the connection and establish the TCP I communication. c) The client sends an XML based configuration file to the Gadgetron server. This XML configuration outlines the Reader, Writers, and Gadgets to assemble a Gadget chain. d) The Gadgetron server loads the required Readers, Writers and Gadgets from shared libraries as specified in the configuration file.

e) The client sends an XML based parameter file to the Gadgetron. The Gadgetron can initialize its reconstruction computation based on the parameters (e.g. acquisition

Magm c Res©sar¾ce in Uedlc ^' ni® Magnetic Resonance in Medicine Page 10 of 38

Apprendix 2

1

2

3 matrix size, field-of-view, acceleration factor etc.). For MRI this XML parameter file

4

5

6 is generally the ISMRMRD XML header describing the experiment.

7

8 f) The client sends every readout data to the Gadgetron in the ISMRMRD format. These

9

^ data are de-serialized by the Reader and passed through the Gadget chain.

12

13 g) The reconstruction results are serialized by the Writer and send back to the client via

14

15 the socket connection.

16

17

18 h) When the end of acquisition is reached, the Gadgetron closes down the Gadget chain

19

20 and closes the connection when the last reconstruction result has been passed back to

21

the client.

Gadgetron Plus (GT-Plus): distributed computing extension of Gadgetron

A schematic outline of GT-Plus extension is shown in Fig. 2. A distributed Gadgetron process has at least one Gadgetron running on a Specific purl for each node (multiple Gadgetron processes can run within the same node at different ports). A software module,

GadgetCloudController, manages the communication between nodes. Typically, the gateway node is receiving the readout data from the client and de-serializes them using a

Reader. Depending on the reconstruction workflow, the gateway node may buffer the incoming readouts and perform some processing before sending reconstruction jobs to the connected nodes, or data can be forwarded directly to the client nodes. The

GadgetCloudController maintains multiple TCP/IP socket connections with every connected node via a set of GadgetronCloudConnector objects (one for each connected note). Each GadgetronCloudConnector has a reader thread (CloudReaderTask) and a writer thread (CloudWriterTask) which are responsible for receiving and sending data (to the node) respectively. There is a Gadgetron Gadget chain running on every connected node.

.. 10 . _x , .. ..

fctegneue Resonance in ¾8e<Sieso© Page 11 of 38 Magnetic Resonance in Medicine

Apprendix 2

The gateway node will send the XML configuration to the connected node to assemble the chain. For different cloud nodes, different Gadget chains can be assembled. In fact, the connected nodes can also be gateway nodes, thus creating a multi-tiered cloud. The typical protocol of GT-Plus distributed computing is as follows:

a) The client issues the connection request to the GT-Plus gateway at a specific network port. Once the connection is established, the client will send the XML based

17

18 configuration and parameter files to establish and initialize the gadget chain at the 19

20 gateway node.

21

2g b) The client starts sending readout data to the gateway node.

24

5 c) The Gateway node establishes;: connection to cloud nodes and supplies them with

26

27 XML configuration and parameter files. As mentioned, different connected nodes can

28

29

be configured with different chains if needed.

30

31

32 d) Job packages are sent to connected cloud nodes for processing via the corresponding

³⁴ CloudWriterTask.

35

36

₃₇ e) When all jobs are sent to nodes (the acquisition is done), the gateway Gadgetron either

38

39 waits for reconstruction results or conducts extra computation. The Readerrask

40

41 objects, at the same time, listen for reconstruction results. Whenever the 42

43

44 reconstruction results are sent back from a cloud node, the gateway Gadgetron is

45

46 notified by the ReaderTask object and will take user-defined actions, e.g. passing the

47

48 results downstream or waiting for the completion of all jobs. Finally the gateway node 49

50

51 will proceed to finish other processing steps if any and send the images down the

52

53 remaining part of the gateway Gadget chain and eventually back to the client.

54

55

56

57

58

59

60 Magnetic Resonance in Medicine Page 12 of 38

Apprendix 2 fj If one or more connected nodes fail to complete a job successfully, the gateway node

will be notified by either receiving an invalid result package or detecting a shutdown

message on the socket. The GadgetCloudController on the gateway node will

9

10

11 keep a record of uncompleted jobs and process them locally. In this way, the GT-Plus

12

13 gains robustness against network instability.

14

15

16 The software modules mentioned above are implemented as C++ template classes and are

17

18

19 capable of handling any typo of custom job packages, i.e. the user is able to configure what

20

21 data types are sent and received from the cloud nodes. The only requirement is to implement

22

23 appropriate functions for serialization; and de-serialization of work packages and results. This

24

25

26 design is a straightforward extension of the Readers and Writers in the original Gadgetron.

27

28

²⁹ In the current implementation, every computing node can be given an index indicating its

30

31

32 computational power. This index is used to allow the gateway node to apply a scheduling

33

34 algorithm where the workload is distributed to the cloud nodes in proportion to their

35

36 computational capabilities. This can be important if the cloud consists of a heterogeneous set

37

38

3g of hardware configuration.

40

41

42

43

44 To supply the network connection information of cloud nodes to the Gadgetron, user can

45

46 specify IP addresses or hostnames of nodes in the gateway XML configuration file.

47

⁴⁸ Alternatively, the framework can read in a cloud structure definition file.

49

50

51

52 Gadgetron Plus (GT-Plus) for 2D time resolved (2D+t) Reconstruction Tasks

53

54 in addition to the software modules for cloud communication, the GT-Plus extensions also

55

56

g ₇ include a specific job type to support 2D+t reconstruction tasks, e.g. multi-slice cine imaging.

58

59

60

.. 1 . _x , .. ..

fctegneue Resonance in Meamm Page 13 of 38 Magnetic Resonance in Medicine

Apprendix 2

This workflow is used here to illustrate a cloud setup with a dual layer topology, as illustrated in Fig. 3. The gateway gadget chain will buffer readouts for a specified ISMRMRD dimension (e.g. for the multi-slice cine, it is usually the slice dimension). Once all data for a slice have arrived, the GtPlusRecon2DTGadgetCloud gadget will forward the job package to a first layer node to start the reconstruction.

For a 2D+t dynamic imaging task, one slice will have multiple 2D A:-spaces. For example, for the cardiac cine acquisition, multiple cardiac phases are usually acquired. The first layer node responsible for the reconstruction of a given slice can choose to further distribute the dataset to a set of sub-nodes in the second layer. The first-layer nodes can serve solely to distribute jobs or they can perform computation as well. In principle, a given reconstruction job can utilize an arbitrary number of node layers to : rm a more complicated cloud topology.

Gadgetron Plus (GT-Plus) for 3D Reconstruction Tasks

35

36 For 3D acquisitions, a single layer cloud topology was used in this paper. The gateway node's

37

38 GtPlusRecon3DTGadgGt receives the 3D acquisition and pcrfb-ms the processing such as

39

40

41 coil compression and estimation of k-sp&ce convolution kernel for parallel imaging. It then 42

43 splits the large 3D reconstruction problem by performing a ID inverse FFT transform along

44

45 the readout direction. Thus, the reconstruction is decoupled along the readout direction. Every

47

chunk of data along the readout direction is then sent to a connect node for non-linear

49

50 reconstruction. The gateway node will wait for all jobs to complete and results to return. It

51

52 then reassembles the 3D volume from all chunks and continues to perform other post-

54

55 processing, e.g. fc-space filtering tasks and finally returns images to the client.

56

57

58

59

60

fctegneue Resonance in Meamm Magnetic Resonance in Medicine Page 14 of 38

Apprendix 2

Toolbox Features

The Gadgetron is divided into Gadgets and Toolbox algorithms that can be called from the Gadgets or standalone applications. In addition to the toolbox features listed in (15), the GT- Plus extensions add additional toolboxes. Here is an incomplete list of key algorithm modules:

2D/3D GRAPPA. A GRAPPA implementation for 2D and 3D acquisition is added. It fully

18 supports the I MRYIRD data format and different parallel imaging modes (embedded,

19

2 ₂ 0 ₁ separate or interleaved auto-calibration lines, as defined in (19)). For the 3D GRAPPA case,

22

23 the implementation supports two-dimensional acceleration. To support parallel imaging in

24

interactive or real-time applications, a real-time high-throughput 2D GRAPPA im lementation using GPU is also provided in the Gadgetron. 2 2D/3D Linear SPIRiT. A linear SPIRiT (7) reconstruction is implemented in the Gadgetron

34 ; ,

35 toolbox. Specifically, if the ^-space x consists of filled points a and missing points m, then:

36

37 x = D ^T a + D m (1)

38

39

4 _Q Here x is an vector containing ¾-space points for all phase encoding lines for all channels; a

41

42 stores the acquired points, and m is for missing points. D and D _c are the sampling pattern

43

44 matrixes for acquired and missing points, Linear SPIRiT solves the following equation:

45

47 {G- I)D ⁷ _c m = G- I D ^T a (2)

48

49

50 Here G is the SPIRiT kernel matrix, which is computed from a fully sampled set of auto

51

52

53 calibration data.

54

55

^ 2D/3D Non linear ll-SPIRiT. Equation 2 is extended by the LI term:

57

58

59

60

htegnetsc Resonance in ¾8e<Sieso© Magnetic Resonance in Medicine

Apprendix 2 argmin _m {\\ (G - I) (D ^T a + Dim) \\ ₂ + λ II W C ^F F ^H {D ^T a + D ^T _c m) || (3) Another variation of 11 SPIRiT is to treat the full ft-space as the unknowns:

argmin _x {\\ (G - I)x || ₂ + A \\ W C ^H F ^H x \\ + β \\ Dx - a || ₂ } (4) Here Ψ is the sparse transform and W is an extra weighting matrix applied on the computed sparse coefficients to compensate for non-isotropic resolution or temporal redundancy. F is the Fourier transform matrix. C is the coil sensitivity.

Redundant 2D and 3D wavelet transform. The redundant wavelet transform for 2D and 3D data arrays are implemented i the toolbox. It is used in the LI regularization term. A fast implementation for redundant Harr wavelet is also provided.

Example Applications

Corresponding to the two types of cloud topologies described in the previous sections, two in vivo experiments were performed on healthy volunteers. The local Institutional Review Board approved the study, and all volunteers gave written informed consent.

Real-time multi-slice myocardial cine imaging using ll-SPIRiT

The aim was to make it clinically feasible to assess myocardial function using real-time acquisitions and non-linear reconstructions covering the entire ventricles. With conventional reconstruction hardware, the reconstruction time for such an application would prohibit clinical adoption. The dual layer cloud topology was used here and every slice was sent a separate node (first layer) which further split cardiac phases into multiple chunks. While processing one chunk itself, the first layer node also sent others to its sub-nodes for parallel processing. The algorithm workflow was as follows: a) Undersampled &-space data were

htegnetsc Resonance in ¾8e<Sieso© Magnetic Resonance in Medicine Page 16 of 38

Apprendix 2 acquired with the time-interleaved sampling pattern, b) The gateway node received the

readout data and performed the on-the-fLy noise pre-whitening (5). c) The data from one slice

was sent to one first layer node, d) To reconstruct the underlying real-time cine images, the

auto-calibration signal (ACS) data for a slice were obtained by averaging all undersampled k- space frames at this slice, e) The SPIRiT kernel was estimated on the assembled ACS data, f)

The data for the slice was split into multiple chunks and sent to sub-nodes, together with the

estimated kernel. The size of a data chunk for a node was proportional to its computing power

index, f) Sub-nodes rece:vcd the data package and solved equation 3. The linear SPIRiT

problem (equation 2) was fir^t solved to initialize the non-linear solver, g) Once the process

for a slice was completed, the node sent the reconstructed frames back to gateway, which then

returned them to the scanner. Not lie reconstruction for a given slice started while

acquisition was still ongoing for subsequent slices. Thus the data acquisition and processing

was overlapped in time to minimize the overall waiting time after the data acquisition.

Fig. 4 outlines the 11 SPIRiT reconstruction gadget chain for multi-slice cine imaging using

the dual layer cloud topology. The raw data first went through ihe NoiseAdjustGadget,

which performed noise prewhitening. After removing the oversampling along the readout

direction, the data was buffered in the AccumulatorGadget. Whenever the acquisition for a

slice was complete, the data package was sent to the GtPlusRecon2DTGadgetCloud

gadget, which established the network connection to the first layer nodes and sent the data.

The first layer node ran only the GtPlusReconJob2DTGadget gadget, which then

connected to the second layer nodes. The CloudJobReader and CloudJobWriter were

used to serialize and de-serialize the cloud job package.

.. 16 . _x , .. ..

fctegneue Resonance in ¾8e<Sieso© Page 17 of 38 Magnetic Resonance in Medicine

Apprendix 2

Imaging experiments were performed on a 1.5T clinical M I system (MAGNETOM Area,

Siemens AG Healthcare Sector, Erlangen, Germany) equipped with a 32-channel surface coil.

A healthy volunteer (female, 23.8yrs) was scanned. Acquisition parameters for free-breathing

9

10

cine were as follows: balanced SSFP readout, TR = 2.53/TE = 1.04ms, acquired matrix size 1 1

13 192x100, flip angle 60°, FOV 320x240mm ² , slice thickness 8mm with a gap of 2mm,

14

^ bandwidth 723 Hz/pixel, interleaved acquisition pattern with acceleration factor R=5. The

17

18 whole left ventricular was? covered by 9 slices and acquisition duration for every slice was 19

20 ~1 .5s with one dummy Heartbeat between slices. The scan time (defined as the time to

21

22 perform data acquisition) to complete all 9 slices was 22.

23

24

25

26 High resolution neuro imaging using ll-SPIRiT

27

28

The second example aimed to use GT-Plus for non-linear reconstruction of a high resolution 29

30

31 3D acquisition. The algorithm workflow was as follows: a) Undersampled /c-space data were 32

³³ acquired with the fully sampled center region, b) The gateway node received the readout data

34

35

2Q and performed the on-the-fly noise pre-whitening. c) When all data was received, the SPIRiT

37

38 kernel calibration was performed on the fully sampled ACS region. The kernel was zero-

39

^ padded only along the readout direction and Fourier transformed to the image domain. This

42

43 was done to reduce the maximal memory needed to store the image domain kernel and

44

45 decrease the amount of data transferred over the network to connected nodes. The fc-space

46

47

data was transformed to image domain along the readout as well, d) The gateway node split 48

49

50 the image domain kernel and data into multiple chunks along the readout direction and sent

51

52 them to multiple connected nodes, e) The connected nodes received the packages and zero-

53

54

padded kernels along the remaining two spatial dimensions and transformed into image

56

57 domain. The image domain kernel was applied to the aliased images by pixel-wise

58

59

60

« 17 . .. Magnetic Resonance in Medicine Page 18 of 38

Apprendix 2

1

2

3 multiplication. This linear reconstruction was performed to initialize the non-linear

4

5

g reconstruction, f) After receiving all reconstructed image from connected nodes, the gateway

7

⁸ assembled the 3D volume and performed some post-processing, such as A:-space filtering and

9

^ then sent results back to the scanner.

12

13

14

15 The imaging experiments were performed on a 3.0T clinical MRI system (MAGNETOM

16

17

18 Skyra, Siemens AG Healthcare Sector, Erlangen, Germany) equipped with a 20-channel head

19

20 coil. The acquisition parameters were as follows: GRE readout, TR = 10.0/TE = 3, 1 1ms,

21

22 acquired matrix size 25(" ^: 25<V< l ')2. flip angle 20°, isotropic spatial resolution 1mm ³ ,

24

5 bandwidth 130 Hz/pixel, two dimension acceleration factor R=3 ^X 2. The embedded parallel

26

27 imaging mode was used with the ACS signal acquired as a 32x32 fully sampled region. The

28

29 total acquisition time was 1.6s.

30

31

32

33

34 Deployment and Scanner Integration

35

36 The cloud extension of Gadgetron (GT-Plus), can be deployed on different types of platforms.

37

Three setups have been tested here for in vivo experiments and the GT-Plus software can be

40

1 deployed on those setups without any code changes. The first setup (referred to as the "casual

42

43 cloud") is a heterogeneous collection of computers, e.g. as one would find in many MRI

44

45 research laboratories. These computers have different hardware and operating system

46

47

48 configurations. The second setup is the NIH Biowulf cluster, which is a custom built cluster

49

50 with totally 2300 nodes (>12000 computing cores); it is a shared system and users request a

51

52

₅₃ specific amount of resources for a given computational task. The third setup is the Amazon

54

55 Elastic Compute Cloud (EC2), where an arbitrary amount of computational power can be

56

57

58

59

60

fctegneue Resonance in ¾8e<Sieso© Page 19 of 38 Magnetic Resonance in Medicine

Apprendix 2 rented by the hour providing the flexibility to tailor the cloud configuration to suit a specific application. "Casual" cloud

The first setup is a solution that almost any MRI research laboratory would be able to use by installing the Gadgetron on a set of networked computers and using one of these computers as the gateway node. No specific operating system or special hardware is needed. Here, six personal computers n the Mi l intranet (IGb/s connections) were used as the cloud nodes and

21 two more computers were used as gateway nodes for 2D+t and 3D experiments respectively. 22

23

24 The Gadgetron software was compiled and installed on all computers. The gateway node for 25

26 2D+t test was a desktop computer, running windows 7 Pro 64 bit (four core Intel Xeon E5- 27

2670 2.60GHz processers, 48GB DDR3 RA M). The gateway node for 3D test also ran the same operating-system (two eight-core Intel Xeon _: E5 -2670 2.60GHz processers, 192GB

32

33 DDR3 RAM).

34

35

36

37

38 Among the six cloud nodes, two of them were running windows 7 Pro 64bit (each had four 39

40 cores of Intel Xeon E5-2670 2.60GHz, and 48GB DDR3 RAM) Ubuntu linux 12.04 were

41

42

₄ g running on other four nodes (two nodes each had six cores of Intel Xeon E5645 2.4GHz and

44

45 24GB DDR3 RAM; the other two had four cores of Intel Xeon X5550 2.67GHz and 24GB

46

⁴⁷ DDR3 RAM).

48

49

50

51

52 For the dual layer cloud test, one Windows and two Ubuntu computers served as first layer

53

nodes. The other three nodes were on the second layer. Each of them was connected to one

55

56

57

58

59

60

fctegneue Resonance in ¾8e<Sieso© Magnetic Resonance in Medicine Page 20 of 38

Apprendix 2 first layer node. For the 3D reconstruction test, all six nodes were connected directly to the

gateway.

NIH Biowulf cloud

The second cloud setup tested in this study utilized the NIH Biowulf cluster. NIH Biowulf

system is a GNU/Linux parallel processing system designed and built at the National

Institutes of Health. Biowulf consists of a main login node and a large number of computing

nodes. The computing rirMes within Biowulf are connected by a 1Gb/ network. The MRI

scanner used in this study w as also connected to Biowulf system with a lGb/s network

connection. For the 2D multi-slice ein¾: experiments 37 nodes were requested from the cluster.

The gateway node had 16 cores (two eight-core Intel Xeon E5-2670 2.60GHz processers) and

72GB DDR3 RAM. All other nodes had identical configurations (two six-core Intel Xeon

X5660 2.80GHz, 24GB DDR3 RAM). For the dual layer cloud test, 9 nodes were used on the

first layer to match the number of acquired slices. Bach of them was connected to three sub- nodes on the second layer. For the 3D reconstrudion test, the same gateway node and 23

connected nodes were requested.

The cloud topology was selected to balance the maximal parallelization and programming

complexity. It is convenient to have dual layer structure for the multi-slice cine imaging,

because this structure allows overlap between data acquisition and reconstruction computation.

For the 3D acquisition, the reconstruction only starts when the data acquisition is completed;

therefore, a simple one layer cloud is used to simplify the synchronization. The number of

nodes used in the study was mainly limited by the available resources during the experiments.

Although Biowulf system has a large number of nodes, hundreds of jobs from multiple users fctegneue Resonance in ¾8e<Sieso© Page 21 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3 can run in parallel at any given time. We also intentionally made number of nodes different 4

5

6 between 2D+t and 3D tests, to challenge the scalability of Gadgetron based cloud software.

7

8

^ Amazon EC2 based cloud

11

12 The last setup utilized the Amazon Elastic Compute Cloud (Amazon EC2). Amazon EC2

13

¹ 4 cloud provides resizable computing capacity in the Amazon Web Services (AWS) cloud. User

15

16

₁₇ can configure and launch flexible number of computing nodes. Every node can have different

18

19 hardware configuration arid operating system, Amazon EC2 provides a broad selection of

20

21 Windows and Linux operating systems. More details about Amazon EC2 can be found in (21).

23

24

25

26 For this study, 19 nodes were launched to build up a virtual cloud on Amazon EC2. Every

27

28 node used the cc2.8xlarge instance type (two eight-core Intel Xeon E5-2670 2.60GHz 29

3 _Q

31 processers, 20MB L3 cache per processor, 60GB DDR3 RAM), running Ubuntu Server 13.10.

32

33 All nodes were connected lOGb/s connections. For the dual layer test, one node was used as

34

35 the gateway and 9 nodes were on the first layer. Each first layer nodes were connected to two 36

37

38 other sub-nodes. Thus, data for every slice was processed by three nodes in total. For the 3D

39

40 reconstruction test, all 18 nodes are connected directly to th¾ gateway. At the time of the

41

42

₄ g study, the price for the nodes used in this test was US $2.4 per hour per node or US $45.60

44

45 per hour for the complete cloud setup. In this case, the number of nodes was chosen as a

46

47 reasonable balance between cost and performance.

48

49

50

51

52 The connection speed from the MRI scanner to the Amazon EC2 cloud was measured to be

53

⁵ 49.8MB/s or 0.39Gb/s at the time of the experiments.

55

56

57

58

59

60 Magnetic Resonance in Medicine Page 22 of 38

Apprendix 2

1

2

3

4 Clinical scanner integration

5

6 As previously described, the Gadgetron enables direct integration with the MRI scanner for

7

Q

° online reconstruction (15). The cloud extension of GT-Plus maintains this advantage of online

10

11 processing. Effectively, this setup enables a seamless connection of cloud computing

12

13 resources directly to the scanner. From the end-user perspective, the only noticeable

14

15

difference between reconstruction in the cloud and locally (on the scanner's reconstruction 16

17

18 hardware) is the improvement in reconstruction speed.

19

20

21

22

₂₃ The integration of cloud computing resources with a clinical scanner raises questions about

24

25 patient privacy. In this study multiple steps were taken to mitigate any risks. Firstly, all data

being sent to the Gadgetron from the scanner was anonymized. Absolutely no identifying

3 _Q information that could be used to trace back to the patient were sent over the network. The

31

32 only information sent was the MR raw data and protocol parameters such as field of view,

33

34 matrix size, bandwidth, etc. Secondly, all data transfer between scanner and gateway node

35

36

37 was encrypted via a secure shell (SSH) tunnel (22). All other nodes connected to the gateway

38

39 were behind firewalls. Besides serving as a security measure, the use of SSH tunnels to

40

41

connect to the Gadgetron was a convenient way to test multiple Gacgctron setups on the same 42

43

44 scanner. The reconstruction could simply be directed to a different Gadgetron by changing the

45

46 SSH tunnel pointing to a different Gadgetron instance.

47

48

49

50

51 Tests were performed on two clinical MRI scanners (Siemens Skyra 3.0T and Area 1.5T,

52

^ Siemens Medical Solutions, Erlangen, Germany).

55

56

57

58

59

60 Page 23 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3

⁴ 5 Results

6

7 Multi-slice myocardial cine imaging

9

10 Fig. 5 shows the reconstruction results generated by the GT-Plus cloud, illustrating the

11

12 noticeable improved in image quality using non-linear reconstruction.

13

14

15 Table 1 shows the total imaging time and computing time. The imaging time is the time

16

17

18 period from the start oj data acquisition to the moment when images of all slices were

19

20 returned to scanner. T e: ³ computing time is defined as the time used to perform the

21

22

23 reconstruction computation. On the described Amazon EC2 cloud, the total imaging time was

24

25 52.6s. The computing time was 48.9s because the computation was overlapped with the data

26

27 acquisition. On the NIH's Biowulf cloud, imaging and computing times were 62.1s and 58.2s 28

29

30 respectively. If only a single node was used, the computing time was 427.9s and 558, 1s for 31

32 two cloud setups. Therefore, the multi-slice cine imaging with entire ventricle coverage was 33

^ ⁴ completed within lmin using the non-linear reconstruction on the GT-Plus cloud. The casual

36

37 cloud gave computing time of 255.0s and if only one node was used, the time went up to

38

39 823.7s. This speedup may be helpful to the development and validation of non-linear

40

reconstruction in a MRI research lab.

42

43

44

45 High resolution 3D neuroimaging

46

47

Fig. 6 illustrates the cloud reconstruction results using 11-SPIRiT. Compared to the GRAPPA,

49

50 the non-linear reconstruction shows noticeable SNR improvements. Table 2 gives the total

51

5 ² imaging time and computing time for three cloud setups and the single node processing. The

53

54

55 total imaging time on the Biowulf cloud was 278.5s which includes the computing time of

56

57 186.4s, since not all computational steps ran in parallel for this 3D reconstruction. As every

58

59

60

23

M grseUc Resonance in ¾8e<slelo© Magnetic Resonance in Medicine Page 24 of 38

Apprendix 2 computing node of the Amazon EC2 cloud setup had 16 cores and a newer CPU model, the

total computing time was further reduced to 146.0s. Total imaging time on the Amazon cloud

was 255.0s. Thus the latency following data acquisition was less than 2mins.

Availability and Platform support

The entire package is integrated with the currently available version of the Gadgetron, which

is distributed under a permissive, free software license based on the Berkeley Software

Distribution license. The licensing terms allow users to use, distribute and modify the

software in source or binary form. The source code is supplied with this paper and

documentation can be found at the Sourceforge Open Source distribution website for

Gadgetron (http://gadgetron.sourceforgc.net/). The software has been compiled and tested on

Microsoft Windows 7 64bit, Ubuntu Linux, CentOS 6.4, and Mac OS X (Note this paper does

not present examples running on CentOS and Mac OS X, but the software has been compiled

and tested on those two operating systems).

Discussion

This work extends the previously published Gadgetron framework to support distributed

computing, which makes it possible to distribute a demanding reconstruction task over

multiple computing nodes and significantly reduce the processing time. This paper focused on

reducing the processing time of non-linear reconstruction applications, which have so far been

difficult to deploy clinically. As demonstrated in the examples, the increased computational

power could make the processing time of non-linear reconstruction feasible for the regular

clinical use.

24

M grseUc Resonance in Meaicim Page 25 of 38 Magnetic Resonance in Medicine

Apprendix 2

The Gadgetron Plus (GT-Plus) extension provides a set of utility functions to manage and communicate with multiple nodes. Based on these functions, flexible cloud topologies can be realized as demonstrated in this paper. In the current Gadgetron software release, the discussed dual-layer and single layer cloud implementation are made available to the community. They are designed and implemented for general-purpose use, although the examples given are specific for cine and 3D neuroimaging. Based on the GT-Plus extensions new cloud topologies will be explored in the future and users can design and implement their own cloud structure.

The Gadgetron based cloud computing was deployed on three different types of cloud computing hardware. This demonstralfe t&e flexibility both Gadgetron framework in general and the GT-Plus extensions specifically. The casual cloud setup is the cheapest, most accessible way of using Gadgetron cloud computing. It can be set up quickly and used for

³⁴ algorithm development and validation in a research laboratory. The disadvantage of using

35

36

₃₇ such a setup is the computing nodes may be very heterogeneous and optimal distribution of

38

39 the work among nodes may become non-triviaL If cloud com pul ing is used in a production

40

⁴¹ type setting on a clinical scanner, it is also problematic that a node may be busy with other

43

44 tasks when it is needed by the scanner.

45

46

47

^° The Amazon EC2 system, on the other hand, provides a professional grade cloud setup with

50

51 24/7 availability. The computational resources can be dedicated for a specific project or

52

53 scanner and the nodes can in general have the same configuration, which reduces the

54

55

complexity of scheduling. This setup is also easy to replicate and share among groups and

57

58

59

60

_« 25 . „ ..

fctegneue Resonance in Meamm Magnetic Resonance in Medicine Page 26 of 38

Apprendix 2

1

2

3 scanners. There are other companies that provide the same type of cloud infrastructure service,

4

5

6 e.g. Rackspace (htt ://w w jackspace.com/). With these vendors the users pay by hour to rent

7

⁸ computers. In the setup we describe in this paper, the cost of running the cloud was on the

9

^ order of US $50 per hour. While this cost could be prohibitive if the cloud is enabled 24 hours

12

13 per day, it is a relatively modest cost in comparison to the cost typical patient scans. Moreover,

14

15 this cost is falling rapidly with more service providers entering into the market and more

16

17

18 powerful computing hardware becoming available. The main downside of using a remote

19

20 cloud service such as A-nazon EC2 is the need for a high-speed Internet connection to the

21

2g cloud provider. At large universities lGb/s connections (which are sufficient for the

24

5 applications presented here) are becoming commonplace. However, this may not be the case

26

27 for hospitals in general.

28

29

30

31

32 The third setup, the NIH's Biowulf, is a dedicated, custom built, cluster system. While not

33

34 available to the entire MRI community, it represents a parallel computing platform often

35

36

37 found at research institutions (such as universities) or large corporations. For large

38

39 organizations aiming to provide imaging, reconstruction iitul proi-.-ssing service for a large

40

⁴¹ number of scanners, this setup may be the most cost effective platform. It is, however,

43

44 important to note that purchasing, configuration, deployment, and management of even

45

46 modest cloud computing resources is a non-trivial task that requires significant resources.

47

48

49

50

51 Several software modules implemented in the Gadgetron framework are GPU accelerated,

52

53 although the demonstrated examples in this paper are mainly using CPUs. The GT-Plus

54

55 extension of distributed computing does not conflict with GPU based computational

57

58

59

60

.. 26 . _x , .. ..

fctegneue Resonance in Meamm Page 27 of 38 Magnetic Resonance in Medicine

Apprendix 2 acceleration in any sense. Every computing node can be equipped with different hardware; for those with good GPU processers or CPU coprocessors (e.g. Intel Xeon Phi coprocessors http;;y ww ntel.c^ the local

9

10

11 processing can utilize those resources, since the Gadgetron framework allows every node to

12

13 be configured with different Gadget chains. Heterogeneous computational hardware across

14

15 nodes can be challenging for optimal load balancing. In the current framework, the user can

16

17

18 supply computing power indexes for nodes to indicate its strength. But the framework does

19

20 not presently im iemen: more complicated approaches to help determine the computing

21

22 ability of every node.

23

24

25

26

27 The Gadgetron framework and its toolboxes are mainly programmed using C++ and fully

28

29 utilize generic template programming. While efforts are made to provide documentation and

30

31

32 examples in the Gadgetron distribution, developers who want to extend the toolboxes still

33

³⁴ need some proficiency with object oriented programming and C++ in particular.

35

36

37

38

39 Conclusion

40

⁴¹ We have developed the Gadgetron Plus extension for the Gadgetron framework to support

43

44 distributed computing using various cloud setups. We have deployed the Gadgetron based

45

46 distributed computing over three very different cloud environments. We have shown the

47

48 increased computational power in the cloud significantly speeds up 11 -SPIRIT reconstruction 49

50

51 for 2D+t dynamic imaging and high-resolution 3D acquisitions.

52

53

54

55

56

57

58

59

60

27

M grseUc Resonance in Meaicim Magnetic Resonance in Medicine Page 28 of 38

Apprendix 2

ACKNOWLEDGEMENTS

This research was supported fully by the Intramural Research Program of the NIH, National

Heart, Lung and Blood Institute. This study utilized the high-performance computational

capabilities of the Biowulf Linux cluster at the National Institutes of Health, Bethesda,

Maryland, USA. (http://biowulf.nih.gov).

28

MagrseUc Resonance in Page 29 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3

4

5 References

6

7 1. Blaimer M, Breuer F, Mueller M, Heidemann RM, Griswold MA, Jakob PM. SMASH,

8 SENSE, PILS, GRAPPA: How to Choose the Optimal Method. Topics in Magnetic

9 Resonance Imaging 2004; 15(4):223-236.

^ 2 2. Breuer FA, Kellman P, Griswold MA, Jakob PM. Dynamic autocalibrated parallel

13 imaging using temporal GRAPPA (TGRAPPA). Magnetic Resonance in Medicine

14 2005,53(4) 981-985.

15

16 3. Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang J, Kiefer B, Haase

¹⁷ A. Generalized autoealibrating partially parallel acquisitions (GRAPPA). Magnetic

^ Resonance in Mediciiu 2002;47(6): 1202- 1210.

1 y

20

21 4. Kellman P, Epstein FH,:: McVeigh ER, Adaptive sensitivity encoding incorporating

22 temporal filtering (TSEN SE). Magnetic Resonance in Medicine 2001 ;45(5):846-852. 23

24 5. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: Sensitivity encoding

25 for fast MRI. Magnetic Resonance in Medicine 1999;42(5):952-962.

26

27

_2g 6. Lustig M, Donoho D, Pauly JM. Sparse MRI: The application of compressed sensing for

₂₉ rapid MR imaging. Magnetic Resonance in Medicine 2007;58(6):1182-1195.

30

31 7. Lustig M, Pauly JM. SPIRiT: Iterative self-consistent parallel imaging reconstruction

32 from arbitrary k-space. Magnetic Resonance in Medicine 2010;64(2):457-471.

33

³⁴ 8. Hansen MS, Atkinson D, Sorensen TS. Cartesian SENSE and k-t SENSE reconstruction

^ using commodity graphics hardware. Magnetic Resonance in Medicine 2008;59(3):463-

37 468.

38

3g 9. Sorensen TS, Schaeffter T, Noe KO, Hansen MS. Accelerating the Nonequispaced Fast

40 Fourier Transform on Commodity Graphics Hardw are. Medical Imaging, IEEE

41 Transactions on 2008;27(4):538-547.

42

^ 10. Jung H, Sung K, Nayak KS, Kim EY, Ye JC. k-t FOCUSS: A general compressed sensing

₄ g framework for high resolution dynamic MRI. Magnetic Resonance in Medicine

₄₆ 2009;61(1): 103-1 16.

47

48 1 1. Liang D, DiBella EVR, Chen R-R, Ying L. k-t ISD: Dynamic cardiac MR imaging using

43 compressed sensing with iterative support detection. Magnetic Resonance in Medicine

∞ 2011 ;68(l):41-53.

12. Lustig M, Santos JM, Donoho DL, Pauly JM. k-t SPARSE: High Frame Rate Dynamic

5 MRI Exploiting Spatio-Temporal Sparsity. 2006 6- 12 May; Seattle, Washington, USA. p

55 2420.

56

57

58

59

60 Magnetic Resonance in Medicine Page 30 of 38

Apprendix 2

1

2

^ 13. Vasanawala SS, Alley MT, Hargreaves BA, Barth RA, Pauly JM, Lustig M. Improved

5 Pediatric MR Imaging with Compressed Sensing. Radiology 2010;256(2):607-616.

6

7 14. Vasanawala SS, Murphy MJ, Alley MT, Lai P, Keutzer K, Pauly JM, Lustig M. Practical

8 parallel imaging compressed sensing MRI: Summary of two years of experience in

⁹ accelerating body MRI of pediatric patients. 2011 March 30 - April 2; Chicago, IL l 039- 1043.

12

13 15. Hansen MS, Sorensen TS. Gadgetron: An open source framework for medical image

14 reconstruction. Magnetic Resonance in Medicine 2012;69(6): 1768-1776.

15

16 16. Feng Y, Song Y, Wang C, Xin X, Feng Q, Chen W. Fast direct fourier reconstruction of

radial and PROPELLER MRI data using the chirp transform algorithm on graphics

1 ? hardware. Magnetic Resonance in Medicine 2013;70(4): 1087-1094.

19

20

₂₁ 17. Simpson R, Keegan J, Gatehouse P, Hansen M, Firmin D. Spiral tissue phase velocity

22 mapping in a breath-hold w ith non-cartesian SENSE. Magnetic Resonance in Medicine

23 2013 :doi: 10.1002 im ii: 24 1 .

24

25 18. Xue H, Kellman P, LaRocca G, Aral A, Hansen M. High spatial and temporal resolution

² ^ retrospective cine cardiovascular magnetic resonance from shortened free breathing real-

_2g time acquisitions. Journal of Cardiovascular Magnetic Resonance 2013; 15: 102.

29 . . .....

30 19. Hansen MS. Sharing Reconstruction Software - Raw Data Format and Gadgetron. 2013;

31 Sedona, AZ, USA.

32

33 20. NIH Biowulf cluster system. http://biowulf.nih.go\,'. Bethesda, Maryland, USA. July 15,

³⁴ 2013.

35

36

21. Amazon Elastic Compute Cloud (Amazon EC2). http://aws.amazon.com/ec2/. July 12,

37

38 2013.

39

40 22. Network Working Group of the IETF. The Secure Shell ( SSI I ) Authentication Protocol.

41 2006.

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

_« 30 . „ ..

fctegneue Resonance in Meamm Page 31 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

4 List of Captions

5

6 FIG 1. A schematic presentation of Gadgetron architecture. The client application communicates with f Gadgetron process via the TCP/IP connection. The Reader de-serializes the incoming data and

8 passes them through the Gadget chain. The results are serialized by the Writer and sent back to client

9 application. The Gadgetron process resides on one computer, the Gadgetron server. The client can be

10 a MRI scanner or other customer processes. Note the algorithm modules implemented in the

11 Gadgetron toolboxes are independent from the streaming framework; therefore, these functionalities

12 can be called by standalone applications which are not using the streaming framework.

13

14 FIG 2. Example presentation of GT-Plus distributed computing architecture for Gadgetron In this

15 example, at least one Gadgetron process is running on a node (Multiple Gadgetron processes can run

16 in one node on different ports). The gateway node communicates with the client application (e.g. MRI

17 scanner). It manages the connections to each of the computing nodes via the software module

18 GadgetCloudContro e¾i; and GadgetronCloudConnector. Whenever sufficient data is

19 buffered on the gateway node, the package can be sent to the computing node for processing.

20 Different computing nodes can run completely different reconstruction chain.

21

22 FIG 3. The dual layer topobqy of cloud computing for 2D+t applications. Every connected node

23 contains its own GadgetCloUdContrcpller arid can split a reconstruction task for a slice to its

24 sub-nodes. Compared to the basic cloud topology shown in Fig. 2, this dual layer example adds extra

25 complexity. This example demonstrates ^he flexibility of GT-Plus architecture for composing different

26 computing cloud topologies to fit different reconstruction tasks.

27

28 FIG 4. The gadget chain used in the multi-slice myocardial cine imaging. Here the

29 GtPlusRecon2DTGadgetCloud gadget coitrcls the connection to the first layer node where the

30 data from a slice was reconstructed. The second layer sub-nodes were further utilized to speed up the

31 reconstruction. The TCP/IP socket connections were established and managed by the cloud software

32 module implemented in the GT-Plus.

33

34

35 FIG 5. Reconstruction results of multi-slice myocardial cine imaging on the GT-Plus cloud. Compared

36 to the linear reconstruction (a), non-linear reconstruction (b) improves the image quality. With the

37 computational power of Gadgetron based cloud, the entire imaging including data acquisition and non- 38 linear reconstruction can be completed in 1 min to achieve the whole heart coverage.

39

40 FIG 6. Reconstruction results of 1 mm ³ brain acquisition on the GT- ⁼ lus cloud. The basic cloud

41 topology shown in Fig. 2 was used here. Both GRAPPA linear reconstruction (a) and (the M SPIRiT

42 results (b) are shown here The single node processing time is over 20mins, which prohibits the

43 clinical usage of non-linear reconstruction. With the Gadgetron based cloud, a computing time of

44 <2.5mins can be achieved for this high resolution acquisition.

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60 Magnetic Resonance in Medicine Page 32 of 38

Apprendix 2

Table 1

Total imaging time (from the start of data acquisition to the moment when all images are returned to the scanner) and computing time in seconds for the multi-slice myocardial cine imaging. The in vivo test was only performed with cloud setups. The single node computing time was recorded with the retrospective reconstruction on the gate-way node. 0 Casual Biowulf Amazon EC2

1

2

3 Single Cloud Single Cloud Single Cloud

4

5

6 Imaging time (s) 259.2 - 62.1 - 52.6

7

8

9

0 Computing time (s) 823;. . 255.0 558.1 58.2 427.9 48.9

1

2

3

4

5

g Tablo 2

7 Total imaging and computing time in seconds fc:r the 3D neuro acquisition . The single node used for

Q comparison is the gateway node for every cloud setup.

9

0 Casual Biowulf Amazon EC2

1

2

3 Single Cloud Single Cloud Single Cloud

4

5

6 Imaging time (s) 541.9 - > "278.5 - 255.0

7

8

9 Computing time (s) 1054.3 449.1 1265.1 186.4 983.5 146.0

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

32 Page 33 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3

4

5

27

28 1008x606mm (120 x 120 DPI)

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

Magnetic Resonance in edicine Magnetic Resonance in Medicine Page 34 of 38

Apprendix 2

FiG 2. Example presentation of GT-Plus distributed computing architecture for Gadgetron. in this example, at least

one Gadgetron process is running on a node (Multiple Gadgetron processes can run in one node on different ports).

The gateway node communicates with the client application (e.g. MRI scanner), it manages the connections to each

of the computing nodes is the software module and

Whenever sufficient data is buffered on the gateway node, the package can be sent to the computing node for

processing. Different computing nodes can run completely different reconstruction chain.

999x643mm ( 120 x 120 DPI)

Magnetic Resossarsce ftiedicine Page 35 of 38 Magnetic Resonance in Medicine

Apprendix 2

1

2

3

4

5

24 FIG 3 The dual layer topology of cloud computing for 2DM applications Every connected node contains its own

25 and can split a reconstruction task for a slice to its sub-nodes. Compared to the basic

26 cloud topology shown Fig. 2, this dual layer example adds extra complexity. This example demonstrates the flexibility of GT-Plus architecture for composing different computing cloud topologies to fit different reconstruction

27 tasks.

28

29

30

31 661x458mm [ 120 X 120 DPI)

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

MagrseUc Resonance in !edSelo© Magnetic Resonance in Medicine Page 36 of 38

Apprendix 2

FIG 4. The gadget chain used in the multi-slice myocardial cine imaging. Here the

gadget controls the connection to the first layer node where the data from a slice was reconstructed. The second layer sub-nodes were further utilized to speed up the

reconstruction. The TCP/IP socket connections were established and managed by the cloud software

module implemented in the GT-Plus.

931x648mm ( 120 x 120 DPI)

MagrseUc Resonance in !edSelo©

91 Magnetic Resonance in Medicine Page 38 of 38

Apprendix 2

(a) GRAPPA (b) H-SPIRiT

F!G 6. Reconstruction results of 1 rnm3 brain acquisition on the GT-Pius c!oud. The basic cbud topology shown in Fig. 2

was used here. Both GRAPPA linear reconstruction (a) and {the h-SPIRiT results (b) are shown here. The stngie node

processing time is over 20mtns, v/hich prohibits the clincal usage of non-iinear reconstruction. With the Gadgetron

based cloud, a computing time of <2.5mins can be achieved for this high resolution acquisition.

388x248mm ( 120 x 120 DPI)

MagrseUc Resonance in

Em loyee Discovery ar* rtveniiort Report (EIR)

An EIR should be completed for each discovery or inverrtion that is a) a novel innovation, b) a new or improved method or process; ©r c) believed to have potential commercial value (e.g. a new reagent, unique antibody, vaccine, medica! device, or therapeutic compound); or d) a request from a commercial organization for use or resale of the new discovery or innovation.

if you are employed by PHS it is presumed that the Invention was made as part your official duties as a Government employee. If this is not the case, you should stili complete the EIR, but you must contact your Technology Development Coordinator (TPC) and provide the details pertaining to this particular discovery or invention so that a determination of rights can be made.

COMPLETION OF THE EiR

1.

2. The TDC will then forward the completed electronic report and the printed, signed EIR to the Office of Technology Transfer (OTP. If your IC in conjunction with the OTT decides not to file a patent application on your invention you may contact your Te ^npjogy, Development Coord in ator (TDC) to request a waiver and, if granted, have an opportunity to obtain the rights to the invention by filing the patent application at your own expense.

Frequently Asked Questions: http://ottintranei.od.nih.gQv/EIR EiR FAQS 2011Q915.htm

General questions regarding the form may be directed to the NiH Office of Technology Transfer (OTT). If is suggested, particu!arfy if you leave government service and are receiving royalties, that you keep the ^ce £in¾D¾ .Ma[liggrn§Dt apprised of changes in your official address.

Thank you for your cooperation

Privaey Aet Notice: The PHS is collecting this information under authority of 45 CFR Part 7 "Employee lD¥.fOfens.'.'. The information will be maintained as a part of the System o^

Patent and Licensing Documents." Provision of this information is mandatory and will be used as the initial step toward obtaining patent protection of inventions submitted by PHS employees, granting licenses to PHS inventions/administering and providing royalty payments to PHS inventors, and the intended "routine uses" of the information. Failure to provide complete information may adversely affect t e Government's rights to future patent applications and licensing agreements.

{Sard 2007) SIR Ver. 20110915,2 Coa«.i®is.ial Page 1 or 14 Appendix 3

Employe® Disco @Ky and Issv®n ion Re or (EI¾)

IC Ref: Date Received OTT Reference # Date Bed 1/28/2014 via email

E - 074 - 2 014 / 0

First Name Hui Last Name Email hui.xue@fiih.gov

Tits® of Discovery: (In 200 characters or less provide a title that is sufficiently descriptive to

identify the discovery.)

' keai-Time Processing of Medicai imaging Raw Data using Ctoud Computing J

2. Please provide a dear and eojieie® description of your discovery. Describe in terms that an individual in a. Provide background information related to tha discovery.

TS s computing prccass (image formation from raw data) irs medical imaging has bsco-me

increasingly eompiex with high computational demands. The computational equipment (irs general a single computer) deployed with clinical imaging systems is often in dequate for achieving dinica!ly practical image reconstruction times. The hardware has to be specified and tested years before the imaging devices are deployed and the hardware is consequently obsolete before it is deployed This invention proposes to eliminate the need to deploy date processing hardware !ocal y by deploying the image reconstruction hardware and software in a (remote) cloud computing system. The

powerful computational resources avaiiabfe in commercial cloud systems cars be leveraged dyedSy

In modem medical imaging devices and by doing so, the image processing hardware is replaced by a flexible software component which can bm seated on-the-fty to mateh modern algorithms and whteh can be serviced and deployed remotely. Forth® end-user of imaging device, e.g. the

physician or technician operating the scanner, they will feel no differences than rf the computation was performed locally, ¥¾fe have demonstrated that this is possible with dirtica! MRI systems and usir¾g t is a a igm, imajj e reconstruction can s ed isp by art order of magn t de for some

Describe how the discovery wilt be used and what produci(s) couid be made.

The proposed inverttiort opens a new paradigm for imaging raw data processing. There are seve al potential uses.

c. Describe the unique features and advantages of this discovery that distinguishes it over the

problems or limitations associated with the current ; science or exisjng pfoduete.

Current imaging devices, e.g. MR\ scanners, perform the imaging computation locally on a computer corrected to the scanner body, This computer has limited hardware resource and cannot deliver images faster enough for clinical usage for advanced imaging stgerttrtms. Therefore, the current sca n r vendors only provide relstsve v simple algo ithms on their c fstputers. The proposed

invention will completely remove this limitation by leveraging a potentially unlimited computing resource in the ctoud.

It is ©QSiry to deploy, service, and upgrade computing hardware in the field. With this invention the vendors can eliminate this part of the imaging system and service S remotely.

SStord 20075 EIR ver . 20110915. 2 Confiden ial Page 2 of 14 Empl yee Discovery ^• »r*nt on Re ort. (EIR)

ar are* ¾n so are ee e on ¾ scars y seasi ass.

not «a y mprove t e magng ¾ cterscy an mage qus or t ose

fi applicable) tf the dssmvsry is or relates to a composition, ctescrfte the cofrrposittars by providing sequence structures of amino acicfe or riucieic acids, provide chemical formulas forchernicaJ compoynds; if the discovery is a met od, outline th^ particular steps of f > method _. . _.

i ... ^~~~ I

Piease list She most peffineni articles, presentations, other public disclosures, relevant patents or paterrt applications made fey others thai are rented te your discovery?

Attach ptm of†Jw uMieations cKsd; copies *f U.S. Patents or pulsl fs&ti jja¾«l apjsiesiiorss %m not

Current St»g$ of ShsvstepsTOst {Pkia insert "X" for s» f?e following sMes

a. Doss this disclosure irsc!uife research msteriajis) ¾iat e&M ® mad® svaitefcle for iissjistng? if Yes, com iete Atiachmerst 1.

b, is this EIR beir¾ sybm.tted based on an outside party s request fw iicesjsing msterial?,

if Yes provide the name of the compsny's contact person and telephone num e or email

c identify am compgriyfs) w resn-profit organjzaSons . that ar¾ conducting similar research.

]

Hs you beers santaeted or ae you in discuss rts with anyone interested in soi!afaorating iE you to tewslop this disoowry?

tHVSA 2087) SIR W. iOUSSIS.S Con dential fogs 3 o£ loyee Disco ery^ Bficf¾n m»ti.on Report (BIB)

NO

Thinking creatively, answer llse folkwirsg;

Describe how the discovery or materia! m¾y enhance the pubiic heaitft. i«, therapeutic, diagnostic, screening, ^as¾¾ y. ^search ^to ¾ ^^ . j*^3fg.>. g^ ^r ..f^ft ^¾ ^ g.-.

a) The dirsiea] applicability of advanced imaging eomp½irii ablfty brought in by tnis

invention wiif S8§rs¾cantiy speed «p ft® Irrssgirsg efffctersey and improve Image qua! . It means tfse redaction of imaging costs, e.g. or> MR! clinical scans for poblic, ecause the imaging time sie«¾ed for esch subject is «¾dye#d.

b) Novel imaging application that currently cannot e use^ srs (Dsttents can fee

deployed with i&e proposed methts siegy, wbidi wl enable testing o »* techniques in patients and provide sfate-of-tfce-®rt technology to patients. c) Tne- ihffoS-pafly entities cars le erage jh¾ l&cti soSogy to compete for freely in i e d) The a«sr®tl cost of an ¾IRS system r»¾y r«<lw««d. Effse we^ trst vsncfcr oars

provide the m ® ( udimentary) >magss fsswistswAM equipmen to ail cus omers an users that don't run wry stdyssced imaging studies uld simply pay for wbat they get and mer¾ advanced amis wouid still be able to ««e ifte Safest ublic Disckmyr»s/Pr®is«isti0i¾s (Past and f uiasmi.

Has srs abstract or poster beers prepared or is now in preparation ! presentstion or publication? Yes sttaefc § copy, {mrty i For sae

Page 4 <:i i · Appendix 3

Em loye© Discovery a!od Invention Report (EIR)

ESMRMB 2014 ogram . htm

Joint Annual 10-18 MAY NOV 20. 2013 May 10, 2014 http : / /www . ism

2. Reeling iS¾«RM- 2014 rm. org/1 /1 pr ESMRSUB 2014

ogram . htm

d. please

Collaborations, Material Transfers, Confidentia! Dssdosure Agreements

The fol owing questions are designed to answer whether any part of the research respiting in this discovery involved collaborations outside of your laboratory, used materials/softwarefequipment obtained from an another party, and/or included outside funding. (Consult with your laboratory chief and/or your TDC if there is a question.)

a. CRADA: Is the subject matter of your disclosure re!ated to a PHS Cooperative Research and Development Agreement (CRADA or M-CRADA) involving you or your laboratory or IC?,

if Yes, pleas® identify the col!aborator, the NIH principle investigator, C AOA title and the institute's CRADA reference number:

YES, we haw access to programming (throssgh the IDEA programming environment) the

image reco structioii system through the Semene MRS CRADA {Andrew Arai is the PI). This invention is, hmmm, not related to a specific CRADA project and is NOT don® in

collaboration with Siemens.

Collaborations: Is the subject matter based on research collaborations other than a CRADA? If Yes, please identify your collaborator and the name of their organization.

[ NO :.: i

Outside ateteriais/lnfarrraafom Transfer: Is there subject matter in this discovery based on proprietary materials or confidential information obtained from an outside party or organization?,

If Yss, please identify Jhe third party contributor their email or telephone number, ;^ name.

Please provide a eopy of any document that reeonfts this trartsfsr. This may include an small, Material Transfer Agreement, Confidential Disclosure Agreement, Clinical Trial Agreement, or Research Collaboration Agreement.

d, e.

Identification ©f Contributors

Hui X e, Peter Kellman, Michael Hansen, Thomas S,

Soerensen, Souheil Inati

List below the names and organizations of aii psopl® who participated in conceiving or continued development of the discovery/invention Examples include those who made intellectual, theoretical, or innovative contribution to the discovery, in the case of software, those individuals who were involved in creating program code, manuals, flowcharts or any related items.

NOTICE: This ma b® fe er indi iduate thm rsamed in 7.a. above. Please b@ aware that inventorship is strictly defined in patent law. Accordingly, contributors you list in this section will be named on patent apptbations resulting from this EIR only if their contributions mee this legal standard. A co-author may or may not qualify based on the particular facts; if you have any questions, contact your TDC.

(Word EIR Ver. Confi ential of 14 Employee Discovery and invention Report (KIR)

The following acknowledgement pertains to Government employees and thosa treated as employees. Under 5 CFR Part 7 "Employee Inventions", al! ©mpbyees of th© Public Health Service have an obligation to report and assign inventions to the Government of the United States, as represented by the Department of Health and Human Services. Specifically, the Government shaSI obtain the entire right, title, and interest in inventions: (i) made during working hours; or (ii) made with a contribution by the Government of facilities, eqiiipment, materials, funds or information or of time or services of ejther Government employees on official duty: or (iii) which bear a direct relationship or are made in consequence of the official duties of th® inventor.

!f you are employed by PHS to conduct or perform research, it is presumed that the invention was made under these circumstances, if this is not the case, you should sti!l complete the EIR, but you must contact your Teehnotoejv

D wlojgmejrt Cop^^ and provide the details pertaining to this particular discovery or invention so that a determination of rights can be made.

1. Submitting H i Kue NHLB!. NIH

Org

Contributor :

2. Co-Caniributor swicfBsei ¾. na is i Org

3. Co-Contributor Peter feiSmsrt Org NHLB, NIH

4. Co-Contributor Org

5 . Enter additional Co-Contributor's names and organizations as necessary.

j ~ !

An Additions! Contributor information document is to be completed for each additional contributor listed in 7b.5.

The form may be downloaded at: http://www.ott. h.aov/decs/EIR~Additiona Contributar.docx

Contributor Information begins on next page

iKord EIR Ver. 20110915.2 Cos¾f icten i&l Page 6 of 14

* indicates a required field

* First

Degree

* Describe this indivisiuai's contribution to the discovery.

Propose the system design, conduct the software development and perform the experiments

Gsirre it Organisation infermatksn:

♦Qrgarsiz atiors Name agrsttic S¾ ssenanc© Teehnalo gy Prog am, National Heart, L ufsg and Blood institute, National institutes of Health, , MO, USA

Division/Branch/Laboratory

*Titie

* Office Address Building 10, BIO

oCity Bet esda * State aryland *Ztp 20814 ♦ Country

code

* E-mail m!cfiasS. ar!sen@rtii,go¥ Telephone +1-301-496- FAX +1 (3Glj 496- Other contact

1457 2389 number, (optional)

* Has your organizatiaial affi Station ch anged during the development of this discovery? Yes/Mo, if ye s, expiain

H&tm informatiere

*City Silver Spring ♦ State *2ipc©de ♦ Country USA Phone 301-412-9671 Email

Please identify with a "X" if this individual talis under one or more of the following training or feibws ip gppointmerits or institutional

Contributor: I have read and understand trie information submitted in the EI .

Signature

<Word 2007> EIR Ver. 20110915,2 3¾ d» «L ¾¾§S3¾ fe 3» S _* -L fef^^ S^ Report <EIS)

* indicates a respired field

* Describe $fis» iswSiwitf ai's te«iri¾w¾&fs la the discovery.

Howe ittfowjtSoff

*Stai8 S NewJereey 07631 USA

Prions 41 S89712

Rease identify with s "X" if this individual fails under one or more of the foitowsrsg ]¾

©n I have read and understand the infomatissi submitted m the EIFt.

f lu I X^- Employs® Discover g«dJayj¾riifers R&pvri {EiR)

CoBtnb momtmSa Sheet

* Indicates a required field

Contributor 3

Narns

* First Peter « Lasi Suffix

Degree PhD « Citizenship USA

* Employ ee identification Ho. « Associated prosed MSH 01

Project Nymberta)

* Des ribe title arscSMdus s contribution to the discove y.

Propose the system design, conduct trie soSsware development and perform trie experiments

Current Orgarsizstten IrefomsatSars:

* Organization Name Wte icai SignaS and Image Pr ocessing Program, Natj Ofiai Heart, Lung and B!ood !r stitute.

National institutes of Health, MO. USA

Division/Branch/Laboratory

» Title

* Office Address

*City Betftssda * State Maryland «2ip 20814 [ * Country

code

* E-mail kelirranp®nhW.rnh.go¥ Telephone +1-301-496- FAX +1 1301) 49 6- Other contact

2513 2389 number, (optional)

♦ Has your organisational affiliation changed during the development of this discovery? Yes/ ^' ίίο, if yes, explain

Home !! reformation

♦ City

Phone I ♦State

Email

Pisase identify wetH a "X ^* if this individual or in iijut ai

Coreiributer: I haye read and understand the information submitted in the E!R.

Date: 2013,12.04 11:23:49 -05W

{Word 2007) EIR Ver. 2C110915.2 Con idential Page 9 of 14

* Indicates a required fieSd

Contributor 4

First «La¾¾ Suffix egree * Citizenship j

EiYi Sovee tetentificatiert No.: * Assoc sated project Ni!H 201

Protect f*iiimber(s)

* Describe Ms SraeSivWual's ctamrSfcution to t ie discovery.

Current Orgarsteatien Information

# Organization Name

DivisiorVBrancrs Laborator

* TilSe

♦ Office Address

♦ City ♦ State *Z3p * Country

code

» E-rnaii FAX Other contact

number, (optional)

♦ Has your organizational affiliation changed during the development of this discovery? Yes fto, if yes, explain

Home infofrostisn

♦ Country Email

Please identify with β "X" if this individual falis under one or more of the following training or fellowship eppointments or institutional

Contrib isr: I ha/e read and understand the information submitted in the E!R.

5 EIR Ver. 20110915.2 Page 10 of 14 Em loyee Dise< Report (EIR)

ecoanmen e propagaton me um an grow c aracers cs: .e. expresson

J

R®«o?nm®rid®d freeze medium

Sterility: (i.e. negative for bacteria, fungi & mycoplasma

Morphology: (i.e. epithelial-like, iymphobtast-iike, fibroblast like)

¾somm¾id«j Storage: (i.e. liquid nitrogen)

(So cS 2007) EIR Ver. 20110915.2 Con£ idte&fcisl Page 11 of 14 Emplo ee Report (ESS)

Attachment 2

Supplemental Information Sheet for Software

The purpose ©f this attachment is to provide !nlortsiati©!. ragarding evaluation of software for potential licensing. The following questions should be answered as completely as possible

Does this software contain cods obtained from a third-party or covered by any Open Source Licens© (e. .,

( ord 200?) SIR Ver. 20110915,2 C©nfi«tell iml Page 12 of 14 Attachment 2

Supplemental Information Sheet, for Software

1 AUTHORS BE UA8tI GfS ANY CLAIM , DAMAGES OR OTHER UABIL!TY, WHETHER ^ m AH ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION ¥¥ITH TH£ SOFTWsRE OR THE USS OR OTHER DEALINGS IN THE SOFT¥¾¾S¾E.

Was a government contractor involved in the writing or e elopment of arty of the code? if yes, identif the

Otoxd 200"?) El f V«r. 2011091.5.2 13 of It Employee DiscQV¾ ®p5dhx«3tton Report (EiR)

w om a i patent recommendations and patent correspondence are to be directe .

The TDC, IC delegate, or Service Center Representative Authorized IC official for expenditure of IC funds for confirms receiving the EIR and acknowledges a

Name

Name

Title

Title

Signature.

Signature.

A complete EIR packet wili be forwarded to the Director, DTDT, which comprises a single, non-stapled EIR packet containing the signed EiR coniaining all documents such as manuscripts, presentation, articles and citations referred to in the EIR, as well as any related IC reviews and authorization documents. The electronic version containing ail the above information shouid be forwarded to OTT at ottflleroom@maii.nlh.HQv

(Word 3007) EIR Vei . 20110915.2 Page 1 Appendix 4

7403

3D High Resolution 11-SPIRiT Reconstruction on Gadgetron based Cloud

Hui Xiie ¹ , Peter Kellman ¹ , Souheil Inati ² , Thomas Sorensen ³ , and Michael Schacht Hansen ¹

!Magnetic Resonance Technology Program, National Heart, Lung and Blood Institute, National Institutes of Health, Bethesda, Maryland, United States, ² National Mental Health Institute, National Institutes of Health, Bethesda, Maryland, United States, ³ Department of Computer Science, Department of Clinical Medicine, Aarhus

University, Aarhus, Denmark

Introduction Non-linear and iterative reconstruction algorithms have been the subject of intense study in the MR imaging community. Very promising algorithms have been published in the literature, including image domain compressive sensing [1| and ^-space SPIRiT and its regularized versions [2] . Given the lengthy acquisition time of high resolution 3D MRI imaging, it is of great interest to apply non-linear reconstruction to shorten the imaging. The robustness against subject movement or uncooperative pediatric patients can be improved. However, the clinical usage of these techniques is often prohibited by the high demand for computational power and very long reconstruction time. To achieve practical reconstruction times for clinical usage of non-linear reconstruction, we have extended the previously published Gadgetron framework [3] to support the distributed computing across multiple computers. This extension is named "Gadgetron Plus" or "GT-Plus". A cloud version of 3D 11-SPIRiT reconstruction was implemented on the GT-Plus cloud framework and applied to high-resolution 3D neuroimaging. With the use of the computational power of the cloud, we demonstrate that a 3mins reconstruction can be achieved for 1mm ³ isotropic neuro scans using 11-SPIRiT. Compared to the linear reconstruction, the image quality was significantly improved.

Architecture and Implementation At least one Gadgetron process was started at every node. The inter-node communication was managed by a software module using TCP/IP sockets. A gateway node was connected to the scanner and received readout data. It then sent buffered data package to computing nodes for processing. Every computing node was responsible for processing the received job via its processing chain and forwarded results back to the gateway. With all expected results received on the gateway, images were sent back to the scanner,

Cloud version of 3D LISPIRiT The gateway node was configured to receive the readouts for a 3D acquisition and perform &-space convolution kernel estimation. It then split the large 3D reconstruction problem by performing the ID inverse FFT transform along the readout direction. Thus, the reconstruction was decoupled along the readout direction. Every chunk of data along the readout direction was sent to one computing node. The LISPIRiT algorithm was running on every node. The redundant Harr wavelet transform was used in the regularization term.

Deployment on cloud systems Two cloud setups were tested. Amazon EC2 based cloud 19 nodes were launched on the Amazon Elastic Compute Cloud (Amazon EC2). All nodes had two eight-core Intel Xeon E5-2670 2.60GHz processers, running Ubuniu Server 13.10. The gateway node had 240GB RAM and others had 60GB. NIH Biowulf cluster 23 nodes were requested from the Biowulf cloud (http://biowulf.nih.gov) which is a linus based parallel computing system (Red Hat Server 5.10) and built at National Institutes of Health, USA. The gateway node had 16 CPU cores (two eight-core Intel Xeon E5-2670 2.60GHz processers) and 256GB RAM. All other nodes had two sis-core Intel Xeon X5660 2.80GHz processers and 24GB RAM. For both cloud setups, "online" reconstruction was achieved, That is, the acquired readout data was sent to cloud while the scan was proceeding. The reconstructed images were directly sent back to the scanner and stored in the clinical PACS.

In-vivo test Cloud based reconstruction was performed for high resolution neuro acquisition. A healthy volunteer was scanned on a 3.0T clinical MRI system (MAGNETOM Skyra, Siemens, 20-channel head coil). Acquisition parameters were: GRE readout, TR = 7.0/TE = 3.07ms, acquired matrix size 256x256x192, flip angle 20°, 1 mm ³ isotropic spatial resolution, bandwidth 120 Hz/pixel, two dimension acceleration R=3x2 and 3 x3 with a 32x32 fully sampled k-space center. The total acquisition time was 65s and 46s for two acceleration levels.

Results Figure 1 shows the reconstruction results generated on the GT-Plus based cloud for R=3x2. Figure 2 is for R=3x3. Both cases indicate non-linear reconstruction noticeably improved image quality, compared to the linear GRAPPA reconstruction. The reconstruction time (defined as the computing time to perform the reconstruction algorithms) was 171s for R=3x2 and 170s for R=3x3 on the described Amazon EC2 cloud. On the Biowulf system, the reconstruction time was 239s and 242s. If only a single node was used, much longer reconstruction time was experienced: for R=3x2 and x3, trie Amazon EC2 cloud recorded 1002s and 1022s and forthe Biowulf system, computation took 1279s and 1338s respectively.

Conclusions The GT-Plus extension for Gadgetron framework was developed to support distributed computing across multiple nodes. The 3D 11-SPIRiT algorithm was implemented on the Gadgetron based cloud, giving significantly reduced reconstruction time for high resolution neuro scans. This speedup can enable the clinical usage of advanced non-linear reconstruction algorithms on 3D MR applications.

References [1] Lustig M, et al., MRM 58:1182-1195 [2] Lustig M, et al., MRM 64:457-471 [3] Hansen MS, et aL, MRM 69:1768-1776 (2013)

Figure

Appendix 5

7284

Distributed Computing on Gadgetron: A new paradigm for MRI reconstruction

Hui Xue ¹ , Peter Kellman ¹ , Souheil Inati ² , Thomas Sorensen ³ , and Michael Schacht Hansen ¹

Magnetic Resonance Technology Program, National Heart, Lung and Blood Institute, National Institutes of Health, Bethesda, Maryland, United States, ² National Mental Health Institute, National Institutes of Health, Bethesda, Maryland, United States, ³ Department of Computer Science, Department of Clinical Medicine, Aarhus

University, Aarhus, Denmark

Introduction MRI reconstruction has moved beyond the

simple Fast Fourier Transform, towards more complicated

parallel imaging and non linear reconstruction algorithms.

Often, developers who wish to implement and deploy their

advanced algorithms on clinical scanners, find the vendor

supported reconstruction hardware is inadequate for

advanced algorithms or the provided programming

environment is not suitable for clinical integration. The

open-source framework, the Gadgetron, which was

published recently, aims to address some of these concerns

[1], With this framework, algorithms can be implemented on

suitable hardware selected by the user and subsequently

connected to the scanner. While this is a valuable step- forward, the original Gadgetron was designed to run the

reconstruction task within one process residing on a single

computer and does not provide explicit support for cloud

computing across multiple computational nodes. Although

multiple CPU or GPU cores on one computer can contribute

Figure 1.

to the computation, single computer may not carry sufficient

computing power to achieve clinical usage of non-linear

reconstruction. To remove this limitation, we have extended GadgetCloudCoiitroLLer GaclgetronCloudCorineccor.

the Gadgetron framework to support cloud computing. With

this extension (named "Gadgetron Plus or GT-Plus"), any Reader/Writer

number of Gadgetron processes can run across multiple

computers (thereafter referred as 'nodes') and a dedicated inter-process controlling scheme is implemented to multiple nodes. We demonstrate that with the GT-Plus cloud non-linear reconstruction of real-time cardiac cine imaging can be deployed in a clinical setting with acceptable reconstruction latency. Specifically, a multi slice real-time cine acquisition covering cardiac ventricles and non-linear reconstruction can be completed within lmin,

Architecture and Implementation Schematic outline of GT-Plus is shown in Fig. 1 , representing a typical setup of Gadgetron based cloud where at least one Gadgetron running on a specific port at each node. A software module Gadge CloudControl ler is implemented to manage the communication between nodes via TCP/ΊΡ sockets. Typically, a gateway node receives readout data from the scanner and distributes them through the cloud. For every connected node, a reader thread (CloudRead.erTo.sl·:) and a writer thread (CloudWriterTask) is spawned as active objects. There is a Gadgetron processing chain running on every connect node and for different cloud nodes, different processing chains can be performed. Whenever the reconstruction results are sent back from a cloud node, the gateway is notified and will take actions defined by the user, e.g. forwarding images back to the scanner or waiting for other jobs to complete.

Deployment The GT-Plus package can be deployed on various platforms. Two cloud setups were tested here. Amazon EC2 based cloud 19 nodes were launched to build up a virtual cloud on the Amazon Elastic Compute Cloud (Amazon EC2). Every node had two eight-core Intel Xeon E5-2670 2.60GHz processers and 60GB RAM, running Ubuntu Server 13.10. NIH Biowulf cluster The Biowulf system (httpY/biowulf.nih.gov) is a GNU Linux parallel processing system built at the National Institutes of Health, USA. A total of 38 nodes were requested from the Biowulf, The gateway node had two eight-core Intel Xeon E5-2670 2.60GHz processers and 72GB RAM. All other nodes have two six-core Intel Xeon X5660 2.80GHz processers and 24GB RAM. For both setups, the "online" reconstruction was achieved.

In-vivo test Cloud based reconstruction was performed for multi-slice free-breathing myocardial cine imaging with non-linear 11-SPIRiT reconstruction [3]. A healthy volunteer (female, 23.8yrs) was scanned on a 1.5T clinical MRI system (MAGNETOM Area, Siemens, 32-channel surface coil). Acquisition parameters were: balanced SSFP readout, TR = 2.53/ΓΕ = 1.04ms, acquired matrix 192x100, flip angle 60°, FOV 320x240mm ² , slice thickness 8mm with a gap of 2mm, bandwidth 723 Hz/pixel, interleaved acquisition pattern with acceleration factor R=5. The ventricles of the heart were covered by 9 slices and for every slice the acquisition lasted 1.5s with one dummy heartbeat between slices.

Results Figure 2 shows the reconstruction results generated on the GT-Plus based cloud, illustrating noticeable improvement in image quality using non-linear reconstruction. The scan time (defined as the time to perform data acquisition) for this test was 22.6s. On the described Amazon EC2 cloud, the total imaging time (defined as the time from the start of data acquisition to the moment when images of all slices were sent back to scanner) was 52,6s. The computing time (defined as the time used to perform the reconstruction computation) was 48.9s. Note the reconstruction started once the first shce was acquired, rather than waiting for the completion of all 9 slices. On the NIH's Biowulf cloud, imaging and computing times were 62.1s and 58.2s. If only the single gateway node was used, the computing time was 427.9s and 558.1 s for two cloud setups respectively.

ro8