Automatic landmark placement

FIELD OF THE INVENTION

The invention relates to the field of image registration and more specifically to landmark-based image registration.

BACKGROUND OF THE INVENTION

Image registration is the key technology for many clinical applications, for example, when changes in a patient's anatomy need to be monitored over time or when comparison of a patient's anatomy with an anatomical atlas is required. All currently known registration methods have certain limitations. An overview of image registration methods is given in the article by J. B. Antoine Metz and Max A. Viergever entitled "A survey of medical image registration", Medical Image Analysis, Vol. 2 (1998) pages 1-37. In particular, standard landmark-based registration techniques require explicit one-to-one landmark correspondences in the registered images. Manual setting of landmarks is a difficult and tedious process, especially in three-dimensional (3-D) data. On the other hand, automated landmark setting methods are mostly application-specific and their design usually requires a considerable effort.

SUMMARY OF THE INVENTION

It would be advantageous to have a system that improves the task of setting the landmarks in the first and second image data.

To better address this issue, in an aspect of the invention, a system for registering first image data and second image data, based on a first set of landmarks comprised in the first image data, is provided, the system comprising: a second landmark unit for mapping each landmark from the first set of landmarks into the second image data space, based on local registration of a first data subset of the first image data, comprising the landmark, and the second image data, thereby creating a second set of landmarks comprised in the second image data; and a registration unit for registering the first and second image data, based on the first and second set of landmarks.

A skilled person will understand that references to a landmark, landmarks and a set of landmarks typically refer to landmark locations because a landmark in image data is defined by the landmark location in the image data space, i.e., the area, volume or time-space portion corresponding to the image data. The landmarks in the first image data may be defined by a user, for example. The second landmark unit is arranged to locally register the first data subset with the second image data. During local registration, the second landmark unit is arranged to search for a local registration map for mapping the first data subset into the second image data to find a second data subset satisfactorily similar to the first data subset. When landmarks of the first set of landmarks are mapped into the second image, the registration unit is arranged to perform landmark-based registration of the first and second image data.

The mapping of landmarks from the first set of landmarks into the second image data space by the system of the invention improves setting the landmarks in the second image data by automating the process of selecting landmarks in the second image data. Thus, the landmark setting process is accelerated. This allows the system users to save time used for registering the first and second image data.

In an embodiment of the system, the second landmark unit comprises: a selection unit for selecting the first data subset of the first image data comprising the landmark from the first set of landmarks; - a local registration unit for registering the first data subset with the second image data, thereby determining a local registration transformation for mapping the first data subset onto a second data subset of the second image data; and a mapping unit for mapping the landmark into the second data subset, using the local registration transformation. The selection unit may be used for finding an optimal first data subset of the first image data.

In an embodiment of the system, the second landmark unit further comprises an evaluation unit for evaluating the results of the local registration and for retaining the landmark in or removing the landmark from the first set of landmarks, based on this evaluation. It may happen that for a certain landmark in the first image data no good, corresponding landmark is found in the second image data. Thus, the evaluation unit is arranged to evaluate the result of the local registration of the first data subset with the second image data. If the result of the local registration is poor, i.e., if the second data subset is not similar enough to the first data subset, the evaluation unit is arranged to remove the landmark

from the first set of landmarks. Otherwise, the evaluation unit accepts the result of the local registration.

In an embodiment of the system, the local registration transformation is a rigid transformation. Rigid transformations are easy to implement and fast. In many cases, rigid transformations are also sufficient to obtain a satisfactory local registration.

In an embodiment of the system, the local registration transformation is an affine transformation. Affine transformations offer more degrees of freedom than rigid transformations and allow for deforming the first data subset. If the result of local registration using rigid transformations is not satisfactory, the local registration unit may be arranged to employ local registration using affine transformations.

In an embodiment, the system further comprises a first landmark unit for selecting landmarks in the first image data, to be included in the first set of landmarks. The first landmark unit may be arranged to employ image segmentation or feature detection to automatically identify landmarks in the first image data, e.g., blood vessel bifurcation points and bone structures. Optionally, the first landmark unit may be arranged to identify candidate landmarks for accepting or rejecting by the user. Thus, the first landmark unit further improves setting landmarks in the first and second image data.

In a further aspect of the invention, a method of registering first image data and second image data, based on a first set of landmarks comprised in the first image data, is provided, the method comprising: a second landmark step for mapping each landmark from the first set of landmarks into the second image data space, based on local registration of a first data subset of the first image data, comprising the landmark, and the second image data, thereby creating a second set of landmarks comprised in the second image data; and - a registration step for registering the first and second image data, based on the first and second set of landmarks.

In a further aspect of the invention, a computer program product to be loaded by a computer arrangement is provided, the computer program product comprising instructions for registering first image data and second image data, based on a first set of landmarks comprised in the first image data, the computer arrangement comprising a processing unit and a memory, the computer program product, after being loaded, providing said processing unit with the capability to carry out the tasks of: mapping each landmark from the first set of landmarks into the second image data space, based on local registration of a first data subset of the first image data, comprising

the landmark, and the second image data, thereby creating a second set of landmarks comprised in the second image data; and registering the first and second image data, based on the first and second set of landmarks. In a further aspect of the invention, the system according to the invention is comprised in an image acquisition apparatus.

In a further aspect of the invention, the system according to the invention is comprised in a workstation.

It will be appreciated by those skilled in the art that two or more of the above- mentioned embodiments, implementations, and/or aspects of the invention may be combined in any way deemed useful.

Modifications and variations of the image acquisition apparatus, of the workstation, of the method, and/or of the computer program product, which correspond to the described modifications and variations of the system, can be carried out by a person skilled in the art on the basis of the present description.

A person skilled in the art will appreciate that the method may be applied to multidimensional image data, e.g., to 2-dimensional (2-D), 3-dimensional (3-D), or 4- dimensional (4-D) images, acquired by various acquisition modalities such as, but not limited to, standard X-ray Imaging, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound (US), Positron Emission Tomography (PET), Single Photon Emission Computed Tomography (SPECT), and Nuclear Medicine (NM).

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will become apparent from and will be elucidated with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein:

Fig. 1 schematically shows a block diagram of an exemplary embodiment of the system;

Fig. 2 shows a flowchart of an exemplary implementation of the method; Fig. 3 schematically shows an exemplary embodiment of the image acquisition apparatus; and

Fig. 4 schematically shows an exemplary embodiment of the workstation.

Identical reference numerals are used to denote similar parts throughout the Figures.

DETAILED DESCRIPTION OF EMBODIMENTS

Fig. 1 schematically shows a block diagram of an exemplary embodiment of the system 100 for registering first image data and second image data, based on a first set of landmarks comprised in the first image data, the system comprising: a second landmark unit 110 for mapping each landmark from the first set of landmarks into the second image data space, based on local registration of a first data subset of the first image data, comprising the landmark, and the second image data, thereby creating a second set of landmarks comprised in the second image data; and - a registration unit 120 for registering the first and second image data, based on the first and second set of landmarks.

The second landmark unit 110 of the exemplary embodiment of the system 100 further comprises: a selection unit 111 for selecting the first data subset of the first image data comprising the landmark from the first set of landmarks; a local registration unit 112 for registering the first data subset with the second image data, thereby determining a local registration transformation for mapping the first data subset onto a second data subset of the second image data; a mapping unit 113 for mapping the landmark into the second data subset, using the local registration transformation; and an evaluation unit 114 for evaluating the results of the local registration and for retaining the landmark in or removing the landmark from the first set of landmarks, based on this evaluation.

The exemplary embodiment of the system 100 further comprises the following units: a first landmark unit 105 for selecting landmarks in the first image data, to be included in the first set of landmarks; a control unit 160 for controlling the workflow in the system 100; a user interface 165 for communicating with a user of the system 100; and - a memory unit 170 for storing data.

In an embodiment of the system 100, there are three input connectors 181, 182 and 183 for the incoming data. The first input connector 181 is arranged to receive data coming in from a data storage means such as, but not limited to, a hard disk, a magnetic tape, a flash memory, or an optical disk. The second input connector 182 is arranged to receive

data coming in from a user input device such as, but not limited to, a mouse or a touch screen. The third input connector 183 is arranged to receive data coming in from a user input device such as a keyboard. The input connectors 181, 182 and 183 are connected to an input control unit 180. In an embodiment of the system 100, there are two output connectors 191 and

192 for the outgoing data. The first output connector 191 is arranged to output the data to a data storage means such as a hard disk, a magnetic tape, a flash memory, or an optical disk. The second output connector 192 is arranged to output the data to a display device. The output connectors 191 and 192 receive the respective data via an output control unit 190. A person skilled in the art will understand that there are many ways to connect input devices to the input connectors 181, 182 and 183 and the output devices to the output connectors 191 and 192 of the system 100. These ways comprise, but are not limited to, a wired and a wireless connection, a digital network such as, but not limited to, a Local Area Network (LAN) and a Wide Area Network (WAN), the Internet, a digital telephone network, and an analog telephone network.

In an embodiment, the system 100 comprises a memory unit 170. The system 100 is arranged to receive input data from external devices via any of the input connectors 181, 182, and 183 and to store the received input data in the memory unit 170. Loading the input data into the memory unit 170 allows quick access to relevant data portions by the units of the system 100. The input data may comprise, for example, the first and second image data. Optionally, the input data may comprise locations of landmarks from the first set of landmarks. The memory unit 170 may be implemented by devices such as, but not limited to, a Random Access Memory (RAM) chip, a Read Only Memory (ROM) chip, and/or a hard disk drive and a hard disk. The memory unit 170 may be further arranged to store the output data. The output data may comprise, for example, the first image data registered with the second image data or the second image data registered with the first image data. The memory unit 170 may be also arranged to receive data from and/or deliver data to the units of the system 100 comprising the first landmark unit 105, the second landmark unit 110 comprising the selection unit 111, the local registration unit 112, the mapping unit 113, and the evaluation unit 114, the system 100 further comprising the registration unit 120, the control unit 160, and the user interface 165, via a memory bus 175. The memory unit 170 is further arranged to make the output data available to external devices via any of the output connectors 191 and 192. Storing data from the units of the system 100 in the memory unit

170 may advantageously improve performance of the units of the system 100 as well as the rate of transfer of the output data from the units of the system 100 to external devices.

Alternatively, the system 100 may comprise no memory unit 170 and no memory bus 175. The input data used by the system 100 may be supplied by at least one external device, such as an external memory or a processor, connected to the units of the system 100. Similarly, the output data produced by the system 100 may be supplied to at least one external device, such as an external memory or a processor, connected to the units of the system 100. The units of the system 100 may be arranged to receive the data from each other via internal connections or via a data bus. In an embodiment, the system 100 comprises a control unit 160 for controlling the workflow in the system 100. The control unit may be arranged to receive control data from and provide control data to the units of the system 100. For example, after mapping the last landmark from the first set of landmarks into the second image data space, the second landmark unit 110 may be arranged to provide control data "the last landmark is mapped into the second image data space" to the control unit 160 and the control unit 160 may be arranged to provide control data "register the first and second image data" to the registration unit 120, thereby requesting the registration unit 120 to register the first and second image data. Alternatively, a control function may be implemented in another unit of the system 100. In an embodiment, the system 100 comprises a user interface 165 for communicating with the user of the system 100. The user interface 165 may be arranged to provide data for displaying views computed from the first and/or second image data. Optionally, the user interface 160 may be arranged to receive user inputs for selecting landmarks in the first image data. Optionally, the user interface may receive a user input for limiting the local registration transformations to rigid transformations, or to any other class of transformations, e.g., similarity transformations or affϊne transformations. A person skilled in the art will understand that more functions may be advantageously implemented in the user interface 165 of the system 100.

In an embodiment of the system 100, the user creates a set of landmarks by setting each landmark in the first image data, using the user interface 165. Alternatively, locations of landmarks from the first set of landmarks may be read in from an external file or identified by the first landmark unit 105. An exemplary method of automatically identifying landmarks in image data is described in "Automatic landmark detection for cervical image registration validation", Juan D. Garcia Arteaga and Jan Kybic, Proceedings of SPIE — Volume 6514, Medical Imaging 2007: Computer-Aided Diagnosis, Maryellen L. Giger, Nico

Karssemeijer, Editors. Alternatively or additionally, landmarks may be selected by a user of the system. Advantageously, enabling the user to select landmarks provides the system 100 with a useful interactive image registration function.

The second landmark unit 110 is arranged for mapping each landmark λ from the first set of landmarks into the second image data space, thereby creating a second set of landmarks comprised in the second image data. In an embodiment, a first data subset Bχ of the first image data, comprising the landmark λ, is chosen by the selection unit 111. Further, the first data subset Bχ is registered with the second image data by the local registration unit 112, i.e., the local registration unit 112 is arranged for finding a local registration transformation Tχ for transforming each location x e Bχ of the first data subset into a corresponding location Tχ(x) in the second image data space. This local registration transformation Tχ is then applied to the landmark λ comprised in the first data subset by the mapping unit 113, thereby obtaining the transformed landmark Tχ( λ) corresponding to the landmark λ. In an embodiment, the first data subset is a hypercube (for 4-D first and second image data), a cube (for 3-D first and second image data) or a square (for 2-D first and second image data). Hereinafter, the drawings and description refer to a square. A person skilled in the art may easily generalize the description to 3-D and 4-D image data. The landmark λ is at the center of the square. The candidate registration transformations are translations. The square is mapped by a candidate local registration transformation T into the second image data space. For example, since the second image data is typically similar to the first image data, the initial candidate local registration transformation T may be an identity and hence, local coordinates of the square vertices in the first and second image data space may be initially identical. The square size must not be too small in order to allow for a meaningful comparison of intensities corresponding to locations of the first and second data subset. On the other hand, the square size must not be too large, so that translations can adequately describe most, and ideally all, possible local differences between intensities of the first and second image data. For example, the side of the square may be 10 pixels long. In an embodiment, the similarity measure for optimizing the registration transformation is based on the sum of square differences of respective intensities in the first and second data subset,

D(t) = ∑(I ₂ (x) -I _λ (x + t)f , xeBi

where I _\ {x + i) denotes the intensity of the first image data at a location T(x) = x + t in the first image data space and h{x) denotes the intensity of the second image data at a location x in the second image data space, t is a translation vector corresponding to the candidate local registration transformation T, Bχ is a square comprising the landmark λ, and the sum runs over all locations x of the square Bχ. Advantageously, the similarity measure D(t) is a differentiable function of the translation vector t. Thus, the search for an optimal candidate local registration transformation T, i.e., for a translation vector t for which the similarity measure D(t) attains a minimum, may employ the steepest descent algorithm, or any other gradient-based minimization algorithm. The optimal candidate local registration transformation denoted Tχ is referred to as the local registration transformation.

Those skilled in the art will understand that other subsets of data sets, e.g., rectangles or ellipses, may be also used. It is also possible, in some embodiments, to optimize a first subset of the first image data, e.g., the size of a square, the aspect ratio of a rectangle, or the shape and orientation of an ellipse. Further, other candidate local registration transformations, e.g., rigid, similarity, or affine transformations, may be employed by the second landmark unit 110.

The registration unit 120 is arranged for registering the first and second image based on the first and second set of landmarks. Landmark-based image registration methods are known to those skilled in the art. For example, a landmark-based image registration method is described in "Spline-Based Elastic Image Registration", Karl Rohr, PAMM Proc. Appl. Math. Mech. 3, 36-39 (2003). Those skilled in the art will further understand that registering the first and second image data may involve either transforming the first image data into the second image data or vice versa.

Those skilled in the art will further understand that other embodiments of the system 100 are also possible. It is possible, among other things, to redefine the units of the system and to redistribute their functions. Although the described embodiments apply to medical images, other applications of the system, not related to medical applications, are also possible.

The units of the system 100 may be implemented using a processor. Normally, their functions are performed under the control of a software program product. During execution, the software program product is normally loaded into a memory, like a RAM, and executed from there. The program may be loaded from a background memory, such as a ROM, hard disk, or magnetic and/or optical storage, or may be loaded via a network like the

Internet. Optionally, an application-specific integrated circuit may provide the described functionality.

Fig. 2 shows a flowchart of registering first image data and second image data, based on a first set of landmarks comprised in the first image data. The method begins with a first landmark step 205 for selecting landmarks in the first image data, to be included in the first set of landmarks. After the first landmark step 205, the method 200 continues to a second landmark step 210 for mapping each landmark from the first set of landmarks into the second image data space, based on local registration of a first data subset of the first image data, comprising the landmark, and the second image data, thereby creating a second set of landmarks comprised in the second image data. In an embodiment of the method 200, the second landmark step 210 comprises a selection step 211 for selecting the first data subset of the first image data comprising the landmark from the first set of landmarks, a local registration step 212 for registering the first data subset with the second image data, thereby determining a local registration transformation for mapping the first data subset onto a second data subset of the second image data, a mapping step 213 for mapping the landmark into the second data subset using the local registration transformation, and an evaluation step 214 for evaluating the results of the local registration and for retaining the landmark in or removing the landmark from the first set of landmarks, based on this evaluation. After the evaluation of each local registration result, the method continues to the continuation step 215 for deciding whether to continue mapping more landmarks into the second image data space. If more pairs of corresponding landmarks are needed for landmark-based image registration, the method continues to the selection step 211 for selecting the first image data subset of the first image data for the next landmark. Otherwise, the second landmark step 210 is terminated. After the second landmark step 210, the method 200 continues to a registration step 220 for registering the first and second image data, based on the first and second set of landmarks. After the registration step 220, the method 200 ends.

A person skilled in the art may change the order of some steps or perform some steps concurrently using threading models, multi-processor systems or multiple processes without departing from the concept as intended by the present invention. Optionally, two or more steps of the method of the current invention may be combined into one step. Optionally, a step of the method of the current invention may be split into a plurality of steps.

Fig. 3 schematically shows an exemplary embodiment of the image acquisition apparatus 300 employing the system 100, said image acquisition apparatus 300 comprising a

CT image acquisition unit 310 connected via an internal connection with the system 100, an input connector 301, and an output connector 302. This arrangement advantageously increases the capabilities of the image acquisition apparatus 300, providing said image acquisition apparatus 300 with advantageous capabilities of the system 100. Fig. 4 schematically shows an exemplary embodiment of the workstation 400.

The workstation comprises a system bus 401. A processor 410, a memory 420, a disk input/output (I/O) adapter 430, and a user interface (UI) 440 are operatively connected to the system bus 401. A disk storage device 431 is operatively coupled to the disk I/O adapter 430. A keyboard 441, a mouse 442, and a display 443 are operatively coupled to the UI 440. The system 100 of the invention, implemented as a computer program, is stored in the disk storage device 431. The workstation 400 is arranged to load the program and input data into memory 420 and execute the program on the processor 410. The user can input information to the workstation 400, using the keyboard 441 and/or the mouse 442. The workstation is arranged to output information to the display device 443 and/or to the disk 431. A person skilled in the art will understand that there are numerous other embodiments of the workstation 400 known in the art and that the present embodiment serves the purpose of illustrating the invention and must not be interpreted as limiting the invention to this particular embodiment.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim or in the description. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a programmed computer. In the system claims enumerating several units, several of these units can be embodied by one and the same item of hardware or software. The usage of the words first, second, third, etc., does not indicate any ordering. These words are to be interpreted as names.