Field of the invention

The invention relates to a method for registration of transmission radiography diagnostic images taken at different time moments for the same patient.

The invention further relates to a system for registration of transmission radiography diagnostic images taken at different time moments for the same patient.

The invention still further relates to a computer program product for registration of transmission radiography diagnostic images taken for different time moments for the same patient.

Background of the invention

In medical practice quite often a patient is being examined for purposes of follow-up examination at different times using substantially the same field of view. For example, for thorax examinations, transmission X-ray images, also referred to as transmission radiography images, of lungs and surrounding tissue, such as ribs, may be taken.

It is a common practice for the radiologists to review the suitable images taken at different moments. These images will be further on referred to as the earlier diagnostic image(s) and the later diagnostic image(s). The procedure of said comparison is generally carried out using light tables or using computer displays, the latter for digitized images.

However, it is found to be quite challenging to carry out accurate assessment of the possible differences between the earlier diagnostic image(s) and the later diagnostic image(s) for a number of reasons. First, the field of view taken for these images may be different. Secondly, an image may be

(slightly) distorted due to organ or patient displacement. Thirdly, a plurality of different diagnostic apparatus having different operational settings may be used for acquiring the earlier and the later diagnostic image(s) under consideration. Fourth, the patient may have changed dimensions, because of age, operation or due to a weight change. And finally, human errors may occur during visual inspection of the images.

Summary of the invention

It is an object of the invention to provide a method for registration of diagnostic transmission radiography images taken for different time moments for the same patient, which is accurate and which allows for quantification of the differences between the earlier and the later diagnostic image.

To this end the method according to the invention comprises the steps of:

- retrieving an earlier diagnostic transmission radiography image; - retrieving a later diagnostic transmission radiography image;

- identifying a set of reference marks in the earlier diagnostic transmission radiography image and the later diagnostic transmission radiography image;

- superpositioning the said earlier image and the said later image by using a set of reference marks;

- identification of a region of interest in the earlier image and in the later image;

- elimination of structures from the earlier image and the later image not corresponding to the said of marks and the region of interest;

- automatically registering the earlier image and the later image;

- automatically identifying differences between the earlier image and the later image;

automatically marking an area corresponding to a difference between the earlier image and the later image on the later image. It is found that with a suitable image processing, drawbacks of the prior art may be mitigated to a substantial extent. For example, when the step of image registration is carried out, misalignments and distortion artefacts can be eliminated. It will be appreciated that any suitable image registration method may be used for practicing the invention. For example, in a region of interest one or more image segments may be identified. This identification may be carried out automatically or manually. Then, for example, linear image warping may be used for the identified segments making use of suitable patterns. Next, a homography matrix composed of rotation, shift and scale may be generated for the individual parts of the identified patterns. It will be appreciated that a great plurality (up to 100) segments may be identified within a region of interest. These segments may be individually warped and corrected for rotation, lateral difference in position and anterior-posterior difference in position.

It will be appreciated that although the invention may be carried out using two-dimensional X-ray transmission images, it is applicable to other imaging modalities and more dimensional images, such as 3D, 4D or 5D images. Those skilled in the art will readily appreciate which three-dimensional image registration algorithms may be applied for the 3D images.

Preferably, the step of superpositioning is carried out using identified locations of the static reference points in the image. For example, specific structures or regions, such as lung top, lung border, bony structures, such as ribs may be used for that purpose. Those skilled in the art will readily appreciate which per se known algorithms may be used for enabling automatic delineation of the bony structures, such as ribs, in the diagnostic images under consideration.

When the earlier image and the later diagnostic image are registered, image intensities may be subtracted yielding the corresponding sought difference. This difference may be used as an aid for making a diagnosis. It is found that in order to distinguish relevant medical differences from signals caused by noise and/or misplacements in the superpositioning process a correlation algorithm may be advantageously applied on the found differences after the steps of matching and subtraction carried out for the earlier diagnostic image and the later diagnostic image.

Preferably, for the earlier diagnostic image and the later diagnostic image comprise respective thorax images are selected. For the thorax images it is found advantageous to use a substantially non-variable reference mark, such as a portion of a bone.

The method according to the invention comprises the steps of identification of a region of interest in the earlier image and in the later image; elimination of structures from the earlier image and the later image not corresponding to the selected set of marks and the region of interest. It is found that by eliminating the structures not part of the image of interest data processing may be further improved and accelerated. More details on this particular embodiment will be given with reference to Figure 3.

An embodiment of a system arranged for detecting changes in a radiograph is known from WO 99/05641. In is a disadvantage of the known system that a normalization algorithm is applied on the full image without suitably adjusting the region of interest. It will be appreciated that, in case the images are acquired using different equipment (different vendors, or the same vendor, but having different settings), different patient positioning, post processing or changes of patient's shape, there will be a substantial variation in lighting and/or contrast of the images. Accordingly, by applying the normalization step for the whole image without correcting the region of interest, a substantial distortion of data corresponding to the region of interest may occur. In particular, undesirable smoothing of the intensity data corresponding to lungs may be expected. Accordingly, very small

abnormalities, such as tumors, may be missed due to a substantial reduction of their detectability due to smoothing. Another disadvantage of the known system is that it is adapted for correcting for lateral inclinations only. However, it will be appreciated that the patient images may be quite different, especially when they are taken with a considerable time span in between. In particular, it is found that the earlier and the later images usually have anterior-posterior differences in position as well as a rotational displacement. It will be appreciated that both these displacement need to be corrected for prior to matching the earlier and the later images.

The known system has a still further disadvantage in that only a top portion of the lung is taken into account when carrying out segmentation of the image data. However, such approach leads to substantial errors due to a difference in movement if the diaphragm and the lung. Similarly,

segmentation of the left lung may be inaccurate in the known system because of the beating heart, which forms part of the image. Accordingly, with these limitations of the known system it appears that the known system uses only about 40% of the image data for carrying out segmentation. Such approach leads to substantial problems when matching the remaining parts of the image.

Further, the known system is arranged to provide an image which results in a subtraction of the earlier and the later image. However, keeping in mind that the known system uses only about 40% of the available data, this subtraction may be highly inaccurate. In particular, some existing patterns within the region of interest, such as diaphragm and surroundings, may be poorly matched. However, it will be appreciated that many clinical problems are occurring near the abdominal diaphragm. Accordingly, accurate

visualization and data analysis of this particular region is highly needed.

In an embodiment of the method according to the invention the step of automatically identifying differences between the earlier diagnostic image and the later diagnostic image comprises the step of:

- transformation of the earlier diagnostic image; - subtraction of intensity signals between the registered earlier diagnostic image and the later diagnostic image.

It is found to be particular advantageous to allow the earlier diagnostic image to be suitably transformed, as presenting data on basis of the true later image may have a higher clinical relevance.

In a further embodiment of the method according to the invention, the method further comprises a step of applying a filter to the earlier diagnostic image and/or the later diagnostic image for further post-processing of the images. This embodiment is explained in further detail with reference to Figure 1. Preferably, the filter comprises a low pass smoothing filter based on data convolution. Those skilled in the art shall readily appreciate how such low pass filter may be defined.

In a still further embodiment of the method according to the invention the method further comprises the step of adjusting the image contrast prior to image registration.

The contrast may be adjusted locally or dynamically. It is found to be particularly advantageous to carry out contrast adjustment of the diagnostic images under consideration, for example to eliminate influence of accidental bright areas surrounding the actual image. This step is found advantageous because contrast adjustments may affect the identification of very small/low intensity differences between the images. This embodiment will be explained in more detail with reference to Figure 2.

In a still further embodiment of the method according to the invention, the method further comprises the step of visualization of the area corresponding to a difference between the earlier and the later diagnostic image.

Preferably, the detected differences are quantified and color coded for indicating the suspected medical relevance. More preferably, an image representing the color coded differences is superimposed on the later diagnostic image. The computer program product for registration of diagnostic images according to the invention comprises instructions for causing a processor to carry out the steps of the method as is discussed in the foregoing.

The system for registration of diagnostic images taken for different time moments comprises:

- a processor adapted to :

i. retrieving an earlier diagnostic transmission radiograph image;

ii. retrieving a later diagnostic transmission radiograph image iii. identifying a set of reference marks in the earlier diagnostic transmission radiograph image and the later diagnostic transmission radiograph image;

iv. superpositioning the said earlier image and the said later image by using a set of reference marks;

v. identifying of a region of interest in the earlier image and in the later image;

vi. eliminating of structures from the earlier image and the later image not corresponding to the said of marks and the region of interest;

vii. automatically registering the earlier image and the later image;

viii. automatically identifying differences between the earlier image and the later image;

ix. automatically marking an area corresponding to a difference between the earlier image and the later image on the later image.

In an embodiment of the system according to the invention it further comprises a display arranged to display the earlier diagnostic image, the later diagnostic image and the area or areas corresponding to a difference between the earlier diagnostic image and the later diagnostic image superimposed on the later diagnostic image. Preferably, the system according to the invention further comprises a display arranged to display the earlier diagnostic image, the later diagnostic image and the area corresponding to a difference between the earlier diagnostic image and the later diagnostic image.

These and other aspects of the invention will be explained in more detail with reference to figures. In the figures like elements are depicted using like reference signs. It will be appreciated that the figures are given for illustrative purposes only and may not be used for limiting the scope of the appended claims.

Brief description of the drawings

Figure 1 presents in a schematic way an embodiment of a method according to the invention.

Figure 2 presents in a schematic way an embodiment of a contrast adjustment step.

Figure 3 presents in a schematic way an embodiment of a region of interest identification step.

Figure 4 presents in a schematic way an embodiment of a further embodiment of a region of interest identification step.

Figure 5 presents an embodiment of an earlier diagnostic image.

Figure 6 presents an embodiment of a later diagnostic image.

Figure 7 presents an embodiment of a diagnostic image comprising color coded data on a difference between the earlier diagnostic image and the later diagnostic image.

Detailed description of the drawings

Figure 1 presents in a schematic way an embodiment of a method according to the invention. It will be appreciated that the given sequence of steps is not exhaustive, nor binding. In accordance with an embodiment of the method of the invention a pyramidal image decomposition may be carried out at step 2. For example, for each analyzed image a lowpass pyramid may be generated, for example by first smoothing the image with an appropriate smoothing filter and then sub- sampling the smoothed image by a factor of two along each coordinate direction. As this process proceeds, the result will be a set of gradually more smoothed images, wherein in addition the spatial sampling density is decreasing.

The low pass filter may be realized with a two dimensional convolution, implemented using two successive one dimensional convolutions along the rows and columns of the image.

At step 4 suitable image processing (trimming) may take place. This step will be explained in more detail with reference to Figure 2.

At step 6 the image is suitably processed, an image central point may be established. This step is explained in more detail with reference to Figure 3. Next, suitable identification of the region of interest (ROI) may be carried out at step 8. Those skilled in the art will readily appreciate which per se known method may be used for this purpose. More particular details on the ROI identification step are given with reference to Figure 4. Afterwards, at step 10 suitable landmarks may be identified. In case when the image comprising bony structures it may be preferable to use bones as suitable markers for purposes of image registration between the earlier diagnostic image and the later diagnostic image. However, other structures or organs may be used for this purpose as well. For example, for identification of markers use may be made of delineation of other areas and borders at the respective steps 9a, 9b. The borders may be obtained using knowledge on the expected structures in the diagnostic image. For example, for thorax images, ribs may be delineated using Gaussian filters 11.

After the set of markers is established at step 10, the method according to the invention proceeds to steps of superpositioning of the earlier diagnostic image with the later diagnostic image at step 12, adjustment of the images, notably the regions of interest, at step 12a and subtraction of data in the images (or regions of interest) at step 12b.

Figure 2 presents in a schematic way an embodiment of a contrast adjustment step. It is found that during the processing of diagnostic images, such as radiographs, bright lines may appear in areas surrounding the actual image and cause undesirable interference with the feature detection algorithm used for detection of patterns in the diagnostic image under consideration. The exemplary steps 12 - 18 may be applied to mitigate this problem.

First, at step 12 duplication of low resolution image may be carried out. At step 14 a suitable auto contrast algorithm may be applied. The auto contrast procedure expands the dynamic range of the image intensity by mapping the lightest and darkest pixels to maximum (for example, 255) and minimum (0) possible values. To determine the lightest and darkest pixels, histogram clipping may be used. The maximum value of the image intensity (lightest pixels) may be determined so that only 0.5% of the image pixels have the intensity higher than the maximum. For the minimum the same levels are applied, only 0.5% of the pixels have the intensity lower than the maximum. It will be appreciated, however, that any pre-determined value may be used for setting the respective lower and upper cut-off maximum/minimum intensities.

Next, at step 16 a search may be carried out for the first 20% of the image from each side; top, bottom, left and right. The first row with a mean intensity value above 10 may be treated as signal level, the respective boundary of the image. All rows towards the image border may be duplicated with the intensity values found in this row.

At step 18 the upper boundary may be set. For the top boundary additional steps may be performed. The row with the minimum intensity is searched within the predetermined level, say 5% of the image. All rows located above this minimum intensity row may be normalized by using I (x) = I (x) Smin/S, wherein I(x) refers to the intensity of the pixel at location x. Smm refers to the intensity sum of found minimum row and S refers to the sum of all pixel intensities of current row.

Figure 3 presents in a schematic way an embodiment of a region of interest identification step. Thorax X-ray examination may contain a great number of informational structures. The region of interest (ROI) for thorax are lung areas. It is found that for enabling accurate assessment of lung areas it may be preferable to eliminate other structures from the image, except for markers used for image registration.

In this exemplary embodiment at step 22 the Gaussian of image A may be calculated as the convolutions of image intensity I(x,y) with the

Gaussian 2 ^nd kernel function. Preferably sigma = 25 is used yielding G(25).

At step 24 the lowest intensity is determined. For this purpose the Gaussian(25) may be divided into two equal sides. For each side a search may be carried out on the top half of the image with the exclusion of a

predetermined portion (20% for example) of the image boundaries for the lowest intensity signal of G > GmmLeft, GmmRight.

At step 26 an image region may be calculated. An image region is a connected image area containing pixels with the same close value of intensity. A list of image regions may be calculated using the intensity GmmLeft, GmmRight for both sides of image G(25). Regions with width and height more than ¼ of image size may be excluded from the list. In each side of image G(25) one may find a central region which is closest to the center of the image. The central region may be used to determine coordinates of centre of both lungs in a thorax image, (Xi _e ft, Yieft) and (X _ng ht, Y _ng ht) as

X— (Xmin+Xmax)/2; Y = (Ymin+Ymax)/2.

Xmin - minimum horizontal coordinate of pixels from central image region;

Xmax - maximum horizontal coordinate of pixels from central image region; Ymin - minimum vertical coordinate of pixels from central image region;

Ymax - maximum vertical coordinate of pixels from central image region.

It will be appreciated that this example is given for one side of the lungs of a thorax photo taken as an exemplary embodiment of the diagnostic image. In practice the calculation may be done for both sides.

At step 28 a central point may be identified. For example, thoracic central point may be calculated as follows:

Xcenter ⁼ (Xleft + Xright)/2

Ycenter— (Yleft Yright)/2

Next, identification of lung areas may be carried out, which is described with reference to Figure 4.

Figure 4 presents in a schematic way an embodiment of a further embodiment of a region of interest identification step. The identification of the region of interest (ROI) may be done using Gaussian filters. This technique results in a clear separation between the ROI and structures outside this region. The image region corresponding to the lung area and it's bounding rectangle may be stored.

At step 32 unsharp masking may be carried out. For the unsharp masking algorithm a Gaussian G(3) of image A may be formed with sigma 3, for example, a Gaussian G(40) of image G(3) may be calculated with sigma 40. It will be appreciated that different values for sigma may be used as well. An unsharp mask image may be calculated using G(3) and G(40) with an amplification factor of 5. It will be appreciated that these examples are not limiting.

At step 34 combining of lung regions is carried out. The Gaussian filter used at step 32 results in a clear separation between the lung and outside area. Within the lung area the collarbone may cause a complete region separation between the top part of the bottom part of the lung. In order to achieve 1 image region of the total ROI for the left and right side lines may be drawn from the weighted center upwards with the following restrictions and settings:

lines drawn with a 3 pixel width;

- dl line is drawn from the weighted centre coordinate till height coordinate image hc/7;

dl length is restricted to distance between the weighted centre and dc/2;

drawn lines in left and right side are inclined to centre of image by dl/8.

At step 36 a threshold adaptation algorithm may be applied/ Adaptation may be done iterative ly with thresholds decreasing from 50 till 27 with steps of 3 in intensity levels. However, other values may be used. For each step a binary image image region may be constructed from USM threshold.

The image regions created during the threshold adaptation algorithm may be tested against the following rules:

outer positions of image region are not less than a predetermined number of pixels (for example, 5) away from the image boundary;

Xieft, Yieft, Xright, Rright coordinates are located inside the image region;

X and Y coordinates of top most pixel for both sides may be annotated as lung top coordinates (ptLeft, ptRight) he distance between X _ce nter and x coordinate for lung top is not allowed to be greater than the distance between Xcenter and xieft;

Difference in Y coordinates of lung top coordinates is not allowed to be greater than distance/const, for example const = 10.

It will be appreciated that different approaches for implementing testing may be used. At step 38 all the above requirements are checked. Should they be met, the image region is treated as the ROI area. If only one region is found after adaptation iterations it is excluded and mirroring of coordinates from other side around X _ce nter is used. It will be appreciated that use can be made of suitable structures, such as lung top, lung border or the like.

Figure 5 presents an embodiment of an earlier diagnostic image. In this particular embodiment a thorax image is used. In the thorax image 50 the relevant structures may relate lung structures 51a and 51b.

When the earlier diagnostic image and the later diagnostic image are processed as described above, the images are registered. However, it may be necessary to carry out the step of image registration prior to image warping. Because relevant differences are to be investigated with respect to the present state of the patient, it is found advantageous to warp the earlier diagnostic image to the later unrectified diagnostic image.

The procedure of image warping is known per se and may comprise the following steps:

checking and adjustment of the feature detection points

(markers);

normalization of point coordinates;

estimation of widths of specific structures, such as bony structures (ribs);

calculation of a central point;

selection/normalization of points in the surroundings (ribcage points);

labeling;

filtering for displacement;

correlation refinement for motion correction;

recalculation of reference points in the surroundings (rib cage points);

calculation of displacement quality measures; warping the image.

Figure 6 presents an embodiment of a later diagnostic image 60. In this image the relevant structures are the ribs 62a, 62b and the lung 61a, 61b. Figure 7 presents an embodiment of a later diagnostic image 70 comprising color coded data 71, 72, 74 representing the difference data, overlaid on the later diagnostic image. For the convenience, the image may be appended with a suitable legend comprising segments 73a, 73b, 73c. It will be appreciated that the difference image may comprise positive areas and negative areas. Interpretation of the difference data may be dependent upon a clinical case.

It is found that provision of such quantified overlaid image comprising accurate information on the difference in volumes of regions of interest of the same patient over time has substantial added value. It will be further appreciated that because the invention is based on analysis of actual images its results are more robust than results obtained using the methods known in the art, such as applications based on pattern recognition and modeling.

It will be appreciated that the invention is not limited to the embodiments which have been just described. It will be appreciated that the invention may be practiced otherwise than as described.