The present invention relates to a method for segmentation of an object of interest from a three dimensional image data set, wherein a deformable model is adapted to the boundary of the object of interest. The invention further relates to an image processing system comprising image data processing means for segmentation of an object of interest, using the multi layer deformable mesh model, relates to a medical examination apparatus, and relates to a computer program which is adapted to perform the segmentation-method of the invention.

Deformable model based segmentation has proven to be an adequate technique for organ segmentation in volumetric medical images. They usually make use of prior knowledge about the geometric shape of the anatomic structure of interest.

In order to improve segmentation, complex models have been developed that consist of multiple interacting parts instead of only one single surface. Most of those models are focused on the anatomical structure of the target organ, e.g. multi-chamber heart models [Y. Zheng, A. Barbu, B. Georgescu, M. Scheuering, and D. Comaniciu, "Fast automatic heart chamber segmentation from 3d ct data using marginal space learning and steerable features," in 11th International Conference on Computer Vision (ICCV), 2007, pp. 1-8; and O. Ecabert, J. Peters, H. Schramm, C. Lorenz, J. von Berg, M. J. Walker, M. Vembar, M. E. Olszewski, K. Subramanyan, G. Lavi, and J. Weese, "Automatic model-based segmentation of the heart in ct images," Medical Imaging, IEEE Transactions on, vol. 27, no. 9, pp. 1189-1201 , 2008] or multi component cardiac left ventricle models [S. Zambal, J. Hladuvka, K. Biihler, and A. Neubauer, "A fully automatic system for segmentation and analysis of the left and right ventricles of the heart using a bi-temporal two-component model," Computer Assisted Radiology and Surgery (CARS), pp. 93-94, 2007]. T. Shen and X. Huang, "3d medical image segmentation by multiple-surface active volume models," in MICCAI (1 ), 2009, pp. 1059-1066, use an outer surface to build a reference frame for the adaptation of two inner surfaces in the application of heart segmentation. The goal of these algorithms is to make the segmentation of all single organ parts more robust by segmenting multiple structures simultaneously. However, the models used in these algorithms are simply a concatenation of single surface models which are somehow coupled during the deformation process. The geometry of the single models does not provide any depth information.

A class of volumetric models has been proposed in N. Barreira and M. G. Penedo "Topological active volumes," EURASIP J. Appl. Signal Process., vol. 2005, pp. 1939-1947, 2005, where the model's interior consists of a 3D grid that can be torn apart in order to segment multiple structures. The internal grid consists of regular cubes. Since the model can be torn apart, the internal structure is not used for form preservation. In the context of model based organ segmentation, this is a limitation, because arbitrary forces may be applied to the surface of the model. That means, the surface may be folded locally which is not desirable since the surface of a real organ can not have such foldings. By using the internal points for form preservation, it is much more unlikely that a surface would fold, because a surface point is also tied to internal points. However, using a grid model as described above for this purpose would be problematic, since a grid is always aligned referring to the axes of a certain coordinate system. This means, the model behaves differently if a force is applied in direction of a coordinate axis or diagonal to it.

Another class of volumetric models [F. Laffargue, M. Fradkin, J-M. Rouet. Mesh models with internal discrete elements. Patent application publication. No.:

US 2007/0165948 A1. Jul 2007] builds an unstructured deformable mesh model from a single outer surface. The interior elements are represented by tetrahedrons and can be refined based on the size of the surface triangles. The interior elements may have a different structure as the surface elements. In addition, neighboring internal elements may vary significantly in size. Therefore, the model leads to implausible deformation behaviour of the surface, if the model's surface is adapted to the boundaries of an object to be segmented.

An object of the invention is to provide a deformable model which leads to an improved segmentation of an object of interest. In particular, the invention relates to the model based segmentation in medical imaging. According to a basic idea, a Multi-Layer Deformable Model (MLDM) based on a composition of several interconnected layers is defined and used for segmentation. The geometry can be coupled to a deformation logic which is used to detect boundaries of an object of interest (e.g. an organ). An organ may have cavities, encapsulations of organ foreign tissue. Boundaries are the limits of the object to the exterior and possibly to interior foreign material or voids. The boundaries, even low contrasted boundaries are detected within an image of the object and its surrounding.

In particular, the model's outer layer is adapted to the outer boundaries of an object of interest by identifying image data set points which lie on the boundary of the object of interest. The boundary points are assigned to mesh points of the MLDM and these mesh points are displaced towards the boundary points in an iterative process until the mesh model fits to the boundaries of the object of interest.

In particular, the present invention provides a method for segmentation of an object of interest from a three dimensional image data set, wherein a deformable multi layer mesh model is adapted to the boundary or boundaries of the object of interest, wherein:

- the deformable multi layer mesh model comprises at least two layers, each layer having a mesh structure of interconnected mesh points, wherein each mesh point of the at least two layers is connected with at least one other mesh point of the same layer by a layer link,

- a first layer forms an outer surface of the model,

- at least one inner layer is located inside the outer surface of the model,

- each of the mesh points of the first layer is connected with a corresponding mesh point of a neighbouring inner layer by a depth link, and

- in case of any further inner layer, each mesh point of each inner layer is connected with a corresponding mesh point of each neighbouring inner layer by a depth link,

The model is a mesh model which means that points of the model are interconnected by connections, preferably by connections which extend along a straight line. The connections can be used to transfer forces from a first mesh point to a second mesh point which is connected to the first mesh point. At least some of the connections are used to transfer forces in order adapt the multi layer mesh model to the boundary or boundaries of an object of interest. A force, preferably an external force, may act on one or more first mesh points of the multi layer mesh model, preferably on one or more mesh points of the first layer. The force is transferred via connections to other mesh points which are directly or indirectly connected with the first mesh point and the structure of connected mesh points is moved in order in order adapt the model to the boundary or boundaries of an object of interest. However, some mesh points may be fixed so that they cannot move. The connections between mesh points can be more or less stiff, which influences the transfer of forces between connected points and the adaptation of the model to the boundary or boundaries of an object of interest. More detailed explanations will be given below.

Each point of the model corresponds to a location within the three-dimensional space in which the object to be segmented is located. As mentioned above, the model has at least two layers, an outer layer and an inner layer. The outer layer is also called the first layer and forms the outer surface of the model. The terms "outer layer" and "first layer" are used in this disclosure interchangeably. The outer layer is to be deformed so that it fits to the boundaries of the object of interest. The inner layer or the plurality of inner layers comprises mesh points which are interconnected by connections to the neighboring layer(s). Each mesh point of the inner layer has one connection to a corresponding mesh point of the neighboring layer(s). In case of two layers, the neighboring layer of the inner layer is the first layer which forms the outer surface of the model. The connections between the corresponding points of neighboring layers are called "depth links". The connections between points within the same layer are called "layer links".

As will be described in more detail, the depth links are a special feature of the model which help to improve the model's ability to deform naturally to the surface of the object of interest. For example, the object of interest may comprise recesses or indentations at the surface. Without the depth links, the model might not be able to deform to a surface shape which fits to the surface of the object including the recess or indentation. Therefore, the depth links are an important feature of the model.

Another feature of the model is the fact that each point of the outer layer has a corresponding point of the neighboring inner layer. In case of more than one inner layer, the same applies to each inner layer which has a neighboring inner layer. For example, in case of three layers, the first layer and two inner layers, each point of the first layer has two corresponding points, one in each inner layer, and the point in the first layer is connected by a depth link to the corresponding point in the neighboring inner layer and this corresponding point in the neighboring inner layer is connected by another depth link to the corresponding point in the next inner layer. As a result, a chain of consecutive depth links is formed.

In particular, the deformable multi layer mesh model is adapted to the boundary of the object of interest by performing at least the following steps:

a) identifying boundary points of the object of interest, by evaluating the three dimensional image data set,

b) assigning the boundary points to mesh points of the first layer,

c) assigning stiffness values to depth links and to layer links, wherein each stiffness value indicates to which extent the mesh points, which are connected by the respective depth link or by the respective layer link, are allowed to move relative to each other,

d) displacing mesh points of the first layer towards their corresponding boundary points, wherein grades of displacement of the mesh points of the first layer depend on the stiffness of the depth links and layer links according to the assigned stiffness values,

e) repeating at least steps c) and d) in order to fit the mesh model to the

boundaries of the object of interest.

According to the foregoing, at least some of the depth links, preferably all depth links have an assigned stiffness value, and at least some of the layer links, preferably all layer links have an assigned stiffness value. Consequently, each depth link and each layer link is less or more stiff. As a result, the point on the outer side of the depth link, in particular the point in the first layer is displaced towards the corresponding boundary point to an extent that depends on the stiffness of the depth links and layer links. Examples how the stiffness values are chosen will be given below. In particular, the stiffness value may depend on the number of neighboring points (the points which have connections to a specific point) or it can be the result of a statistical analysis which takes a large number of real objects and their shapes into account. Surface regions with large variations in shape may be modelled using smaller stiffness values than regions with smaller variations.

As stated above, stiffness values are not only assigned to depth links, but also to layer links between points within the same layer. For example, each point in the first layer may have 3 to 8 neighboring points in the first layer which are interconnected to the point by layer links. In this case, each point has, according to the number of layer links within the same layer, 4 to 9 neighboring points which are interconnected to the point. One of these interconnected points is the corresponding point in the

neighboring inner layer. Preferably, each point in the first layer has only one interconnection to a point in the neighboring inner layer, namely to the corresponding point via the depth link. Preferably, the same applies to each inner layer which exists. For example, in case of three layers, the points in the middle layer have two interconnections to points in neighboring layers (depth links) and additional layer links to points in the middle layer.

The displacement of mesh points of the first layer towards their corresponding boundary point depend on the stiffness of the depth link of the point and also on the stiffness values of layer links to points within the same (first) layer. Since these neighboring points also have further neighboring points which are connected to these points, and since some or all of these further connections also have stiffness values, the displacement of mesh points of the first layer depends on a mesh of points and their interconnections with the corresponding stiffness values. Amending a single stiffness value within the mesh would influence the displacement of a large number or all mesh points of the first layer, unless certain points are fixed or interconnections have a very large stiffness.

According to a specific embodiment of the model, forces can be assigned to at least some of the points in the first layer. If applied, the force is effective so that the point is forced towards its corresponding boundary point (if any). However, whether the point in the first layer is actually displaced towards the corresponding boundary point depends on the stiffness of the interconnections within the model, and optionally, also depends on other factors. Advantages of using a multi layer mesh model (MLDM) together with the described adaptation logic in image segmentation are the following, without being restricted to these advantages:

1 Internal points of the MLDM can be used to sample the interior of the object of interest. This additional information can be used to

1.1 detect low contrasted boundaries between the object of interest, e.g. a

human or animal organ, and neighboring structure, e.g. neighboring tissue of the organ. This can be done by evaluating mesh points whether they are inside or outside of the object of interest. In case one or more mesh points are outside, the positions of remaining points which are inside can be used to force the points of the first layer to adapt towards these positions. This prevents the model from leaking into structures that do not belong to the object of interest, as for example organ foreign tissue. Therefore the segmentation robustness is improved.

1.2 detect interior structures within the object of interest. Mesh points of inner layers can be used to detect contrasted vessels within a human or animal organ, e.g. by evaluating whether the image intensity at the mesh points of inner layers matches typical vessel intensity. Using this knowledge, the model can be adapted to the real boundaries of the organ. A false adaptation to said internal tissue like vessels can be prevented.

1.3 increase the search space at model initialization. If the model is placed only partly over the object of interest, the image intensity at the mesh points of inner layers is also evaluated. This accelerates the convergence of the model towards the boundaries. Furthermore, it makes the adaptation process more robust, since the model can recover more easily from a false positioning. Therefore the segmentation process is less dependent on the initial model positioning.

1.4 detect anomalies in the object of interest (e.g. segmenting an organ and detecting lesions in one step).

2. Higher stability of the model because of connections in depth direction. Therefore, less surface artifacts like self-intersections of surface mesh points occur. 3. A dynamic flexibility allows the adaptation of the model to indentations of the boundary and to detected cavities. Therefore, the model can accurately fit the underlying data while preserving its form stability.

4. Inner layers of the MLDM can be created based on a scaling of the first layer.

Therefore the inner layers have the same or similar shape as the first layer. This common shape increases the stability to the whole model while allowing a smooth deformation at the same time. In contrast to unstructured internal elements or regular grids used in other volumetric models, forces applied in surface normal direction can be linearly propagated to lower layers. This eases the mathematical computation of the effect of a deformation force applied on the model.

The general field of the application is the segmentation of structures, for example in the fields of medicine, engineering, material science and others. In the field of engineering, the present invention can, for example, be used for segmentation of components, building elements, and constructional elements. The preferred field of application of the method of the present invention is the field of medical image segmentation, i.e. segmentation of structures of the human and animal body in computer aided medical diagnostics technologies. Preferred objects of interest in this connection are organs, bone structures, boundaries between specific regions, lesions or abnormalities within the human and animal body.

In the method of the present invention, a deformable multi layer mesh model is adapted to the boundaries of the object of interest. The adaptation process starts with an initial multi layer mesh model and the adaptation process is performed preferably interatively.

The deformable multi layer mesh model which is used in the method of the present invention is the MLDM described above that comprises at least two layers, each layer having a mesh structure of interconnected mesh points. One of the at least two layers is the so-called first layer, forming the outer surface of the model. The other layer(s) are the so-called inner layer(s). The interconnected mesh points of the same layer form polygons with at least 3 vertices, for example triangles, quadrangles, pentagons etc. One layer can comprise different types and shapes of polygons. The interconnections do not have to be located in a common plane. The initial multi layer mesh model, which is used as the starting point in the adaptation-process can be created is follows:

An initial first layer can be based on a geometric surface mean model of the object of interest. However, the model does not necessarily have to be based on a geometric mean. Any model that is similar to the object of interest can be used. For example, if the object of interest usually does not vary significantly between subjects, the first layer can be based on the object shape of a certain individual.

A surface mean model can be created by averaging manually created reference forms of the object of interest. For example, the form of a human organ can be created manually by contouring the organ boundaries in images of Computed

Tomography of a human body and subsequently creating a mesh from the contoured organ using e.g. the Marching Cubes algorithm. Several manually created forms, created on basis of different data sets, are used to create a geometric surface mean model of the object of interest. The surface of the mean model, represented by a mesh structure of interconnected mesh points, is the first layer of the MLDM and forms the outer surface of the MLDM.

Starting with the initial first layer, an inner layer, which can also be called depth layer, is added by scaling the surface mesh structure of the first layer with a factor in inverse normal direction such that a constant distance between the layers is reached. The distance is equal to the length of the depth link between the corresponding points. However, deviations of the constant distance can be allowed in specific cases.

The term "normal direction" means in direction of the normal vector on the surface of a layer, and the orientation of the normal vector is the orientation towards the outside of the object of interest. It can be determined on basis of a geometric surface model of the object of interest, where "outside" and "inside" of the object is. The creation of a geometric model was described before. The term "inverse normal direction" means with opposite orientation of the normal vector on the surface of a layer. Therefore, the orientation of the inverse normal vector is the orientation towards the inside of the object of interest.

In case of segmentation of organ, the distance between the layers is preferably 1 to 40 mm, more preferably 10 to 20 mm.

The new point coordinates for the new layer L _k are given by mu = (- n s + s,). n,- denotes the normalized vector in inverse normal direction and f _s denotes the scaling factor and s, denotes the coordinate of a point in the surface layer.

In curved regions of the first layer, intersections of the normal vectors on the surface of the first layer or intersections of depth links or chains of consecutive depth links may occur. Preferably, the inner layer is optimized to assure that its distance to the neighboring first layer is constant and that its mesh structure remains similar to the neighboring first layer.

For example, this can be done manually using mesh deformation tools to correct the inner layer. Here, the surface is first scaled to a smaller size and afterwards the user drags the resulting surface such that a constant distance to the first layer is reached. The new point coordinates l _ki for the new layer L _k are optimized such that

PM ^N ' fct

(Formula 1 ) becomes minimal.

PM denotes the number of layer mesh points and NM the number of neighbors for layer point l _M . m, and m, denote the points of the outer layer after scaling. U i denotes a reference point that is generated from user input as follows: the user sets a number of points in regions of strong curvature. From those points a new mesh U _k with the points U i is interpolated. The optimization is then performed between the three meshes M, L _k and U _k as described in Formula 1 . The optimization basically preserves the distances from a point to its neighbors in comparison to the uncorrected layer mesh. A detailed explanation is given in: Lorenz, C. and v Berg, J., "A comprehensive shape model of the heart," Medical image analysis 10(4), 657-670 (2006).

As an alternative, intersections can be avoided by automatically optimizing all points of the resulting inner layer such that their relative distances to neighboring points are preserved. This can be done by replacing the term (l _ki - Uki) ² from Formula 1 with a constraint term which includes the desired distance d to the surface layer points s,:

{{l _ki - _Si )-df

The afore described procedure is repeated for the desired number of further inner layers, wherein the respective last inner layer is used as the starting point for creating the next inner layer.

After building the desired number of inner layers each of the mesh points of the outer layer is connected with a corresponding mesh point of a neighbouring inner layer by a so-called depth link, and for each further inner layer, each mesh point of each inner layer is connected with a corresponding mesh point of each neighbouring inner layer by a depth link. By creating depth links forces applied in surface normal direction can be linearly propagated to inner layers. The "corresponding mesh point" is the mesh point in the corresponding position within the mesh structure in the neighboring inner layer. Corresponding mesh points with point coordinates and / _/ <+?, _/ are created by scaling the mesh structure of a layer Lk with a factor f _s in order to build a neighboring inner layer Lk+i with a similar mesh structure, as described before.

The process of adaptation of the deformable multi layer mesh model to the boundary of the object of interest may be performed as follows:

In a first step a), so-called boundary points of the object of interest are identified by evaluating the three-dimensional image data set. Boundary points are points of the data set which are potentially located on the boundary of the object of interest, e.g. the boundary of an organ of the human or animal body. In order to identify boundary points, mesh points of the first layer of the MLDM (mesh points of the outer layer) can be taken as starting points. The boundary points can be found e.g. by sampling gray value profiles in the data set along the normal vector of the outer layer, starting at the position of a mesh point of the outer layer. The search is preferably performed in normal direction and in inverse normal direction. Usually the search range for boundary points is limited to an extent between 5 and 20 mm, starting from a mesh point of the outer layer. The position of the profile with the best fit to previously trained profiles is taken as the boundary point. The trained profiles are extracted from images where the model is already aligned with the real boundaries, e.g. through user interaction. This is a standard procedure used in active shape models [Y. Zhen, A. Barbu, B. Georgescu, M. Scheuering, and D. Comaniciu, "Fast automatic heart chamber segmentation from 3d ct data using marginal space learning and steerable features," in 11th International Conference on Computer Vision (ICCV), 2007, pp. 1- 8; T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active shape models - their training and application. CVIU, 61 (1 ):38-59, 1995]. An alternative is to search along the normal for the strongest gray value gradient and use this position as the position of the boundary point.

After a boundary point has been found, the boundary point is assigned to a mesh point of the first layer in step b). In case that the boundary point was found by starting at the position of a mesh point of the outer layer and searching along the normal vector of the outer layer, the boundary point is preferably assigned to the mesh point which was used as the starting position. The boundary point which was assigned to a mesh point is also called the "corresponding boundary point" of the mesh point.

In case that no boundary point can be found with the above-described procedure, a default boundary point can optionally be set and assigned to a mesh point of the first layer. If a default boundary point shall be set, information is needed whether the mesh point of the first layer is inside or outside of the object of interest in order to determine where the default boundary point shall be set. This information can be obtained in the following manner: gray value samples from the data set in the neighbourhood around a mesh point of the first layer are taken to determine whether the point is inside or outside of the object of interest. The samples may be simply checked whether their intensities fall into a predefined gray value range. In addition a texture may be generated from the samples in order to check whether the pattern of the structure is similar to the according object of interest. Texture is commonly defined as a classification characteristic derived from the analysis of an image region in its frequency domain. Usually the texture at a certain image point can be approximated to belong to a certain class of known textures. The class of known textures can be obtained by a prior statistical analysis of a set of texture samples, manually classified by trained experts. Based on the result of the sample check the mesh point of the first layer assumes a state indicating whether it is inside or outside of the object of interest. If the mesh point of the first layer is outside of the object of interest, the default boundary point is set on the side of the inner layer(s) of the MLDM, preferably in inverse normal direction, and if the mesh point of the first layer is inside of the object of interest the default boundary point is set on the side which is opposite to the side of the inner layer(s), preferably in normal direction. In one embodiment of the invention, the default boundary point is set in normal or inverse normal direction starting from the mesh point of the first layer to whom it is assigned. The distance of the default boundary point to the mesh point of the first layer to whom it is assigned is preferably 1 to 40 mm, more preferably 5 to 15mm.

In one embodiment of the present invention a boundary point is assigned to every mesh point of the first layer. This can be either a found boundary point or a default boundary point.

In order to use also mesh points of the inner layer for the adaptation process, boundary points may also be assigned to mesh points of inner layer(s).

In a special embodiment a boundary point is assigned to every mesh point in the MLDM.

A boundary point can be set at the same position as a mesh point to which it is assigned to. This case leads to the same result as the case when no boundary point is assigned to said mesh point.

In another embodiment, mesh points of inner layers can be used as boundary points for mesh points of the first layer which is explained in more detail afterwards. In step c) stiffness values are assigned to the depth links and to the layer links. The stiffness values are part of a deformation-logic that controls the adaptation of the MLDM to the boundary of the object of interest. The stiffness values indicate to what extent corresponding mesh points of neighbouring layers are allowed to move relatively to each other. A weight (hereinafter abbreviated as "w"), representing the stiffness value, is assigned to the depth links.

In one embodiment a weight w in the range of 0 to 1 is assigned to the depth links. A weight of 0 means that two points connected by a depth link can deform

independently of each other and weight of 1 means that two points connected by a depth link are rigidly connected. Interim values between 0 and 1 indicate a state between the two extreme cases. In a special embodiment, a value of either 0 or 1 is assigned without using any interim values.

For example, if all stiffness values are set to 0.5, one boundary point which is assigned to a mesh point of the first layer will affect all mesh points in all layers.

However, the degree of displacement of a given mesh point is decreasing with the amount of interconnections between this mesh point and the mesh point of the first layer to whom the boundary point is assigned to.

The function of the stiffness values w, which were assigned to the depth links and layer links in step c), is a best possible form preservation of the MLDM when the mesh points of the first layer are displaced towards their corresponding boundary points in step d). The term "form preservation" means that the form of the MLDM should be as similar as possible to the initial MLDM which is used as the starting point in the adaptation process, wherein the term "form" means the three dimensional shape of the MLDM. Particularly, the term "form preservation" means that local deformations in comparison to the initial MLDM are avoided as far as possible.

In order to reach form preservation, the values w are chosen such that they reflect the volumetric stiffness of the object of interest. For example, if the goal is to segment an organ in two different time steps (e.g. pre- and post-operative scans) the adapted model from the pre-operative scan can be taken and its stiffness values can be set to the estimated elasticity of the organ tissue (e.g. w = 1 in the case of a totally rigid bone).

More generally speaking, the model can be used to segment the same object at two different points in time from two corresponding image data sets. Information from the image data at the earlier time and/or from the mode, which was fitted to the boundaries of the object at the earlier time, can be used to segment the object using the image data at the later time. In particular, this information can be the location of the mesh points of the first layer. The information can be used to set initial values of the location of the mesh points at the beginning of the segmentation using the image data of the later time and/or to set the initial values of the link stiffness.

Form preservation can be realized by optimizing the point coordinates of the MLDM such that the lengths of the depth links and the layer links stay similar in relation to each other compared to a reference MLDM. The reference MLDM can be an arbitrary shape, e.g. the non-deformed initial MLDM. As an alternative, the reference MLDM can be created using the method of well known statistical shape models [T. F.

Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active shape models - their training and application. CVIU, 61 (1 ):38-59, 1995]. First, a set of images and a set of training MLDMs is needed where the MLDMs are already aligned to the boundaries of the object of interest in the image (e.g. manually). Using principle component analysis, a statistical shape model can be built from those training MLDMs. The reference MLDM is then computed by projection of the set of boundary points to the training space. In the training space, the set of boundary points is equal to the set of points in the MLDM.

In step d) the mesh points of the first layer are displaced towards their corresponding boundary points according to the above principles. The grade of displacement depends on the distance of the point to its boundary point and the stiffness values. If, for example, a stiffness value of 0.3 is used for all depth links and layer links the displacement of a mesh point of the first layer is usually in the range of 1 to 10 mm, preferably with a distance between the layers of 1 to 40 mm, more preferably 10 to 20 mm. At least steps c) and d) as described before are repeated in iterative manner in order to fit the mesh model to the boundary of the object of interest. If the search for boundary points is only performed once, only steps c) to d) need to be repeated. Otherwise, steps a) and b) can be repeated as well. The adaptation process is finished when the MLDM adequately fits to the data, i.e. when all mesh points of the first layer of the MLDM are matched with their corresponding boundary points within the limits of the form preservation defined by the stiffness values w. If, for example, the values of w are close to 1 (e.g. w=1 ) the first layer points will hardly exactly match the boundary points, since the stiffness prevents the model from deviating from the reference shape. If, for example, the values of w is 1 (w=1 ) the first layer points will never exactly match the boundary points, since the stiffness prevents the model from deviating from the reference shape.

In one embodiment of the method the invention it is determined for mesh points of the one or more inner layer(s) whether the mesh point is located outside or inside the boundary of the object, by evaluating the three dimensional image data sets.

Preferably, the determination is started at a mesh point of the first layer and the status of the neighboring mesh points of the inner layer is checked in direction of the depth links to inner layers. Following the depth link for a mesh point of the first layer, the state of the neighboring mesh points of the inner layer is successively checked.

In particular, gray value samples from an image region of an inner layer mesh point can be taken to determine whether the points are inside or outside of the object of interest. The samples may be simply checked whether their intensities fall into a predefined gray value range, i.e. the range typical for the object of interest. For example in case of CT most organs intensities are limited to a certain gray value range. In addition a texture may be generated from the samples in order to check whether the pattern of the structure is similar to the according object of interest.

Based on the result of the sample check the inner points assume a state indicating whether they are inside or outside of the object of interest.

A yes-state or a no-state can be assigned to the inner layer mesh points in order to express whether a point is inside or outside of the object of interest. Instead of using yes/no states for the inside/outside decision of internal points, a probability between 0 and 1 of being inside/outside can be assigned.

In a special embodiment, the status whether the mesh points of one or more inner layer(s) are located outside or inside the boundary of the object is only determined for the inner layer which neighbors the first layer. In other embodiments, the status is determined for the mesh points of all inner layers.

The states of the inner layer mesh points - inside or outside of the object of interest - can be used to detect interior structures within the object of interest, and/or to make (during the adaptation process) a decision about the assignment of stiffness values and so-called displacement values, as explained hereinafter.

The states of the inner layer mesh points - inside or outside of the object of interest - can further be used to model an indentation of the boundary of the model of interest. In order to model an indentation of the boundary of the model of interest

• a first mesh point of the first layer is identified which is located outside of the boundary of the object of interest, wherein a corresponding inner layer mesh point which is connected to the first mesh point of the first layer by a depth link and optionally at least one further inner layer mesh point which is connected to the first mesh point of the first layer by a chain of consecutive depth links, is/are located outside of the boundary of the object of interest as well,

• at least one second mesh point of the first layer is identified, which second mesh point is located inside of the boundary of the object of interest and which is interconnected to the first mesh point via a layer link within the first layer,

• the stiffness value(s)

- which is assigned to the depth link from the first mesh point of the first layer to its corresponding inner layer mesh point and

- if such further inner layer mesh points exist, which are assigned to further depth links in the chain of consecutive depth links to all further inner layer mesh points which are located outside of the object of interest are increased, and • the stiffness value(s) which are assigned to the layer links

- from the at least one second mesh point of the first layer to the first mesh point of the first layer,

- from the inner layer mesh point, which is connected to the at least one second mesh point of the first layer by a depth link, to the inner layer mesh point, which is connected to the first mesh point of the first layer by a depth link, and

- if such further inner layer mesh points exist, from further inner layer mesh points, which are connected by a chain of consecutive depth links to the at least one second mesh point of the first layer, to further inner layer mesh points, which are connected by a chain of consecutive depth links to the first mesh point of the first layer and which are located outside of the object of interest,

are decreased.

Moreover, inner layer mesh points which are located close to the boundary of the object of interest can be used as boundary points for the mesh points of the first (outer) layer. More preferably, inner layer mesh points which are located inside, and closer to the boundary of the object of interest than any other mesh point which is inside the object, can be used as boundary points for the mesh points of the first (outer) layer. This can be done as follows:

A check is done for a mesh point of the first layer and for each inner layer mesh point, which is connected to said mesh point of the first layer by a depth link or by a chain of consecutive depth links, whether they are located inside or outside of the object of interest, and

if the mesh point of the first layer is outside of the object of interest and at least one inner layer mesh point, which is connected to the mesh point of the first layer by a depth link or by a chain of consecutive depth links, is located inside the object of interest, one of the at least one inner layer mesh points which is inside the object of interest is assigned as the boundary point to the mesh point of the first layer.

Preferably, the inner layer mesh point, which is located inside the object of interest and which is next to the mesh point of the first layer is assigned as the boundary point to the mesh point of the first layer. The step of determination for mesh points of the one or more inner layer(s) whether the mesh point is located outside or inside the boundary of the object can be performed anywhere in the process, but before step f).

In another embodiment of the method of the present invention, which can be combined with the aforementioned embodiments, so-called displacement values are assigned to mesh points. Said displacement values are a measure for the attractive force of a boundary point on the mesh points, to which they are assigned to. The displacement values are part of a deformation logic that controls the adaptation of the MLDM to the boundary of the object of interest. In other words, the displacement value represents the influence of a boundary point on the mesh point. If a

corresponding boundary point was found for a mesh point of the first layer or a default boundary point was assigned to a mesh point of the first layer, as described above, a displacement value indicating that the respective mesh point should deform towards the boundary point is assigned. The displacement value (hereinafter abbreviated as "v") is a value which is assigned to a mesh point. In particular, the displacement value can define whether a force can act on a mesh point or not or can define the size of the force (e.g. by multiplying the force with a factor which

corresponds to the displacement value or it can directly represent the size of the force).

Displacement values can be set as default values or be variably assigned before step d) of the process is performed.

In a special embodiment of the method of the invention a weight v in the range of 0 to 1 is assigned to mesh points. A weight of v = 1 means that the mesh point should deform towards a corresponding boundary point and a weight of v = 0 indicates that the mesh point is not influenced by any boundary point. If a boundary point has a low probability of representing the real boundary of the object of interest, the

corresponding value v can be set to 0. If a boundary point has a high probability of representing the real boundary of the object of interest, the according v can be set to 1 or near to 1. The probability of the boundary point representing the boundary can be taken e.g. from the fitness quality of a current profile at the boundary with a trained profile as described in [T. F. Cootes, C. J. Taylor, D. H. Cooper, and J.

Graham. Active shape models - their training and application. CVIU, 61 (1 ):38-59, 1995]. The value v = 0 can also be assigned to a mesh point if no corresponding boundary point (i.e. found boundary point or default boundary point) exists.

Displacement values v (e.g. in the range of 0 to 1 ) can be assigned to all mesh points of the first layer and the inner layer(s). Additional displacement values are suitable for computing the new shape of the model according to the optimization procedure which is explained below.

In an embodiment of the method of the present invention, initial values for v and w are assigned before the adaptation process is started for the first time.

The initial value for w may be set to 1 divided by the number of neighbours.

In another embodiment, stiffness values can be assigned to depth links and to layer links depending on the result of a variability determination for mesh points which determines the variability of the position of the mesh points in a plurality of different known objects of the type of objects which the object of interest belongs to. In this connection, the stiffness value can be determined based on the local variability of the object of interest extracted from a statistical analysis of training surfaces. For example, the standard deviation of the local curvature of the object of interest over a training data set can be encoded in w.

In a specific embodiment, an initial displacement value of v = 1 is assigned to all mesh points of the first layer having a corresponding boundary point, and an initial displacement value of v = 0 is assigned to all mesh points of the inner layer(s), and the stiffness value is chosen according to the method explained above. Then, it is determined whether the inner layer mesh points are located inside or outside of the object of interest in order to make a decision about the assignment of displacement values and stiffness values in the further adaptation process. Following the chain of consecutive depth links for each mesh point of the outer layer, the states of the neighboring points of the inner layers are successively checked. The check is only made along the depth links from mesh point to mesh point. Following principal states can be assigned to the mesh points of the MLDM during the successive check:

Case 1 :

A boundary point was assigned to the mesh point of the first (outer) layer, either a found boundary point or a default boundary point. The boundary point is located in normal direction, starting from the mesh point of the first layer. The check reveals that all mesh points of inner layers, which are linked by depth links to said mesh point of the outer layer, are inside of the object of interest. The displacement value v of the mesh point of the outer layer is set to a value indicating that it should deform towards the boundary point. The value v for the mesh points of the inner layer is set to 0.

Case 2:

A boundary point was assigned to the mesh point of the first (outer) layer, either a found boundary point or a default boundary point. The boundary point is located in inverse normal direction, starting from the mesh point of the first layer. The check reveals that all mesh points of inner layers, which are linked by depth links to said mesh point of the outer layer, are outside of the object of interest. The displacement value v of the mesh point of the outer layer is set to a value indicating that the respective mesh point of the outer layer should deform towards the boundary point. The value v for the mesh points of the inner layer is set to 0.

Case 3:

A boundary point was assigned to the mesh point of the first (outer) layer, either a found boundary point or a default boundary point. The boundary point is located in inverse normal direction, starting from the mesh point of the first layer. The check reveals that a mesh point of the inner layer is outside of the object of interest and the corresponding mesh point of the next inner layer, to which it is connected by a depth link, is also outside of the object of interest. The displacement value v is set to v = 0 for all upper neighbors, i.e. neighbors in direction to the outer layer, of the mesh point, including the mesh point of the outer layer. In comparison to case 2 in this case the check has not been finished for all points. In comparison to case 2 this case reflects an interim state. Case 4:

A boundary point was assigned to the mesh point of the first (outer) layer, either a found boundary point or a default boundary point. The check reveals that a mesh point of an inner layer is inside of the object of interest and an upper neighbour is outside of the object of interest. Said mesh point of the inner layer becomes the new boundary point for the mesh point of the outer layer to whom it is linked by depth links. The displacement value v of the mesh point of the outer layer is set to a value indicating that the respective mesh point of the outer layer should deform towards said mesh point of the inner layer. The value v for the mesh points of the inner layer is set to 0. If, however, one of the lower neighbors, i.e. neighbors of further inner layers connected by depth links, of said mesh point is outside of the object of interest, case 3 applies.

Case 5:

No boundary point is assigned to the mesh point of the first (outer) layer. The displacement value v for all mesh points is set to v = 0.

A yes-state or a no-state can be assigned to the inner layer mesh points in order to express whether a point is inside or outside of the object of interest. Instead of using yes/no states for the inside/outside decision of internal points, a probability between 0 and 1 of being inside/outside can be assigned. Then, the values for v increase or decrease for cases 1 to 4 according to the probability of being inside or outside of the object of interest. However, in order to assign a unique state to every point, a probable threshold is preferably set to determine whether a point is inside or outside.

After setting the values for v and w in the above-described cases, the model is deformed towards the boundary points and the new shape of the model is computed. Thereby, a best possible form preservation of the MLDM is obtained. The form preservation can be realized by optimizing the point coordinates of the MLDM such that the lengths of the depth links and the layer links stay similar in relation to each other compared to a reference MLDM. The reference MLDM can be created as already described above. A preferred optimization procedure is performed as follows: Given the points / of the MLDM, the points Im of the reference MLDM, the number of layers L, the number of points of each la er P and the boundary points a the following equation is minimized:

VY (Formula 2)

keL ieP d edeplh _ki with

^E kis = ^w kis ((hi - - ( ^lm ki - )) ² (Formula 3)

^F kid = ^w kid ((hi ~ ^l d ) ~ ( ^lm H - ^lm d )Y (Formula 4)

^K ki - ^v ki (hi - ^a ki (Formula 5)

de th _k i and surf _k , denote the sets of neighbor point indices for the point k; in depth and surface direction, respectively, w denotes the stiffness value, and v denotes the displacement value.

Since Formula 2 is a least squares optimization, the length of links may be slightly changed even if the weights w are high. For example, if w^s, Wkid and v are set to 0.5, all three terms of Formula 3-5 are equally weighted which leads to a change of link lengths.

In another aspect the present invention relates to an image processing system comprising image data processing means for segmentation of an object of interest, wherein the data processing means are adapted to perform the method as described before using the multi layer deformable mesh model. The image processing system preferably comprises following data processing means to adapt the outer layer of an MLDM of the invention to the boundary of an object of interest:

a) evaluating means for identifying boundary points of the object of interest, by evaluating the three dimensional image data set,

b) processing means for assigning the boundary points to a mesh point of the first layer,

c) processing means for assigning stiffness values to the depth links and to the layer links, wherein each stiffness value indicates to which extent the mesh points, which are connected by the respective depth link or by the respective layer link, are allowed to move relatively to each other,

d) processing means for displacing mesh points of the first layer towards their corresponding boundary points, wherein grades of displacement of the mesh points of the first layer depend on the stiffness of the depth links and layer links according to the assigned stiffness values,

Further means that could be included in the image processing system are:

^■ determination means for determining for mesh points of the one or more inner layer(s) whether the mesh point is located outside or inside the boundary of the object.

^■ processing means for assigning displacement values to mesh points.

^■ computing means for creating an initial multi layer deformable mesh model, including means for creating a surface mean model and means for creating further inner layers.

A further aspect of the invention is a computer or computer system, comprising the image processing system as described above.

Still another aspect of the invention is a medical examination apparatus comprising a device adapted to acquire a three-dimensional image data set of an organ of a body, the organ being an object of interest, and further comprising the image processing system as described above. The medical examination apparatus may be a computer tomography medical examination apparatus, a magnetic resonance imaging medical examination apparatus or an X-ray medical examination apparatus or any other 3-D medical imaging apparatus.

The invention further relates to a computer program, adapted to perform the method steps as described above if the program is executed by a computer or computer system, and to a digital data storage, wherein the computer program of the preceding claim is stored in the data storage

The invention especially targets at the improvement of medical image segmentation systems for Computer Aided Diagnosis (CAD), therapy planning or visualization. In particular, the invention can be used to improve three-dimensional segmentation of organs in volumetric images. Such an image may be a Computed Tomography (CT) scan of the liver, where the vessels are contrasted, or a CT scan of the kidneys with inhomogeneous distributed contrast agent. Another possible application is the segmentation of an inhomogeneous liver with multiple hyper-dense or hypo-dense lesions, which can be detected using the invention.

Hereinafter, aspects of the present invention are illustrated by means of specific examples which are not to be construed as limitation of the general ideas and principles of the invention as described before and in the appended claims.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 shows a detail of the structure of a MLDM of the present invention. Figure 2 shows a detail of the structure of a MLDM of the present invention in a region of high curvature of the first layer.

Figure 3 shows a MLDM of the invention which is located inside of an object of interest, during the adaptation process to the boundary.

Figure 4 shows a MLDM of the invention which is partly located inside of an

object of interest and partly outside, during the adaptation process to the boundary.

Figure 5 shows the adaptation of an MLDM of the invention into a cavity of the boundary of an object of interest.

Figure 6 shows the initial location of an MLDM with respect to the boundary of an object of interest before the first iteration step has been started.

Figure 7 shows the location of an MLDM with respect to the boundary of an

object of interest after the first iteration step.

Figure 8 shows the final location of an MLDM with respect to the boundary of an object of interest after a further and last iteration step.

Figure 9 illustrates a medical viewing system coupled to a medical examination apparatus.

Figure 1 shows a detail of the structure of a multi layer deformable mesh model of the present invention. The first (outer layer) is referred to as L1. Two inner layers are shown and referred to as L2 and L3. The mesh points form a structure of connected triangular polygons in each layer. A triangular polygon of interconnected mesh points in every layer and a similar structure of the layers is shown. The mesh points of the first layer L1 are referred to as M 1-1 , M 1-2 and M1-3, the mesh points of the neighboring inner layer L2 are referred to as M2-1 , M2-2 and M3-3, and the mesh points of the second inner layer L3 are referred to as M3-1 , M3-2 and M3-3. M1- 1/M2-1 and M2-1/M3-1 is a pair of corresponding mesh points which are connected via a depth link, as well as M1-2/M2-2, M2-2/M3-2, M1-3/M2-3 and M2-3/M3-3. The depth links between the first layer L1 and the neighboring inner layer L2 are referred to as D12-1 , D12-2 and D12-3. The depth links between the inner layer L2 and the inner layer L3 are referred to as D23-1 , D23-2 and D23-3. Links between mesh points within the same layer (layer links) are referred to as 11-1 , 11-2, and 11-3 with respect to the first layer L1 , 12-1 , I2-2, and I2-3 with respect to the neighboring inner layer L2, and 13-1 , I3-2, and I3-3 with respect to the inner layer L3.

Figure 2 shows another detail of the structure of a multi layer deformable mesh model of the present invention. The reference symbols were already explained with respect to Figure 1. Figure 2 exemplary shows a region of high curvature of the first layer. The dashed depth links D34-1 and D34-2 show how intersections of depth links occur, if the position of mesh points in inner layers is not optimized. The dashed depth links D23-1 , D23-2, D34-1 and D34-2 are depth links before optimization. The Figure shows the positions of points M3-1 ' and M3-2' of layer L3, the positions of points M4-1' and M4-2' of layer L4, the positions of depth links D23-1', D23-2', D34-1 ' and D34-2', and the position of layer links I 3-1 ' and I 4-1 ' after optimization (positions of points M3-1 , M3-2, M4-1 , M4-2 and layer links I 3-1 and I 4-1 before optimization are not shown). It is shown how intersections of depth links can be avoided by optimizing points of the layer L3 and L4 such that their relative distances to

neighboring points are preserved.

Figure 3 and Figure 4 show different situations how the multi layer deformable mesh model (MLDM) of the invention is located relatively to the boundary of an object of interest and how the MLDM can be adapted to the boundary of an object of interest, which is designated as BOI. The mesh points of the MLDM are shown as circles in Figures 3 and 4 and the links between the mesh points are shown as continuous lines. The first (outer) layer of the MLDM is referred to as L1. Inner layers are referred to as L2 and L3. Boundary points are referred to as A and D,

respectively, wherein A is a found boundary point and D is a default boundary point. Displacement values v are assigned to the mesh points.

The adaptation process of the MLDM to the boundary of the object of interest is performed as follows:

The starting value for all v on the outer layer is 1 and for all v on inner layers 0. The starting value for w may be set to 1 divided by the number of neighbors or can be trained based on the local variability of the organ extracted from a statistical analysis of reference surfaces. In regions of high variability the initial value of w is low or near 0 and in regions of low variability the initial value of w is high or near 1.

The adaptation logic is started after boundary points for the mesh points of the outer layer L1 have been found. The following states for the mesh points of the outer layer are distinguished:

• A mesh point of the outer layer has found a boundary point (marked as A in Figure 3).

• A mesh point of the outer layer has not found a boundary point, but the point is inside of the object of interest: here, a default boundary point (marked as D in Figure 3) is assigned to the point. This default boundary point is placed in normal direction.

• A mesh point of the outer layer has not found a boundary point and the point is outside of the object of interest (point is marked with a cross in Figure 4): a default boundary point (marked as D in Figure 4) in inverse normal direction is assigned to the point. The states of the internal points now control the assignment of the final values for v and w. Following the depth link for each outer layer point, the state of the neighboring points is successively checked. Following cases are distinguished:

1. All neighboring mesh points of inner layers are inside of the object of interest: the weight v of the boundary point remains 1 (see Figure 3 middle). The same applies if a default boundary point for the mesh point of the outer layer was assigned (see Figure 3 right).

2. The outer layer point is outside of the object of interest and all neighboring points are outside of the object of interest: v for the outer layer point remains 1 and a default boundary point in the interior of the model is assigned (see Figure 4 right).

3. An inner point is outside of the object of interest, but the corresponding outer layer point has got a boundary point (either from search or a default point): v is set to 0 for all upper neighbors of the inner point.

4. An inner point is inside of the object of interest and an upper neighbor is

outside of the object of interest: the inner point becomes the new boundary point for the corresponding mesh point of the outer layer, v is set to 1 for this outer layer point (see Figure 4 left, middle). If one of the lower neighbors is outside of the object of interest, case 3 applies.

Instead of using yes/no states for the inside/outside decision of internal points, a probability between 0 and 1 of being inside/outside can be assigned. The values for v then increase or decrease for cases 1-4 according to the probability.

In the present example, after the assignment of states every surface point in the MLDM has a boundary point (a boundary point from search, e.g. along the normal, or a default boundary point). The values of v control the weight of the boundary points, i.e. if a boundary point has a low probability of representing the real object of interest boundary, the according v can be set to 0.

Figure 3 shows the MLDM where all visible mesh points are lying inside the object of interest. One outer layer point has found a boundary point (A). For the right neighbor of this point, a default boundary point (D) was assigned, since no boundary point has been found through search. Figure 4 shows the MLDM partly inside the object of interest. All outlying mesh points are marked with a cross. In the left and middle case, the outer layer points are outside of the object of interest. For these cases, the first internal point which is inside of the object of interest becomes the new boundary point for the according outer layer point.

The flexibility is now determined at every point and every link of the model. In order to compute the new shape of the model, the model is deformed towards the boundary points. The form preservation can be realized by optimizing the point coordinates of the MLDM such that the link lengths stay similar in relation to each other compared to a reference MLDM. A preferred mathematical optimization procedure was already explained above in the general description with respect to formulas 2 - 5.

The procedure illustrated with Figure 3 and 4 is performed in an iterative manner, i.e. the process of finding boundary points, assigning displacement values to mesh points and stiffness values to links between mesh points, as well as the deformation is repeated until the model adequately fits to the data.

Figure 6, Figure 7 and Figure 8 show the single iterations of an exemplary adaptation process. The graphic symbols and reference symbols are similar to those in Figure 3 and 4 and were already explained above. The boundary of the object of interest (BOI) to be detected is shown as the dashed line. Figure 6 shows the MLDM after placement in the data set. The states of the points are determined, boundary points A are searched and default boundary points D are set. In Figure 7, the MLDM has deformed towards the found points A and D. Now, new states and boundary points are determined. In this iteration, the rightmost point of layer L1 is already located at the correct boundary position, i.e. at the position of the boundary point A, and the position of the mesh point and the boundary point are equal. Figure 8 shows the final iteration, where all mesh points of the outer layer of the MLDM are matched with the contour of the object in the image. Since all mesh points of the outer layer are matched with their corresponding boundary points, the adaptation process is finished. In order to model an indentation of the boundary of the model of interest, i.e. to allow the model to adequately fit into cavities, the flexibility of the model can be

dynamically adapted on basis of the local changes of the w-values. Figure 5 shows the detection of such a cavity where all internal points for a mesh point M1-1 of the outer layer L1 are outside of the object of interest (marked as crossed nodes). The reference symbols as to mesh points and links were already explained with respect to Figure 1. In this case, the stiffness values to the depth links w _d are increased by a value of Φ in order to increase the stiffness of that model part. This preserves the local stability of the model and prevents mutual intersection of the layers. The weights w _s for the neighboring links to mesh points within a same layer (layer links) are set to a softer value by value delta Δ. This allows the model to adapt to the cavity with less affection on the neighboring points within a same layer that already have found the boundary of the object of interest.

Figure 9 shows the basic components of an embodiment of an image viewing system in accordance to the present invention, incorporated in a medical examination apparatus. The medical examination apparatus 100 may include a bed 101 on which the patient lies or another element for localizing the patient relative to the imaging apparatus. The medical imaging apparatus 100 may be a CT scanner or other medical imaging apparatus such as x-rays or ultrasound apparatus. The image data produced by the apparatus 100 is fed to data processing means 70, such as a general-purpose computer, that comprises computation means and user control means appropriate to form the interactive adaptation means of the invention. The data processing means 70 is typically associated with a visualization device, such as a monitor 60, and an input device 72, such as a keyboard, or a mouse 71 , pointing device, etc. operative by the user so that he can interact with the system. The data processing device 70 is programmed to implement the processing means for processing medical image data according to the invention. In particular, the data processing device 70 has computing means and memory means necessary to perform the operations described in relation to Figure 3, 4, 5, 6, 7 and 8. A computer program product having pre-programmed instructions to carry out these operations can also be implemented.