Field of the invention

The present invention is directed at a method of processing medical images by an analysis system for enabling radiomics signature analysis, for enabling classification of a clinical picture in a human or animal body, such as a neoplasm, a lesion or a disorder.

Background

Radiomics signature analysis of medical images, for enabling classification of a clinical picture in a human or animal body, such as to classify a neoplasm, a lesion or a disorder, e.g. in terms of its expected behavior under treatment, is a relatively new field of image analysis providing promising results in terms of increasing effectiveness of treatment of neoplasms by therapy or to assess whether a certain type of treatment would be preferred, e.g. surgery, or a specific chemical therapy or radiation treatment. Radiomics signature analysis is based on identification of specific patterns of image features or combinations of certain image features from the imaging data of an image. Because most of these features are only discoverable by in depth analysis of the imaging data, performing radiomics signature analysis to confirm whether or not a radiomics signature is occurring in at least a part of such image data, relies on the computational power presently available to us.

Since the development of radiomics, many radiomics signatures that are indicative of some type of neoplasm or neoplasm behavior, have been identified. A problem typically encountered with the identification of radiomics signatures, is that there are many different brands and types of imaging systems, and each individual imaging system has its individual settings. As a result, it cannot be relied upon that a radiomics signature identified and associated with images from one imaging system also applies to images from an arbitrary different imaging system. The individual differences between imaging systems require that radiomics signatures identified with certain known imaging systems need to be validated first with new imaging systems before they may be applied, and in some cases if they cannot be validated, new radiomics signatures that can be associated with the new or unknown imaging system need to be established.

The above prevents radiomics signatures to be universally applicable to many imaging systems. Validating radiomics signatures for a new imaging system requires a sufficient amount of imaging data from that system, which may not be available. Furthermore the validation process likewise provides a heavy computational burden. If the radiomics signature cannot be validated, e.g. because it fails to apply to the imaging data of the particular imaging system, the finding of new radiomics signatures that can be associated with that particular imaging system likewise provides a heavy computational burden to evaluate the data.

Summary of the invention

It is an object of the present invention to provide a method of processing medical images for enabling radiomics signature analysis, which overcomes the abovementioned drawbacks and enables imaging data from different imaging systems to become comparable, such as to allow evaluation of existing radiomics signatures for a new imaging system or identification of new radiomics signatures.

To this end, there is provided herewith a method of processing medical images by an analysis system for enabling radiomics signature analysis, for enabling classification of a neoplasm in a human or animal body, the method comprising: obtaining, by a controller of the analysis system, at least one first imaging data set of at least one first image obtained using a first imaging system; obtaining, by the controller, at least one second imaging data set of at least one original second image obtained using a second imaging system; providing, by the controller, the at least one first imaging data set as input to a first data processing model, wherein the first data processing model is configured for analyzing the first imaging data set such as to provide, at an output of the first data processing model, a style representation data associated with the first imaging system, wherein the style representation data is indicative of an imaging style of the first imaging system independent of an image content of the first image; providing, by the controller, the style representation data and the at least one second imaging data set to a second data processing model, wherein the second data processing model is configured for, based on the style representation data, applying the imaging style to the at least one second imaging data set to obtain at least one modified second image, the at least one modified second image being a representation of the at least one original second image as if it would have been obtained with the first imaging system.

The present invention applies a first data processing model to retrieve only the style features from the first image, i.e. the features that are related to texture and perceptual characteristic of the image separated from the semantic content. This style representation data is then, together with a second imaging data set of at least one image from a second imaging system, used as input to a second data processing model. This second data processing model applies the imaging style of the first image to the at least one second image to provide at least one modified second image, in such a way that the modified second image is a representation of the original second image as if it would have been obtained with the first imaging system. The insight underlying the present invention is that if a single sample, e.g. the exact same volumetric area of a single human being diagnosed with a neoplasm located in that volumetric area, would theoretically be imaged using both the first and the second imaging system, the semantic content of these two images has to be identical. Hence, differences between the two images that are caused by system settings or hardware elements of the two systems are primarily part of the style features of these images. Therefore, retrieving the image style from the first image using a known style retrieval method and applying it to the second image using a style transfer method, allows to provide a modified second image in the style of the first imaging system, i.e. as if it would have been obtained using the first imaging system.

The above can be used in multiple ways. For example, this allows to find new radiomics signatures based on known images that are style transferred into the imaging style of a new or unknown imaging system such as to enable associating these new radiomics signatures to images of the new imaging system. As a further example, this also allows to convert or transfer images of a new imaging system into the imaging style of a known imaging system for which radiomics signatures are known, such as to enable applying such existing radiomics signatures to the images of the new imaging system. Therefore, in accordance with a first embodiment, the analysis system is configured for applying an algorithm on imaging data obtained from at least one imaging system from a pool of known imaging systems, such as to identify the occurrence of a radiomics signature in at least a part of the imaging data, and wherein the second imaging system is formed by an imaging system from the pool of known imaging systems, and the first imaging system is not comprised by the pool of known imaging systems; wherein the at least one second imaging data set is associated with a plurality of original second images; and wherein based on a comparison between the plurality of original second images and a plurality of modified second images associated therewith, a validity of the radiomics signature is verified for images obtained with the first imaging system.

In these embodiments, the new imaging system will be the first imaging system (the system which’ s image style is retrieved and transferred), and the known imaging system (for which associated radiomics signatures are known) is the second imaging system (the images of which will be converted into the new style). Radiomics data of these second images, i.e. data indicative of whether or not certain associated radiomics signatures are occurring or identifiable in (at least a part of) these images, is already available. Hence, by converting the second images into the style of the new imaging system, the radiomics signatures can be validated with the imaging style of the second imaging system. Thus, it can be derived from the modified second images whether the radiomics signatures still apply in this new imaging style, and therefore whether these radiomics signatures may be associated with the new imaging system. If validation of the radiomics signatures fails on the basis of these modified second images, it is possible to use the style converted modified second images and the radiomics data that provides the classification of neoplasms visible in these images as training data input to e.g. a third data processing method that enables to find a new radiomics signature associable with the new imaging system.

Therefore, in accordance with an embodiment, the method further comprising the steps of providing the plurality of modified second images to an analysis system for calculating a plurality of image features from the images, calculating based on the plurality of original second images signature values for the radiomics signature, training a third data processing model for establishing a new radiomics signature from the plurality of modified second images, the training comprising: providing, for each second image of the plurality of modified second images, the plurality of image features to the third data processing model; providing, for each second image of the plurality of original second images, the signature values to the third data processing model; and training the third data processing model based on the signature values received in step b. and the plurality of image features received in step a., for providing the new radiomics signature associated with images obtained from the first imaging system.

The third data processing model may for example be a machine learning data processing model wherein on the basis of training data including the modified second images and the radiomics data thereof, e.g. the radiomics data including at least one of the signature values of the radiomics signature or the image features, the machine learning data processing model is trained to recognize a new radiomics signature in the modified second images that are now in the style of the new imaging system (used in this embodiment as first imaging system in the method). This therefore provides a radiomics signature that can be associated with images in the style of the new imaging system, to recognize the occurrence of the new radiomics signature in arbitrary images of neoplasms taken with the new imaging system. The advantages are of great importance, because this enables to relatively easily obtain new radiomics signatures for images taken with a new imaging system, e.g. for which no radiomics signatures are yet available. This in turn enables to develop an expertise system that enables to process images from arbitrary imaging systems in order to classify neoplasms based on radiomics. In accordance with some embodiments, the method thus further comprises a step of applying the new radiomics signature to images obtained from the first imaging system, such as to perform classification of neoplasms in said images.

As will be appreciated, the data from the images considered is medical data and the method can be applied to perform radiomics on this data in order to assess the treatability of e.g. tumours or the expected response of a tumour to a certain treatment. In this field, reliability of the methods applied is of paramount importance. In some embodiments of the above, the method therefore comprises obtaining, by the controller, a further second imaging data set of one or more further second images obtained using the second imaging system; providing, by the controller, the further second imaging data set as input to the first data processing model such as to provide a further style representation data associated with the second imaging system; providing, by the controller, the style representation data of the first imaging system and the further second imaging data set to the second data processing model for applying the imaging style of the first imaging system to the one or more further second images to obtain a further modified second images thereof; providing, by the controller, the further style representation data of the second imaging system and the further modified second images to the second machine learning data processing system for applying the imaging style of the second imaging system to the further modified second images such as to obtain a forward-backward converted second images thereof; calculate the signature values of the radiomics signature based on the forward-backward converted second images, such as to validate the method.

In the above embodiments, imaging data obtained with the second imaging system - which, in the above class of embodiments, is the known imaging system for which radiomics signatures associated therewith are available - is offered as input to the first data processing model in order to retrieve the imaging style thereof and make this available as further style representation data in addition to the (earlier mentioned) style representation data of the first imaging system. These images are then first converted, using the style representation data of the first imaging system, into modified second images (as above) in the style of the first imaging system. Next, these modified second images are again converted using the further style representation data back into the style of the second imaging system. This yields the forward-backward converted images. The radiomics signature values of the original second images and the forward- backward converted images may now be calculated and compared to determine whether the conversion method delivers reliable results. This is because if any disturbing artefacts would have been added by the conversion process, the forward- backward conversion would cause a discrepancy between the radiomics signature values. Therefore, if such a discrepancy is found, the conversion may be considered insufficiently reliable. If the discrepancy is insignificant or even absent, the application of the validated radiomics signatures or the newly found radiomics signatures discussed above can be considered sufficiently reliable. The reliability test, in the above embodiments, is optional.

As referred to above, the present invention may also be applied in order to convert or transfer images of a new imaging system into the imaging style of a known imaging system for which radiomics signatures are known, such as to enable applying such existing radiomics signatures to the images of the new imaging system. To this end, in accordance with yet a further class of embodiments, the first imaging system is at least one imaging system from a plurality of known imaging systems, wherein the at least one imaging system is associated with at least one radiomics signature for enabling a classification of a neoplasm based on imaging data from said at least one imaging system; wherein the second imaging system is a further imaging system different from the plurality of known imaging systems; and wherein, after the step of applying the imaging style of the first imaging system to the at least one second imaging data set to obtain the at least one modified second image, the at least one radiomics signature is further associated with the at least one second imaging system for enabling said classification from images obtained from the at least one second imaging system.

In the above embodiments, a known imaging system with which radiomics signatures are associated is used as the first imaging system, whereas the new or unknown imaging system is used as second imaging system. Images of the new imaging system are thus converted into the style of the known imaging system. By converting the images from the new imaging system into the style of the images of the known imaging systems, it becomes possible to apply the existing radiomics signatures that are associated with the (known) first imaging system to the (new) second imaging system. Thus, in this case it is not necessary to perform a new search for radiomics signature patterns, because the existing radiomics signatures can be applied to the modified second images. Therefore, in some embodiments, the method further comprises a step of applying the at least one radiomics signature to images obtained from the second imaging system, such as to perform classification of neoplasms in said images.

Similar to the above, also in these embodiments the method may be validated by a forward-backward conversion of images. To this end, the method in some embodiments includes providing, by the controller, the at least one second imaging data set as input to the first data processing model such as to provide, at the output of the first data processing model, further style representation data associated with the second imaging system, wherein the further style representation data is indicative of an imaging style of the second imaging system; providing, by the controller, the further style representation data and the at least one first imaging data set to the second data processing model for, based on the further style representation data, applying the imaging style of the second imaging system to the at least one first imaging data set to obtain at least one modified first image, the at least one modified first image being a representation of the at least one first image as if it would have been obtained with the second imaging system; providing, by the controller, the style representation data of the first imaging system and the at least one modified first image to the second data processing model for obtaining at least one forward-backward converted first image; and calculate the signature values of the radiomics signature associated with the first imaging system based on the forward-backward converted second images, such as to validate the method.

The forward-backward conversion is always applied on the imaging data of the known imaging system, because radiomics signatures that enable to verify the correctness (validate) of the conversion are only available for the known imaging system. Therefore, because here the image data of the known imaging system is the first imaging data set, it is this data that is converted into the style of the new imaging system and thereafter converted back into the style of the original known imaging system. Thereafter, as above, radiomics signature values of the original second images and the forward-backward converted images may now be calculated and compared to determine whether the conversion method delivers reliable results.

As referred to above, various methods for transferring a style of one image to the content of another image are known in literature, and may be applied in the present invention. Earlier methods rely for example on non-parametric algorithms that can synthesize textures by resampling pixels of a given source texture. These methods are primarily, but not exclusively, directed at analysis of low-level image features, i.e. style features directly obtainable from analysis of the images. Later style transfer methods are based on the application of a system of convolutional neural networks, e.g. containing multiple complementary neural networks or machine learning data processing methods. For example, to transfer the style representation data to the new image, use may be made of a VGG network, such as a VGG- 16 or VGG- 19 network (wherein the acronym VGG refers to the designers of the network: Visual Geometry Group of Oxford University). This may for example be combined with a deep residual convolutional neural network that may be parametrized by weights. Given the number of style transfer methods available, many of which are powerful enough to be applicable in embodiments of the present invention, the invention is not limited to a particular style transfer method. However, some details of style transfer methods are discussed herein below to the benefit of the skilled person.

In accordance with some embodiments, the at least one of the first data processing model, the second data processing model or, where dependent on claim 3, the third data processing model, includes at least one element of a group comprising: at least one machine learning data processing model, such as a convolutional neural network; a system of two or more machine learning data processing models, such as a convolutional neural network in addition to a further machine learning data processing model; or a combination of at least one image transformation network including a deep residual convolutional neural network, and at least one loss network pre-trained for performing perceptual optimization; or one or more functional data processing steps or algorithms, such as: an image segmentation step, optimization steps such as based on Markov Random Field analysis, or image reconstruction steps; or image decomposition based on Laplacian stack analysis, image blending based on energy computation of Laplacian levels, and level aggregation.

In some embodiments, at least one of the first data processing model and the second data processing model both include a convolutional neural network in addition to a further machine learning data processing model, wherein the convolutional neural network of the first and second machine learning data processing system is the same convolutional neural network, and wherein the second machine learning data processing system is configured for applying both the style representation data and the second imaging data set independently to the convolutional neural network. For example, in one embodiment, the convolutional neural network is configured for performing image pattern recognition, such as object detection or image reconstruction. And even in one embodiment, the convolutional neural network comprises a plurality of layers, wherein based on the at least one first imaging data set each layer provides a first image filter response data set for the respective layer; and wherein the further machine learning data processing model of the first machine learning data processing system is configured for receiving the first image filter response data sets of the plurality of layers at a first input and calculating a correlation between the plurality of layers.

In accordance with a second aspect, there is provided a method of testing a style transfer mechanism for a processing of medical images by an analysis system for enabling radiomics signature analysis, the method comprising: obtaining, by a controller of the analysis system, at least one first imaging data set of at least one first image obtained using a first imaging system; providing, by the controller, the at least one first imaging data set as input to a first data processing model, wherein the first data processing model is configured for analyzing the first imaging data set such as to provide, at an output of the first data processing model, a style representation data associated with the first imaging system, wherein the style representation data is indicative of an imaging style of the first imaging system independent of an image content of the first image; providing, by the controller, the style representation data and the at least one second imaging data set to a second data processing model, wherein the second data processing model is configured for, based on the style representation data, applying the imaging style to at least one second imaging data set to obtain at least one modified second image; wherein the at least one second imaging data set is identical to the at least one first imaging data set; and wherein the method further includes a step of comparing the at least one modified second image with the at least one first image, for determining a difference between the at least one modified second image and the at least one first image, for performing an evaluation of the style transfer mechanism. In the above manner, the quality of a style transfer mechanism may be tested and compared to the quality of alternative style transfer methods. Brief description of the drawings

The invention will further be elucidated by description of some specific embodiments thereof, making reference to the attached drawings. The detailed description provides examples of possible implementations of the invention, but is not to be regarded as describing the only embodiments falling under the scope. The scope of the invention is defined in the claims, and the description is to be regarded as illustrative without being restrictive on the invention. In the drawings:

Figure 1 schematically illustrates a method in accordance with an embodiment of the invention;

Figure 2 schematically illustrates a method in accordance with an embodiment of the invention;

Figure 3 schematically illustrates an alternative style decomposition step for use in a method in accordance with an embodiment of the invention;

Figure 4 schematically illustrates a method in accordance with an embodiment of the invention;

Figure 5 schematically illustrates validation of a method in accordance with an embodiment of the invention.

Detailed description

In the past years, various methods have been developed directed at identification of the style of an image and application of that style to a different image. In particular, these style transfer methods for example enable to apply the style of an arbitrary image, e.g. a Van Gogh painting, to a different image, e.g. a portrait photograph. In particular, these methods apply advanced data processing models in order to separate the content information of an image from the style information. The advanced data processing methods include for example mathematical algorithms, image processing techniques or neural networks (this list is not exclusive). Various style transfer methods have been described in literature, are presently still in development or will be developed, each one of which may be applied in the present invention such as to enable applying the style of the first imaging system to images of the second imaging system. Non-limiting examples of such style transfer methods for example are described in the following documents: “Learning from Multi-domain Artistic Images for Arbitrary Style transfer”, Zheng Xu et al., 8 ^th ACM/EG Expressive Symposium, 2019; “Style Transfer via Image Content Analysis”, in particular section IV “Image Style Transfer”, Wei Zhang et al., IEEE Transactions on Multimedia, vol. 15, no. 7, pages 1594-1601, November 2013; “Style Transfer on Headshot Portraits”, in particular section 3 “Multiscale Local Transfer”, YiChang Shih et al., ACM Transactions on Graphics, vol. 33, no. 4, Proc. SIGGRAPH, 2014; “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”, in particular section 3 “Method”, Justin Johnson et al., European Conference on Computer Vision (ECCV), 27-3-2016; “A Neural Algorithm of Artistic Style”, in particular section “Methods” on page 9, Leon Gatys et al., arXiv preprint arXiv:1508.06576v2, 2-9-2015.

Whichever particular style transfer method is applied, the method of the present invention at least includes two main steps: a first step to obtain the style of an image in order to provide style data, and a second step to apply the style to the other image that needs to be converted. In some methods, non-parametric data clustering techniques may be applied to separate the style and content in both a first and second image, after which the separate elements of content from the second image and style from the first image may be processed via an optimization algorithm to obtain a style transferred modified second image. In some other methods, content, style and a residual are separated in both images using Laplacian stacks, after which the style is applied by computing local energy maps of the Laplacian stacks and aggregate these in a style transferred modified second image. In yet other methods, style data is obtained from a first image using a convolutional neural network (CNN), such as a VGG-19 network. The style data is applied by submitting the content information from the second image in one layer and the style data in one or more other layers of another or the same CNN. A loss function may be minimized against a white noise image, also applied as input to the CNN, until the style transferred modified second image is obtained. Various alternative implementations are available for such type of style transfer.

Figure 1 schematically illustrates a method in accordance with a first embodiment. The method in accordance with the present invention starts with the obtaining, in step 1, a first imaging data from a first imaging system (e.g. imaging system 16 in figure 2). For example, a first imaging system may be a new CT scanner for which no radiomic signature data has yet been established. The imaging data 11 for example relates to an image including a (section of a) neoplasm, for example a tumour. For the image 11 obtained, no radiomic signatures have yet been validated, such that the image 11 cannot yet be used as input to a radiomics signature analysis process. To enable radiomics signature analysis on images, such as the image 11, from the first imaging system 16, the method in figure 1 will be applied. In step 3, at least one second image from a second imaging system is obtained (e.g. imaging system 17 in figure 2). Ideally, a plurality of second images 12 from second imaging systems 17 that are known and for which radiomic signatures have been associated, are obtained in step 3.

Next, in step 5, the first image 11 of the first imaging system, which has been obtained in step 1, will be used as input to a first data processing model. The first data processing model may for example be in the form of an algorithm or a series of steps, or a trained convolutional neural network (CNN), which enables to obtain style data representative of an imaging style of the first image 11. The imaging style of the first image 11 relates to all imaging characteristics that are not related to the content of the image 11. For example, textures, colors, noise, artifacts, and many other similar image characteristics related to the imaging style, rather than to the content of the image. The first data processing model is able to separate the content information from the style information, and provides style representation data 13 at its output.

The style representation data 13 together with the second images 12 obtained in step 3, are provided as input in step 7 to a second data processing model. The second data processing model may for example be an algorithm or series of steps, or a machine learning data processing model such as a convolutional neural network or a system of neural networks, that enables to apply the style data 13 representative of the imaging style of the first image 11 to the one or more second images 12 obtained from the second imaging system 17 in step 3. Optionally, as an intermediate step somewhere between steps 3 and 7, the content related data may have been obtained from the second images 12 such that for example in step 7 the style representation data of the first image 11 and the content data of the second images 12 -without any style elements of the second images 12- will be provided as input to step 7. It is to be understood that the above optional feature may be dispensed with, and the style transfer method performed in step 7 may be performed directly to the second images 12 obtained from step 3 in a preferred embodiment.

Step 7, at its output, yields one or more modified second images 14. The one or more modified second images 14 relate to the second images 12 obtained in step 3, in the sense that the content of the modified second images 14 corresponds to the content of the second images 12. However, the imaging style of the modified second images 14 will be the imaging style of the first images 11 as provided by the style representation data 13. Various methods exist for applying the style representation data 13 to the second images 12. For example, both may be applied as input to a convolutional neural network, and a loss function may be calculated in combination with gradient descent to provide the modified second images 14.

The one or more modified second images 14 together form a reference data set that can be used for validation of radiomics signatures to the first images 11 of the first imaging system, or as a training data set in order to train a third machine learning data processing model for establishing a new radiomics signature that applies to the images 11 of the first imaging system. Typically a published radiomics signature also comes with a published paper, which describes in detail how the radiomics signature was obtained (which machine learning tool was used, e.g., random forest, which radiomics features were used and how they were calculated, etc). In the main approach, the exact same process maybe followed. Alternatively or additionally, a further convolutional neural network or a random forest network may be trained with this training data set to perform such a task. To do so, for example, the modified second images 14 may first be analyzed statistically to obtain therefrom, for each image, a plurality of statistical image features (for example, but not limited to, statistical features such as described in International patent applications WO 2014/171830 or WO 2016/060557). For training, these may be provided to the further convolutional neural network or random forest network together with their associated radiomics signatures referred to in step 3 for the second images 12 associated with the modified second images 14. All these further processing steps that may be performed for validating or generating a radiomic signature for the first images 11 of the first imaging system, are in figure 1 summarized by the step 9 indicative of further processing of the data. Figure 2 illustrates a style transfer method that is based on a system of convolutional neural networks to perform the style transfer. In figure 2, the first image 11 obtained from first imaging systems 16 is used as input to a convolutional neural network (CNN) 20 trained to perform style retrieval from images. At the output of the CNN 20, style representation data 13 is obtained. The style representation data 13 is used as input to a second CNN 23 for performing the style transfer. In some embodiments, the style transfer CNN 23 may be similar or even identical to the style retrieval CNN 20. However, this is merely optional and by no means a requirement. CNN 23 may be a different data processing model than CNN 20, and may have been trained specifically to perform the style transfer. As further input to the style transfer CNN 23, the CNN 23 receives a plurality of second images 12 obtained from one or more second imaging systems 17. For example, the imaging systems 17 may be a class of imaging systems for which radiomics signatures have been validated. The style transfer CNN 23 applies the style representation data 13 to the second images 12 to yield a set of modified second images 14 as illustrated in figure 2. The radiomics signature data 18 that is associated (as indicated by double arrow 19) with the second images 12, for example may contain data for each of the second images 12 on whether or not a certain radiomic signature occurs in the image or applies to the image. Whether or not a radiomic signature applies depends on a set of conditions that is characteristic for that respective radiomic signature. These conditions relate to imaging features of the image, and for example relate to whether or not a certain imaging feature may be present or whether or not a statistical value indicative of the image feature exceeds a threshold value. The imaging features and their characteristic values may be obtained by statistical analysis of the second images 12. Important for the present method, is that the radiomics signature data 18 associated with each of the second images 12 is available.

At the output of the style transfer CNN 23, the CNN provides a set of modified second images 14. The contents of the modified second images 14 is identical to the content of the second images 12, however the style of the images 14 corresponds to the imaging style of the first image 11. By obtaining the set of modified second images 14, the present method now presents a set of modified second images 14 with their associated radiomics signature data 18. As may be appreciated, for each of the images 12, the radiomics signature data 18 indicates whether or not a certain radiomics signature applies to the respective image.

By converting the respective image into the style of the first image 11, it now becomes possible to validate whether or not the radiomics signature data 18 still applies to the converted modified second images 14. This is done in step 25 wherein it is validated whether or not the radiomics signature data 18 associated with each of the modified second images 14 still applies to these modified second images 14 which are in the style of the first image 11. If the radiomics signature data still applies to the modified second images 14, then the outcome of decision step 28 is that the radiomics signatures associated with the second images 12 can be applied likewise to the first images 11 coming from the first imaging system 16. If, however, in step 28 it is determined that the radiomics signature data 18 that is associated with the second images 12 no longer applies to the modified second images 14, then the radiomics signature data 18 and the modified second images 14 are used as input to a further machine learning data processing model 30. In the further machine learning data processing model 30, the modified second images 14 with the radiomics signature data 18 are used as training data to find a new radiomics signature that does apply to the modified second images 14. This works as follows.

In principal, the second images 12 with which the radiomics data 18 is associated, are already analyzed in a radiomics signature analysis method. Therefore, it is known whether or not these radiomics signatures of which the radiomic signature data 18 is indicative, occur or apply to each one of the second images 12. This, in turn is indicative of the type of neoplasm visualized in each of the images 12: with the ‘type’ of neoplasm it is meant here that the radiomics signature is indicative of how the image neoplasm will respond to a certain treatment, e.g. by chemical therapy or radiation therapy. Because for each of the neoplasms in the second images 12 this data is available, the modified second images 14, which in content are identical to the second images 12 but which bear the style of the first image 11 (as if these were obtained using the first imaging system 16), can be used as training data for training the third machine learning data processing model 30 to obtain a new radiomics signature which is in the same way indicative of the type of the neoplasm (i.e. how it responds to therapy). This news radiomics signature 32 may then be associated with all images coming from the first imaging system 16.

Figure 3 schematically illustrates an alternative to the obtaining of the style representation data 13 applied in figure 2. In figure 3, a plurality of first images 11 is used as input to the CNN 20 to thereby provide a plurality of associated style representation data sets 13. These style representation data sets 13 associated to each of the images 11 are then aggregated in step 35 to provide a single style representation data set 113 indicative of the general style of a first imaging system 16. For example, this variant may be applied to prevent that certain individual differences between the individual first images 11, e.g. caused by momentary disturbances for example may affect the style representation data 113. The style representation data 113 may then be used in a method illustrated in figure 2 to replace the style data 13 at the input of CNN 23.

Figure 4 illustrates a further alternative to the method of the present invention. In the method of figure 4, one or more second images 12 from the second imaging system are used as input to the convolutional neural network 20 that performs style retrieval. At the output of the CNN 20, style representation data 213 for the imaging style of the second images 12 is obtained. The second images 12 are associated, as illustrated by double dotted arrow 19 with radiomics data 18. The style representation data 213 of the second images 12 is provided as input to the CNN 23. As further input to the CNN 23 there is provided a first image 11 of the first imaging system 16. The CNN 23 applies the imaging style of the second images 12, represented by the style representation data 213, to the first images 11 of the first imaging system. This yields the modified first image 15 in the style of the second imaging system 17. Because of the fact that the first image 11 is now in the style of the second images 12 from the second imaging system 17, the radiomics signatures 18 used with the second images 12 may now likewise directly be applied to the modified first images 15 in step 29. This allows direct radiomic signature analysis to be performed on the images of the first imaging system 16.

A disadvantage of this method of figure 4 may be that it does not provide a validation of a radiomics signature to the original first images 11 of the first imaging system 16, nor does it provide the generation of a new radiomics signature. The big advantage is that it directly allows to apply the radiomics signature associated with the second imaging system 17 to the converted images 15 that are in the style of the second images 12. Therefore, this method allows to directly analyze each individual image from the first imaging system 16 using the radiomics signatures 18 of the second imaging system 17. The result is provided in step 40.

In a further aspect of the invention, there is provided a method of testing a style transfer mechanism for a processing of medical images by an analysis system for enabling radiomics signature analysis. This may be achieved by obtaining, by a controller of the analysis system, a first imaging data set of a first image obtained using a first imaging system. As described above, the first imaging data set can be provided as input to a first data processing model for analyzing and providing style representation data at an output of the data processing model. The style representation data is indicative of an imaging style independent of the image content. Next, the style representation data is provided to a second data processing model which applies the determined imaging style to the first image itself in order to obtain a modified second image. Thereafter, in order to evaluate the quality of the style transfer mechanism, the modified second image is compared with the original first image for determining any differences between the images.

The above method may well be applied to select a most suitable style transfer method for performing the method of the present invention in accordance with the first aspect, as described herein. For example, suppose there are five style transfer methods available. It may not be necessary to immediately select the most advanced style transfer method, if less sophisticated style transfer methods may possibly provide a sufficiently well style transfer result to enable performing the radiomics signature analysis. One way to decide whether a style transfer method is good enough is as follows. Take an image (or set of images) from the data set that was used to build the original radiomic biomarker. Learn the style, this may be done either from a single image, or an average style from the set of images. Then apply this style to the image or set of images. The image or set of images should thereafter, if the style transfer method is good enough, basically be unaltered. More importantly, the value of the radiomic biomarker for the image or set of images should be unaltered. The latter can be established by statistical tests already defined. If this is not the case, then the style transfer method is of insufficient quality for applying it in a method in accordance with a first aspect of the present invention. So instead of a forward-backward transfer to establish that nothing has changed, it can be determined in this manner whether or not the style transfer method is capable of doing an identity (i.e., no change) transformation.

Although the above-mentioned method in the embodiments described works well to enable validation of radiomic signatures or generation of new signatures for images obtained from new or unknown imaging systems, the optional further steps described below and schematically illustrated in figure 5 may be performed in order to obtain a further validation for the conversion method itself. In particular, the additional method steps illustrated in figure 5 enable to verify whether the style transversion method that is or will be applied, will deliver a reliable conversion of only style features without introducing new disturbances or artifacts that deteriorate the quality of the results. Because, as described hereinabove, the present application is applied in a medical context for making an assessment on whether or not tumours respond well to certain treatment or whether a different type of treatment may be better, performing such a validation of a conversion method may be well desired under certain circumstances. For example, in the method of figure 4, the additional method steps of figure 5 may be highly desired. This is because the radiomics signature data applied to the second images 12 of the second imaging system 17, in this embodiment of the present invention, will be applied directly to the first images after conversion thereof by the CNN 23. Therefore, if the conversion itself is of insufficient quality, the application of the radiomics signatures 18 to the modified first images 15 may provide incorrect results. To a lesser degree, the validation method of the additional steps illustrated in figure 5 may provide added value in the method of for example figures 1 and 2. In these embodiments of the method of the present invention, performing the additional method steps of figure 5 is less critical because, as illustrated in figure 2, the method is followed by a validation in step 25 of the radiomics signature data or alternatively by the generation of a new radiomics signature by neural network 30. Any errors or disturbances created during conversion will therefore be of less impact on the end results of the method. Reliability, however, may nevertheless be improved by performing the additional steps of figure 5. The validation step illustrated in figure 5 works as follows. A plurality of second images 12 obtained from second imaging system 17 is applied to the style retrieval CNN 23-1 as input. CNN 23-1 further receive the style 13-1 of the first imaging system 16. The second images 12 are converted into modified second images 14. These modified second images 14 are then applied as input to a second style transfer CNN 23-2, together with style representation data 13-2 of the second imaging system 17 that provided the original second images 12. The CNN 23-2 will therefore back convert the modified second images 14 into forward backward converted second images 12’. Thereafter, in step 38, the original second images 12 and the forward backward converted second images 12’ may be compared with each other to establish whether the style transfer method has affected the quality of the second images 12. If large discrepancies are found in the comparison step 38, a style transfer method as applied by CNN 23-1 and 23-2 may be unreliable.

Hereinafter, various characteristics of various embodiments of the invention will be highlighted in a set of numbered clauses or paragraphs. These characteristics are not to be interpreted as being limiting on the invention or inventive concept, but are provided merely as a highlighting of some characteristics of the invention as described in the description, without suggesting a particular order of importance or relevancy of such characteristics. The invention is only limited by the appended claims presented in the correspondingly named section. The numbered clauses include:

1. Method of processing medical images by an analysis system for enabling radiomics signature analysis, for enabling classification of a clinical picture in a human or animal body, such as a neoplasm, a lesion or a disorder, the method comprising: obtaining, by a controller of the analysis system, at least one first imaging data set of at least one first image obtained using a first imaging system; obtaining, by the controller, at least one second imaging data set of at least one original second image obtained using a second imaging system; providing, by the controller, the at least one first imaging data set as input to a first data processing model, wherein the first data processing model is configured for analyzing the first imaging data set such as to provide, at an output of the first data processing model, a style representation data associated with the first imaging system, wherein the style representation data is indicative of an imaging style of the first imaging system independent of an image content of the first image; providing, by the controller, the style representation data and the at least one second imaging data set to a second data processing model, wherein the second data processing model is configured for, based on the style representation data, applying the imaging style to the at least one second imaging data set to obtain at least one modified second image, the at least one modified second image being a representation of the at least one original second image as if it would have been obtained with the first imaging system.

2. Method according to clause 1, wherein the analysis system is configured for applying an algorithm on imaging data obtained from at least one imaging system from a pool of known imaging systems, such as to identify the occurrence of a radiomics signature in at least a part of the imaging data, and wherein the second imaging system is formed by an imaging system from the pool of known imaging systems, and the first imaging system is not comprised by the pool of known imaging systems; wherein the at least one second imaging data set is associated with a plurality of original second images; and wherein based on a comparison between the plurality of original second images and a plurality of modified second images associated therewith, a validity of the radiomics signature is verified for images obtained with the first imaging system.

3. Method according to clause 2, further comprising the steps of providing the plurality of modified second images to an analysis system for calculating a plurality of image features from the images, calculating based on the plurality of original second images signature values for the radiomics signature, training a third data processing model for establishing a new radiomics signature from the plurality of modified second images, the training comprising: a. providing, for each second image of the plurality of modified second images, the plurality of image features to the third data processing model; b. providing, for each second image of the plurality of original second images, the signature values to the third data processing model; and c. training the third data processing model based on the signature values received in step b. and the plurality of image features received in step a., for providing the new radiomics signature associated with images obtained from the first imaging system.

4. Method according to clause 3, further comprising a step of applying the new radiomics signature to images obtained from the first imaging system, such as to perform classification of said clinical picture in said images.

5. Method according to any one or more of clauses 2-4, wherein the method comprises: obtaining, by the controller, a further second imaging data set of one or more further second images obtained using the second imaging system; providing, by the controller, the further second imaging data set as input to the first data processing model such as to provide a further style representation data associated with the second imaging system; providing, by the controller, the style representation data of the first imaging system and the further second imaging data set to the second data processing model for applying the imaging style of the first imaging system to the one or more further second images to obtain a further modified second images thereof; providing, by the controller, the further style representation data of the second imaging system and the further modified second images to the second data processing model for applying the imaging style of the second imaging system to the further modified second images such as to obtain a forward-backward converted second images thereof; calculate the signature values of the radiomics signature based on the forward-backward converted second images, such as to validate the method.

6. Method according to clause 1, wherein the first imaging system is at least one imaging system from a plurality of known imaging systems, wherein the at least one imaging system is associated with at least one radiomics signature for enabling a classification of the clinical picture based on imaging data from said at least one imaging system; wherein the second imaging system is a further imaging system different from the plurality of known imaging systems; and wherein, after the step of applying the imaging style of the first imaging system to the at least one second imaging data set to obtain the at least one modified second image, the at least one radiomics signature is further associated with the at least one second imaging system for enabling said classification from images obtained from the at least one second imaging system.

7. Method according to clause 6, further comprising the steps of: providing, by the controller, the at least one second imaging data set as input to the first data processing model such as to provide, at the output of the first data processing model, further style representation data associated with the second imaging system, wherein the further style representation data is indicative of an imaging style of the second imaging system; providing, by the controller, the further style representation data and the at least one first imaging data set to the second data processing model for, based on the further style representation data, applying the imaging style of the second imaging system to the at least one first imaging data set to obtain at least one modified first image, the at least one modified first image being a representation of the at least one first image as if it would have been obtained with the second imaging system; providing, by the controller, the style representation data of the first imaging system and the at least one modified first image to the second data processing model for obtaining at least one forward-backward converted first image; and calculate the signature values of the radiomics signature associated with the first imaging system based on the forward-backward converted first image, such as to validate the method.

8. Method according to any one or more of clauses 6-7, further comprising a step of applying the at least one radiomics signature to images obtained from the second imaging system, such as to perform classification of a clinical picture in said images.

9. Method according to any one or more of the preceding clauses, wherein at least one of the first data processing model, the second data processing model or, where dependent on clause 3, the third data processing model, includes at least one element of a group comprising: at least one machine learning data processing model, such as a convolutional neural network; a system of two or more machine learning data processing models, such as a convolutional neural network in addition to a further machine learning data processing model; or a combination of at least one image transformation network including a deep residual convolutional neural network, and at least one loss network pre- trained for performing perceptual optimization; or one or more functional data processing steps or algorithms, such as: an image segmentation step, optimization steps such as based on Markov Random Field analysis, or image reconstruction steps; or image decomposition based on Laplacian stack analysis, image blending based on energy computation of Laplacian levels, and level aggregation.

10. Method according to any one or more of the clauses 1-8, wherein at least one of the first data processing model and the second data processing model both include a convolutional neural network in addition to a further machine learning data processing model, and wherein the convolutional neural network of the first and second data processing model is the same convolutional neural network, and wherein the second data processing model is configured for applying both the style representation data and the second imaging data set independently to the convolutional neural network.

11. Method according to clauses 10, wherein the convolutional neural network is configured for performing image pattern recognition, such as object detection or image reconstruction.

12. Method according to clause 10 or 11, wherein the convolutional neural network comprises a plurality of layers, wherein based on the at least one first imaging data set each layer provides a first image filter response data set for the respective layer; and wherein the further machine learning data processing model of the first data processing model is configured for receiving the first image filter response data sets of the plurality of layers at a first input and calculating a correlation between the plurality of layers. 13. Method of testing a style transfer mechanism for a processing of medical images by an analysis system for enabling radiomics signature analysis, the method comprising: obtaining, by a controller of the analysis system, at least one first imaging data set of at least one first image obtained using a first imaging system; providing, by the controller, the at least one first imaging data set as input to a first data processing model, wherein the first data processing model is configured for analyzing the first imaging data set such as to provide, at an output of the first data processing model, a style representation data associated with the first imaging system, wherein the style representation data is indicative of an imaging style of the first imaging system independent of an image content of the first image; providing, by the controller, the style representation data and the at least one second imaging data set to a second data processing model, wherein the second data processing model is configured for, based on the style representation data, applying the imaging style to at least one second imaging data set to obtain at least one modified second image; wherein the at least one second imaging data set is identical to the at least one first imaging data set; and wherein the method further includes a step of comparing the at least one modified second image with the at least one first image, for determining a difference between the at least one modified second image and the at least one first image, for performing an evaluation of the style transfer mechanism.

The present invention has been described in terms of some specific embodiments thereof. It will be appreciated that the embodiments shown in the drawings and described herein are intended for illustrated purposes only and are not by any manner or means intended to be restrictive on the invention. It is believed that the operation and construction of the present invention will be apparent from the foregoing description and drawings appended thereto. It will be clear to the skilled person that the invention is not limited to any embodiment herein described and that modifications are possible which should be considered within the scope of the appended claims. Also kinematic inversions are considered inherently disclosed and to be within the scope of the invention. Moreover, any of the components and elements of the various embodiments disclosed may be combined or may be incorporated in other embodiments where considered necessary, desired or preferred, without departing from the scope of the invention as defined in the claims. The invention has been explained in respect of the field of radiomics signature analysis. Examples of such analysis methods are for example described in International patent applications WO 2014/171830, WO 2016/060557 In the claims, any reference signs shall not be construed as limiting the claim. The term 'comprising' and ‘including’ when used in this description or the appended claims should not be construed in an exclusive or exhaustive sense but rather in an inclusive sense. Thus the expression ‘comprising’ as used herein does not exclude the presence of other elements or steps in addition to those listed in any claim. Furthermore, the words ‘a’ and ‘an’ shall not be construed as limited to ‘only one’, but instead are used to mean ‘at least one’, and do not exclude a plurality. Features that are not specifically or explicitly described or claimed may be additionally included in the structure of the invention within its scope. Expressions such as: "means for ...” should be read as: "component configured for ..." or "member constructed to ..." and should be construed to include equivalents for the structures disclosed. The use of expressions like: "critical", "preferred", "especially preferred" etc. is not intended to limit the invention. Additions, deletions, and modifications within the purview of the skilled person may generally be made without departing from the spirit and scope of the invention, as is determined by the claims. The invention may be practiced otherwise then as specifically described herein, and is only limited by the appended claims.