CROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No.

62/415,418 filed on October 31 , 2016, the entire content of which is incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present disclosure relates to a method and system for image segmentation using controlled feedback, and more particularly, to a neural network- based method and system for image segmentation with controlled feedback that allows segmented images with unbalanced class information and also allow the network to initialize the weights properly.

BACKGROUND OF THE INVENTION

[0003] Detecting, segmenting, and classifying objects, for example, in medical images can be important for detection and diagnosis of diseases. Deep neural networks (NNs), including convolutional neural networks (CNN), as well as other types of multilevel neural networks, are an existing method for improved feature learning, classification, and detection.

[0004] Pixel-wise labeling or semantic segmentation is a process of assigning each pixel a label of the class to which they belong. For example, a segmented image will have the same labels for all the pixels that correspond, for example, to human, in an image. However, one problem with current convolution neural networks is that they need weight initialization. In addition, weights can be initialized randomly, however, it can take a long time for the weights to converge.

[0005] For example, methods have been proposed that take into account class imbalance information at the last stage (loss computation) of the network, however, the methods still require a long time for the network to converge, in addition, there has been work to strengthen the weights of convolution layer by domain transfer knowledge. However, these methods rely on the output of the pre-trained network, and generally tend to strengthen the edge information. SUMMARY OF THE INVENTION

[0006] In accordance with an exemplary embodiment, a system and method are disclosed, which are capable of strengthening the weights of edges as well as entire region. Further, the controlled nature of the disclosed method allows the model to strengthen the weights of a particular class which is not possible with techniques such as domain transfer knowledge, for example, edges detected via domain transform based models are for an entire image, and since the system may not be able to classify which edge belongs to which object and hence, makes it difficult to apply for a particular class.

[0007] For example, accurate cell body extraction can greatly help to quantify ceil features for further pathological analysis of cancer cells. In a practical scenario, for example, ceil image data often has the following issues: a wide variety of appearances resulting from different tissue types, block cuttings, staining process, equipment and hospitals, and ceil image data is gradually collected over time and the collected data is usually unbalanced, for example, some types of cell images are greater than other types of ceil images.

[0008] In this disclosure, a method is disclosed to provide feedback early in the network so that network can initialize with strong weights (or probabilities) and converge earlier, thus reducing the training time and can improve learning, for example, for extraction or identification of cell bodies.

[0009] In consideration of the above issues, it would be desirable to have a system and method to control the weights of the neural network by feedback. In accordance with an exemplary embodiment, the method and system emphasizes the weights that are important and de-emphasizes (or un-emphasizes) the weights that are less important. Emphasizing the weights (or probabilities) earlier in the process can help in initializing the network weights properly, and which can help the network to converge earlier and improve the learning of the network.

[0010] A method is disclosed for image segmentation using controlled feedback in a neural network, the method comprising: extracting image data from an image; performing one or more semantic segmentations on the extracted image data;

introducing one or more classifiers to each of the one or more semantic

segmentations, each of the one or more classifiers assigning a probability to one or more classes of objects within the image; and generating a segmentation mask from the one or more semantic segmentations.

[0011] A non-transitory computer readable recording medium stored with a computer readable program code for image segmentation using controlled feedback in a neural network is disclosed, the computer readable program code configured to execute a process comprising: extracting image data from an image; performing one or more semantic segmentations on the extracted image data; introducing one or more classifiers to each of the one or more semantic segmentations, each of the one or more classifiers assigning a probability to one or more classes of objects within the image; and generating a segmentation mask from the one or more semantic segmentations,

[0012] A system is disclosed for image segmentation using controlled feedback in a neural network, the system comprising: a processor; and a memory storing instructions that, when executed, cause the system to: extract image data from an image; perform one or more semantic segmentations on the extracted image data; introduce one or more classifiers to each of the one or more semantic

segmentations, each of the one or more classifiers assigning a probability to one or more classes of objects within the image; and generate a segmentation mask from the one or more semantic segmentations.

[0013] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are included to provide a further

understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] FIG. 1 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment.

[0016] FIG. 2 is another illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment. [0017] FIG. 3 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment with a cell region as a feedback.

[0018] FIG, 4 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment with a cell boundary as a feedback.

[0019] FIG. 5 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment during a testing phase with feedback.

[0020] FIG. 6 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment with multiple image class regions as a feedback.

Ut ! AiLKlU ut¾¾# sr 8 SUPi

[0021] Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0022] In accordance with an exemplary embodiment, a method and system are disclosed, which can instruct (or tell) the convolution neural network that certain neurons are important and thus, emphasizes the weights corresponding to those neurons. For example, in accordance with an exemplary embodiment, the method and system allows the network to emphasize, de-emphasize, or un-change the weights of the network. For neural networks to converge, weight initialization can be a very important step and several methods have been proposed for weight initialization. Once the weights are initialized for different layers, data is passed through the network several times, so that network can converge. Usually, however, it takes a lot of time for a network to converge.

[0023] In accordance with an exemplary embodiment, a method and system are disclosed that instructs (or tells) the network that these are important neurons by means of feedback and thus emphasizes the weights of corresponding neurons. In addition, the controlled nature of the method as disclosed allows the model to strengthen the weights of a particular class, which is not possible, for example, with techniques such as domain transfer knowledge.

[0024] In cyclic learning, the networks currently can be trained in stages whereby a model is first or initially trained with the easy data and then fine-tuned using the difficult data. In addition to this type of learning, the method as disclosed allows the system or method to learn the network in cycles for same (data that can be easily learned) or different (data is difficult to learn) data. For example, the first 2 epochs (trainable encoders and/or trainable decoders) can be learned with feedback while the next, for example, 5 epochs (trainable encoders and/or trainable decoders) can be learned without feedback and so on until the network converges, which can help with the learning such that the model can find a local minima relatively early.

[0025] In accordance with an exemplary embodiment, due to the controlled nature, the system and method as disclosed can be used for semi-supervised or unsupervised learning. In accordance with an exemplary embodiment, in a prediction phase, the method and system can use previous results as the masks to conduct the feedbacks, for example, to periodically improve a current model.

[0026] For example, cell images are unbalanced class images where background information is generally greater (or more prevalent) in comparison to foreground (such as cell). In accordance with an exemplary embodiment, for example, the method as disclosed can emphasize the weights of cells while de-emphasizing, for example, the weights of the background.

[0027] FIG. 1 is an illustration of an encoder-decoder system 100 for semantic segmentation in accordance with an exemplary embodiment without feedback. As shown in FIG. 1 , the encoder-decoder system 100 includes an input image 1 10, a plurality of trainable encoder blocks 120, 122, 124, a plurality of trainable decoder blocks 130, 132, 134, and a segmentation mask 140. in accordance with an exemplary embodiment, the plurality of encoder blocks 120, 122, 124, or non-linear processing layers, can consist of operations such as convolution, activation, batch normalization, and down sampling. The corresponding plurality of decoder blocks 130, 132, 134 can consist of operations such as deconvolution, activation, batch normalization, and up-sampling. [0028] In accordance with an exemplary embodiment, the plurality of trainable encoder blocks 120, 122, 124, and the plurality of trainable decoder blocks 130, 132, 134, can be hosted on a computer system or processing unit 150, which can include a processor or central processing unit (CPU) and one or more memories for storing software programs and data. The processor or CPU carries out the instructions of a computer program, which operates and/or controls at least a portion of the functionality of the computer system or processing unit 150. The computer system or processing unit 150 can also include an input unit, a display unit or graphical user interface (GUI), and a network interface (l/F), which is connected to a network communication (or network). The computer system or processing unit 150 can also include an operating system (OS), which manages the computer hardware and provides common services for efficient execution of various software programs. For example, some embodiments may include additional or fewer computer system or processing unit 150, services, and/or networks, and may implement various functionality locally or remotely on other computing devices (not shown). Further, various entities may be integrated into to a single computing system or processing unit 150 or distributed across additional computing devices or systems 150.

[0029] FIG. 2 is an illustration of an encoder-decoder system 200 for semantic segmentation in accordance with an exemplary embodiment. As shown in FIG. 2, the system 200 can include the input image 1 10, the plurality of trainable encoder blocks 120, 122, 124, the plurality of trainable decoder blocks 130, 132, 134, the segmentation mask 140, a plurality of not trainable feedback blocks for the encoder 220, 222, 224, a plurality of not trainable feedback blocks for the decoder 230, 232, 234, a plurality of weight functions (or bound the weight between (a. * a, a * h)} 240, 241 , 242, 243, 244, 245, and a plurality of merging operations 250, 251 , 252, 253, 254, 255. In accordance with an exemplary embodiment, the plurality of not trainable encoder blocks 220, 222, 224, can consist of operations, for example, such as convolution and down sampling. The corresponding plurality of not trainable decoder blocks 230, 232, 234 can consist of operations, for example, such as deconvolution and up-sampling.

[0030] In accordance with an exemplary embodiment, the system 200 also includes a feedback controller 280. The feedback controller 280 can be configured to change or adjust the respective weights of one or more classes by assigning a weight, to each of the one or more classes within the image 1 10. In accordance with an exemplary embodiment, the plurality of weight functions 240, 241 , 242, 243, 244, 245 can assign a probability to each of the plurality of pixels of the input image 1 10, if each of the plurality of pixels belongs to a certain class of pixels. For example, in ceil detection, the classification weights of the foreground, which can include cell regions or boundaries between cell regions can be greater than the classification weights of the background, and, for example, a stain color, in addition, the feedback controller 280 can be "ON", or alternatively, can be "OFF", such that each of the classification weights is equal or set to set number, for example, one (1 ).

[0031] In accordance with an exemplary embodiment, the feedback controller 260 can be hosted on a computer system or processing unit 150 as shown in FIG. 1 , or alternatively, can be hosted on a separate computer system or processing unit 270. For example, the separate computer system or processing unit 270 can include a processor or central processing unit (CPU) and one or more memories for storing software programs and data. The processor or CPU carries out the instructions of a computer program, which operates and/or controls at least a portion of the functionality of the computer system or processing unit 150. The computer system or processing unit 270 can also include an input unit, a display unit or graphical user interface (GUI) for imputing data, and a network interface (l/F), which is connected to a network communication (or network). The computer system or processing unit 270 can also include an operating system (OS), which manages the computer hardware and provides common services for efficient execution of various software programs. For example, some embodiments may include additional or fewer computer system or processing unit 150, 270, and/or networks, and may implement various

functionality locally or remotely on other computing devices (not shown). Further, various entities may be integrated into a single computing system or processing unit 150, 270 or distributed across additional computing devices or systems 150, 270. in accordance with exemplary embodiment, for example, the display unit or GUI can be used to input the image 1 10 into the system or processing unit 150, 270, to visualize the segmentation mask 140, or input information pertaining to classes via a feedback map.

[0032] In accordance with an exemplary embodiment, the system and method for semantic segmentation can include a training phase having an input training data set denoted b S = {(X _n ; Y _n ), n =1 ... N}, where sample X _n = \x ^' "' = 1, ... | ^" _n |} denotes

corresponding ground truth label for image X _n . The subscript n for notational simplicity has been subsequently dropped. In accordance with an exemplary embodiment, W _e and W _d denotes the layer parameters for the encoder and decoder respectively.

[0033] In accordance with an exemplary embodiment, a network is disclosed that can be configured to emphasize the weights for certain (or all, excluding

background) classes and de-emphasize (or remain same as initialized) for other classes. For example, in accordance with an exemplary embodiment, to emphasize important class information over other information such as background, a class selection weight γ can be introduced on a per class basis. A feedback map is then generated as Y _f = j = 1, . . . |X _n j], y > e {0,1}, c e {0, C] where G denotes

the number of classes. In accordance with an exemplary embodiment, a feedback map is then passed through the feedback network to generate weights w _e and w _d . The weights of feedback layers can be represented as (wg, , v/ _d , . . . . , w _d ). In accordance with an exemplary embodiment, the value of w can be greater than 1 , however, if the value of w is greater than 1 , the value may result in the network not converging to a local minima. In accordance with an exemplary embodiment, the weights of feedback network layers can be updated as: where W represents the encoder and decoder weights for the feedback network, respectively

[0034] In accordance with an exemplary embodiment, the weight emphasis function or merging operation for the encoder and decoder can be defined as:

£(W _e , w _e ) = W _e * aw _t e where * can be any element wise operation (addition, multiplication, subtraction, etc.), a and β are scaling parameters for the encoding and decoding stages respectively.

[0035] In accordance with exemplary embodiment, each of the plurality of weight functions 240, 241 , 242, 243, 244, 245 for the feedback network 220, 222, 224, 230, 232, 234 as disclosed herein can be the same for each of the feedback networks 220, 222, 224, 230, 232, 234, or alternatively, one or more of the plurality of weight functions 240, 241 , 242, 243, 244, 245 as disclosed herein can be different. For example, as shown in FIG. 2, the first 2 feedback networks (or epochs) 220, 222 can be learned with feedback while next, for example, 4 feedback networks (or epochs) 224, 230, 232, 234 can be learned without feedback and so on until the network converges, which can help with the learning such that the model can find the local minima earlier.

[0036] In accordance with an exemplary embodiment, in image-to-image training, for example, the loss function can be computed over all pixels in a training image X and ground truth label image Y. For example, during the testing phase, given image X, the segmentation predictions were obtained, for example, as:

Y - CCNNSS (X, (W _e , W _d ))

[0037] In accordance with an exemplary embodiment, a number of object classes can be different. For example, in ceil images, background pixels can be more prevalent in comparison to boundary and ceil pixels. Accordingly, in the system and method as disclosed, emphasizing the weights of different classes, for example, ceil boundaries or cell regions over background pixels can be performed.

[0038] FIG. 3 is an illustration of an encoder-decoder system 300 for semantic segmentation in accordance with an exemplary embodiment with a ceil region 310 used as feedback. As shown in FIG. 3, for example, the system 300 can be configured using the feedback controller 260 to emphasize a cell region (or ceil region mask) 310 on an input image 1 10 from an analysis, for example, for cancer ceils, by assigning a probability to each of the foreground pixels and background pixels, which can likely represent, for example, cell regions and non-cell regions, respectively. [0039] FIG. 4 is an illustration of an encoder-decoder system for semantic segmentation in accordance with an exemplary embodiment with a cell boundary 410 used as feedback. As shown in FIG. 4, for example, the system 400 can be configured using the feedback controller 280 to emphasize a cell boundary (or cell boundary mask) 410 on an input image 1 10 from an analysis, for example, for cancer cells, by assigning a probability to each of the foreground and background pixels, which can likely represent, for example, cell boundaries and non-cell boundaries or regions.

[0040] FIG. 5 is an illustration of an encoder-decoder system 500 for semantic segmentation in accordance with an exemplary embodiment during, for example, a testing phase with feedback. For example, manually annotating images can be difficult and time consuming. In accordance with an exemplary embodiment, data available for training a neural network for medical data is not as large as general images (in general, medical image datasets contain a few thousand images, while general image datasets can contain several thousand images). Thus, generating good segmentation results can be difficult.

[0041] In accordance with an exemplary embodiment, due to the feedback nature of the method as disclosed, the method and system 500 can allow the network to learn even in case of testing (or training) time. For example, the method as disclosed can give the flexibility that the user can discard the incorrect labels or correct them and then feed the output to the network for fine-tuning the weights via user input 520. The user input 520 can be input via the computer system or processing unit 150, 270, which processes the image 1 10, or alternatively, can be performed by a remote computer system or processing unit 530. In accordance with an exemplary embodiment, the remote computer system or processing unit 530 can be in communication computer system or processing unit 150 via a communication network.

[0042] FIG. 8 is an illustration of an encoder-decoder system 800 for semantic segmentation in accordance with an exemplary embodiment with multiple image class regions as a feedback. As shown in FIG. 8, the system and methods as disclosed can also be used for general images 840, containing or illustrating, for example, people, cars, motorcycles, trees, etc. In FIG. 6, the input image 810 can contain multiple classes, such that, for example, the system and method as disclosed herein can be applied to emphasize the weights of a human and/or motorbike instead of emphasizing all other classes such as trees, road, etc. In accordance with an exemplary embodiment, for example, as shown in FIG. 6, the feedback channel can treat a human and/or a motorbike as a foreground class and generate a mask 610 from the human and/or motorbike.

[0043] in accordance with an exemplary embodiment, a non-transitory computer readable recording medium stored with a computer readable program code for image segmentation using controlled feedback in a neural network is disclosed. The computer readable program code configured to execute a process comprising:

extracting image data from an image; performing one or more semantic

segmentations on the extracted image data; introducing one or more classifiers to each of the one or more semantic segmentations, each of the one or more classifiers assigning a probability to one or more classes of objects within the image; and generating a segmentation mask from the one or more semantic segmentations.

[0044] The non-transitory computer readable medium may be a magnetic recording medium, a magneto-optic recording medium, or any other recording medium which will be developed in future, ail of which can be considered applicable to the present invention in ail the same way. Duplicates of such medium including primary and secondary duplicate products and others are considered equivalent to the above medium without doubt. Furthermore, even if an embodiment of the present invention is a combination of software and hardware, it does not deviate from the concept of the invention at all. The present invention may be implemented such that its software part has been written onto a recording medium in advance and will be read as required in operation.

[004S] It will be apparent to those skilled in the art that various modifications and variation can be made to the structure of the present invention without departing from the scope or spirit of the invention, in view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims and their equivalents.