Cross-Reference to Related Applications

[0001] This application claims priority from US application No. 63/136567 filed 12 January 2021 and entitled SEQUENTIAL CONVOLUTIONAL NEURAL NETWORKS FOR NUCLEI SEGMENTATION which is hereby incorporated herein by reference for all purposes. For purposes of the United States of America, this application claims the benefit under 35 U.S.C. §119 of US application No. 63/136567 filed 12 January 2021 and entitled SEQUENTIAL CONVOLUTIONAL NEURAL NETWORKS FOR NUCLEI SEGMENTATION which is hereby incorporated herein by reference for all purposes.

Field

[0002] The present invention relates to image recognition and to histology. Example embodiments provide automatic segmentation of cell nuclei in images of tissue sections.

Background

[0003] There is a need for better ways to detect neoplastic diseases such as cancer and to assess the likely progress of such diseases. Histopathology (the microscopic study of diseased tissue) has been the gold standard for cancer diagnosis and prognosis for over 100 years. In histopathology thin (typically several pm thick) tissue sections are reviewed by specialist physicians who may observe patterns that allow them to characterize the observed tissue. The observed patterns result from the aggregate are effects of molecular alterations on cell behavior and represent phenotypic information that has been found to be correlated to certain disease states and to expected outcomes. For the neoplastic process this information provides a convenient visual readout of disease state and aggressiveness.

[0004] Whole slide scanning has enabled the creation of digital pathology representations (digital images) of the entire section that normally would be examined visually through a microscope. These digital representations of the tissue are sufficiently accurate that expert visual interpretation of the digital images (as opposed to direct microscopic examination of the tissue samples themselves) has been approved for use in some patient diagnosis (Ref. 1). [0005] It would be beneficial to take advantage of computer-aided analysis to make processing of histology specimens more efficient and possibly also to discover novel features to predict cancer behavior (outcomes) and to improve understanding of the neoplastic process.

[0006] A range of machine learning (ML) technologies have been applied in an attempt to provide systems capable of automatic classification of tissues, identification of diseases and assessment of the progress of diseases .All machine learning methods applied to images form intermediate representations of the observed data and extract information from those representations. However, it is often difficult or impossible to interpret features of the intermediate representations themselves. It can therefore be difficult to achieve a level of trust in such ML systems that is sufficient to justify relying on such systems in cases where a person’s health is at stake.

[0007] Deep Learning (DL) using some form of Convolutional Neural Networks (CNN) have been successfully applied to images of sectioned tissue to recognize cancer, stage cancer and even to predict biological aggressiveness of the cancers from which the tissue came (Ref 2). While some DL has demonstrated the ability to perform as well as or better than expert pathologists for the recognition and classification of cancerous tissue, the DL only produces the answer it was trained to recognize and does not easily offer additional information (Refs 2,3). The general consensus is that the black box nature of many DL-based systems does not inspire trustworthiness which will be an impediment to clinical adoption. Thus explainable-AI, “Interpretability” of how the DL comes to its conclusions has become an area of active research(4,5). [0008] Characteristics of cell nuclei tend to be important phenotypic information. However, it is challenging to make a practical image processing system capable of reliably automatically segmenting cell nuclei in digital pathology representations. One reason for this is that digital pathology representations often include clusters of overlapping cell nuclei such that individual pixels may correspond to zero, one or plural nuclei. Additional complicating factors include intensity variations caused by noise and uneven absorption of stains.

[0009] Robust accurate segmentation of cell nuclei for overlapping nuclei clusters is one of the most significant unsolved issues in digital pathology. A segmentation algorithm as accurate as human annotators would be a great enabler for many fields of research and clinical utility in digital pathology. It would in fact be transformative to the field. Increased cell density associated with dysregulated growth is frequently where interesting cancer biology takes place and because of the increased cell density the likelihood of cell overlaps is much higher and quantitative analysis is traditionally more difficult to perform.

[0010] Despite progress that has been made in the field of image recognition and digital pathology there remains a need for improved and alternative technologies for segmenting cell nuclei in digital pathology representations.

Summary

[0011] The present invention provides methods and apparatus for automatic segmentation of cell nuclei in digital histopathology representations (DHR mages). In example embodiments the methods and apparatus apply separate ML algorithms in sequence. A first ML algorithm determines locations of cell nuclei in a DHR image. A second ML algorithm processes a patch around each of the determined locations to determine a boundary of a corresponding cell nucleus.

[0012] One example aspect of the invention provides a method for segmenting cell nuclei in medical images. The medical images may, for example be digital histopathology representations, cytology images, cytopathology images, or in vivo histology images (obtained by any modality). The method comprises: by a first trained machine learning algorithm processing a medical image to provide center locations of cell nuclei depicted in the medical image; and by a second trained machine learning algorithm processing each of a plurality of patches of the medical image, each of the patches corresponding to one of the plurality of center locations, the processing by the second trained machine learning algorithm outputting a nuclear boundary corresponding to the corresponding one of the center locations.

[0013] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network. In some embodiments the first convolutional neural network has a UNet configuration. In some embodiments the UNet configuration comprises 5 or more layers. In some embodiments the first convolutional neural network has a configuration selected from UNet++, Mask R-CNN, FastFCN, Gated - SCNN, and DeepLab.

[0014] In some embodiments, processing each of the plurality of patches of the medical image by the second machine learning algorithm comprises receiving each of the patches as input to a second convolutional neural network. In some embodiments the second convolutional neural network has a UNet configuration. In some embodiments the second convolutional neural network has a configuration selected from UNet++, Mask R-CNN, FastFCN, Gated - SCNN, and DeepLab.

[0015] In some embodiments the patches are equal in size. In some embodiments the patches of the medical image are centered on the corresponding one of the plurality of center locations. In some embodiments the patches of the digital histopathology representation are square. In some embodiments the patches of the digital histopathology representation have dimension of at least 80 by 80 pixels. In some embodiments the patches of the digital histopathology representation have dimension of at least 128 by 128 pixels.

[0016] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are different from one another. [0017] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are the same as one another.

[0018] In some embodiments the method further comprises obtaining cell information corresponding to the center locations and processing the cell information together with the center locations to perform cell type based cell-cell association quantification. The cell information may, for example, comprise morphologically based and/or immunohistochemistry (I HO) based characterization information.

[0019] In some embodiments the medical image includes one or more clusters of overlapping cell nuclei.

[0020] In some embodiments the method further comprises applying feature calculations and a binary classification tree to classify objects corresponding to the nuclear boundaries.

[0021] Another aspect of the invention provides apparatus for segmenting cell nuclei in medical images. The medical images may, for example be digital histopathology representations, cytology images, cytopathology images, or in vivo histology images (obtained by any modality). The apparatus comprises: a first trained machine learning algorithm operative to process a medical image to provide center locations of cell nuclei depicted in the medical image; and a second trained machine learning algorithm operative to process each of a plurality of patches of the medical image, each of the patches corresponding to one of the plurality of center locations, the processing by the second trained machine learning algorithm outputting a nuclear boundary corresponding to the corresponding one of the center locations.

[0022] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network. In some embodiments the first convolutional neural network has a UNet configuration.

In some embodiments the UNet configuration comprises 5 or more layers.

In some embodiments the first convolutional neural network has a configuration selected from UNet++, Mask R-CNN, FastFCN, Gated - SCNN, and DeepLab. [0023] In some embodiments the second machine learning algorithm is configured to receive each of the patches as input to a second convolutional neural network. In some embodiments the second convolutional neural network has a UNet configuration. In some embodiments the second neural networh has five or more layers. In some embodiments the second convolutional neural network has a configuration selected from UNet++, Mask R-CNN, FastFCN, Gated - SCNN, and DeepLab.

[0024] In some embodiments the patches are equal in size. In some embodiments the patches of the medical image are centered on the corresponding one of the plurality of center locations. In some embodiments the patches of the digital histopathology representation are square. In some embodiments the patches of the digital histopathology representation have dimension of at least 80 by 80 pixels. In some embodiments the patches of the digital histopathology representation have dimension of at least 128 by 128 pixels.

[0025] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are different from one another. [0026] In some embodiments the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are the same as one another.

[0027] In some embodiments The apparatus further comprises a data processor configured to obtain cell information corresponding to the center locations and processing the cell information together with the center locations to perform cell type based cell-cell association quantification. The data processor optionally also implements one or both of the first and second machine learning algorithms. The cell information may, for example, comprise morphologically based and/or immunohistochemistry (IHC) based characterization information.

[0028] In some embodiments the apparatus is operable to instance segment individual cell nuclei in one or more clusters of overlapping cell nuclei included in the medical image.

[0029] In some embodiments the apparatus comprises a data processor configured to apply one or more feature calculations and a binary classification tree to classify objects corresponding to the nuclear boundaries.

[0030] Other aspects of the invention provide apparatus having any new and inventive feature, combination of features, or sub-combination of features as described herein and/or methods having any new and inventive steps, acts, combination of steps and/or acts or sub-combination of steps and/or acts as described herein.

[0031] Further aspects and example embodiments are illustrated in the accompanying drawings and/or described in the following description.

[0032] It is emphasized that the invention relates to all combinations of the above features with one another and with any features described elsewhere herein, even if these are recited in different claims.

Brief Description of the Drawings

[0033] The accompanying drawings illustrate non-limiting example embodiments of the invention.

[0034] Fig. 1 is a block diagram that shows apparatus according to an example embodiment as well as a flow of a method according to an example embodiment of the invention.

[0035] Fig. 1 A is a portion of an image of a histology slide in which cell nuclei have been stained.

[0036] Fig. 2 is a block diagram showing an example structure of a UNet convolutional neural network.

[0037] Fig. 3A is an example DHR image that includes cell nuclei marked with center locations that have been determined by a nuclear center finder.

[0038] Fig. 3B is an example training DHR image marked to show cell nuclei that have been eroded to generate additional training data. [0039] Figs.3C and 3D are portions of DHR images that have been marked to show nuclear centers and nuclear boundaries determined by a prototype system.

[0040] Fig. 4 shows an example user interface for a tool useful for viewing and annotating DHR images.

[0041] Fig. 5 is an example portion of a DHR image that has been marked to show nuclear centers and nuclear boundaries determined by a prototype system.

[0042] Figs 6A, 6B and 6C are respectively a section of a DHR image showing a cluster of cell nuclei, the section of the image marked to show boundaries of the cell nuclei in the cluster and a mask showing the boundaries only.

[0043] Figs 7A, 7B and 7C are respectively a section of a DHR image showing a cluster of cell nuclei, the section of the image marked to show boundaries of the cell nuclei in the cluster and a mask showing the boundaries only.

[0044] Figs 8A, 8B and 8C are respectively a section of a DHR image showing a cluster of cell nuclei, the section of the image marked to show boundaries of the cell nuclei in the cluster and a mask showing the boundaries only.

Detailed Description

[0045] Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive sense.

[0046] The present technology provides methods and apparatus for automated cell nucleus segmentation. Here “segmentation” involves identifying individual cell nuclei. Once individual nuclei have been identified, characteristics of the nuclei may be quantified.

[0047] A general approach applied by the present technology is to use sequential rounds of processing by two different trained convolutional neural networks (CNNs). In the following examples the CNNs have a UNet architecture.

[0048] The inventors have determined that the sequential application of two separate CNNs can facilitate reparsing an original image into multiple overlapping sub images. This feature explicitly allows for the present technology to perform a one pixel mapping to many objects (e.g. plural nuclei). An additional benefit of allowing a one to many assignment of pixels, is that the complete shapes (boundaries) of the nuclei can be more accurately identified since the same pixel(s) in an image may be assigned to multiple nuclei involved in the area of their overlap.

[0049] The inventors have found that on images with highly complex overlapping clusters of nuclei prototype implementations of the present technology were able to correctly segment more (10-20%, image complexity dependent) nuclei than other methods in areas of high complexity. In addition the prototype implementations were able to recognize and segment large numbers of nuclei completely missed by the other methods.

[0050] More recent prototype implementations trained using larger training data sets demonstrated segmentation accuracy on a 200K plus nuclei validation set at 92%. [0051] Fig. 1 is a block diagram of example apparatus 10 according to the invention. Fig. 1 also illustrates flow of an example method according to the invention. Apparatus 10 comprises a scanner 14 that is operative to scan a slide 12 to yield image data 15 for a digital histology representation (DHR). Slide 12 may for example comprise a thin tissue sample. The tissue sample may be prepared (e.g. by a suitable stain - for example thionin or hematoxylin) to enhance visibility of nuclei in the tissue sample. Scanner 14 may comprise a suitable commercial histology slide scanner for example. Histology slide scanners are commercially available from several suppliers. [0052] Fig 1A is a partial image of a slide 12 containing an example histology section. A histology section is composed of a tissue slice (a 3D collection of cells and parts of cells). The thickness of the section is typically on the same order of many of the components (nuclei, chromosomes, mitochondria, cell membranes) or much larger (proteins, lipids, etc.). Almost all of these are transparent to visible light. The tissue components may be visualized by labeling them (e.g. with a stain such as a suitable chromophore or fluorophore). For an imaging microscope the data collected is a projection of light through all or some of the sectioned stained tissue. It is highly likely that at least some pixels in the image contain information from two or more cells or cellular components. In slide 12 nuclei 13 of cells have been stained. Slide 12 includes several areas (e.g. 13A) in which plural nuclei 13 overlap with one another. [0053] DHR image data 15 is provided as input to a nuclear center finder 16 which is operative to fine centers of nuclei depicted in DHR image data 15. Nuclear center finder 16 may, for example comprise a trained machine learning (ML) system such as a trained CNN.

[0054] Nuclear center finder 16 outputs patches 17 from DHR image data 15 which each correspond to a located center of a cell nucleus. If centers of N cell nuclei are found by nuclear center finder 16, then nuclear center finder 16 may output N patches (17-1 , 17-2 ... 17-N). Each patch 17 includes the corresponding nuclear center determined by nuclear center finder 16. The corresponding nuclear center is consistently located in patches 17. For example, the corresponding nuclear center may be located at the center of each patch 17. For example, each patch 17 may be a square patch centered on a nuclear center determined by nuclear center finder 16. [0055] Patches 17 are dimensioned to be large enough to include boundaries of the corresponding nuclei. The minimum sizes for patches 17 depends on the resolution of DHR image data 15, optical magnification provided by scanner 14 and the actual sizes of nuclei in the tissue of slide 12. In a prototype embodiment patches 17 have sizes of 128 by 128 pixels.

[0056] Patches 17 are provided as input to a nuclear boundary finder 18. Nuclear boundary finder 18 may, for example comprise a trained ML system such as a trained CNN. For each patch 17, nuclear boundary finder 18 outputs boundary data 19 indicating a boundary for the corresponding cell nucleus. Boundary data 19 includes data 19-1 , 19-2, 19-N which each provides a boundary for one cell nucleus corresponding to one of patches 17-1 , 17-2 ... 17-N.

[0057] Optional data analyzer 20 processes boundary data 19 and outputs statistics 21 . Data analyzer 20 may, for example, output statistics data 21 indicative of one or more of:

• statistics (e.g. mean, median, standard deviation) of the areas of nuclei depicted in DHR image data 15;

• statistics regarding the shapes (e.g. eccentricity) of nuclei depicted in DHR image data 15;

• statistics regarding the length and/or tortuosity and/or curvatures of boundaries of nuclei depicted in DHR image data 15;

• statistics regarding the spatial distribution of nuclei depicted in DHR image data 15;

• etc.

[0058] Statistics data 21 may be provided as input to a further system 22 that processes statistics data 21 to yield indications of likely pathologies for the tissue of slide 12 and/or the likely future development of a pathology. System 22 may output class information 23. [0059] In some embodiments additional cell information 25 for cells depicted in DHR image 15 is supplied to data analyzer 20. Cell information 25 may, for example comprise cell characterization information (e.g. morphologically based and/or immunohistochemistry (IHC) based characterization). In such embodiments data analyzer 20 may perform cell type based cell-cell association quantification.

[0060] It is possible but not necessary that CNNs that are used to implement nuclear center finder 16 and nuclear boundary finder 18 be of the same type. In some embodiments the CNNs used to implement nuclear center finder 16 and nuclear boundary finder 18 are of different types. In a prototype system the CNNs used to implement nuclear center finder 16 and nuclear boundary finder 18 each had a UNet architecture. However, either or both nuclear center finder 16 and nuclear boundary finder 18 could be implemented by a CNN having an alternative architecture such as UNet++, Mask R-CNN, FastFCN, Gated - SCNN, DeepLab and others.

[0061] Methods for training CNNs are well understood by those of skilled in the art and are not described herein in detail. CNNs used to implement nuclear center finder 16 and/or nuclear boundary finder 18 may, for example be trained using iterative training which, in a training loop, optimizes weights in convolutional layers (learnable parameters) of the CNNs by minimizing a value of a loss function using gradients of the loss function with respect to the learnable parameters. The gradients may be determined from a model gradients function.

Example Prototype System

[0062] CNNs and Deep learning has demonstrated an ability to recognize objects as well as or better than humans. While most CNNs result in a classification or call for the image, there are a subset that can define the pixels involved in the object sought. Such CNNs can perform semantic segmentation or instance segmentation. UNet/UNet++ is one such CNN structure (Refs. 7,8,15). UNet is a form of CNN that can recognize specific objects and mark their boundaries once identified.

[0063] Fig 2 shows an example architecture for a UNet CNN 200. CNN 200 includes a contracting path or encoder 201 that receives DHR image data 15 and, over a series of levels 205-1 , 205-2, ... 205-N applies convolution layers 206 and max pooling 207 to produce a reduced size and increased depth intermediate representation 15A of DHR image data 15.

[0064] An expanding path or decoder 202 applies a series of levels that perform upsampling 208 and apply convolutional layers 209 to increase the size of intermediate representation 15A to produce an output image 15B. At each level a feature map 211 from encoder 201 is passed to decoder 202 and concatenated with an upsampled representation. CNN 200 may have any suitable number of layers and different numbers of convolution blocks in each layer.

[0065] In a prototype embodiment UNet CNNs having the general architecture shown in Fig. 2 were used to implement each of nuclear center finder 16 and nuclear boundary finder 18. Each of these CNNs had 5 levels with 2 layers of convolution blocks in each level. The number of convolutions in each block is doubled in every level starting from 32 (32, 64, 128, 256, 512 and back up again 256, 128, 64, 32). For the prototype implementation 3 by 3 convolutions were used everywhere except for the first two blocks of level 1 in the nuclear center finder 16 where it was found to be advantageous to use 5 by 5 convolutions and in the nuclear boundary finder 18 where it was found to be advantageous to use 5 by 5 convolutions for the first two levels of the encoding part (downwards, left side of the UNet) and in the final level of the decoding side (upwards right side of the UNet).

[0066] The UNet used to implement nuclear center finder 16 is configured by suitable training to identify the geometric centers of all nuclei within an image such as DHR image 15 (Fig. 1).

[0067] Using the nuclei geometric centers output by nuclear center finder 15, multiple patches centered on the previously identified nuclei geometric centers are selected from the DHR image 15. In the prototype the patches were each 128 by 128 pixel images however, patches of other suitable sizes could be used. It is desirable that the patches be large enough that the boundary of all or substantially all of any cell nucleus of interest that is depicted in DHR image 15 will lie within the patch when the geometric center of the cell nucleus is centered in the patch.

[0068] The UNet used to implement nuclear boundary finder 18 is configured by suitable training to identify the nuclear boundary for only the nucleus which is centered in each of these patch images.

[0069] The division between center finding and boundary finding and reparsing the original image into multiple potentially overlapping sub images explicitly allows for mapping of one pixel in an image such as a DHR image to many objects. It was found that providing patch images to nuclear boundary finder 18 which are aligned with corresponding nuclei (e.g. the center of a corresponding nucleus is at a predetermined location in patch image) facilitates accurate identification of the corresponding nuclear boundary even where the corresponding nucleus is part of a cluster of nuclei that overlap in DHR image 15.

[0070] In a prototype system used to demonstrate operation of the present technology included an AMD based computer workstation with an Nvidia 2070 based graphics card which was used to accelerate CNN calculations. UNet CNNs as described herein were implemented on this computer workstation. The prototype system was able to segment the nuclei in an entire prostate needle biopsy specimen section in 2-3 minutes.

Example Training Data

[0071] Another aspect of the invention relates to training of CNNs (e.g. UNet CNNs) that may be used to implement nuclear center finder 16 and nuclear boundary finder 18. A CNN may be trained to identify the geometric centers of all nuclei within a DHR image 15 using training data (e.g. DHR images in which cell nuclei and their geometric centers have been labelled.

[0072] Since, in a typical DHR image only relatively few pixels happen to lie at the geometric center of a cell nucleus, there is a large class imbalance between positive (cell nucleus center pixels) vs non-cell-nucleus center pixels a large amount of training data may be required to train a CNN used for nuclear center finder 16. An available set of training data may be augmented by creating altered training data as is known in the art. For example, a labelled DHR image that can be used as training data may be transformed (e.g. by one or more of scaling, rotation, tone mapping, geometric distortion, one or more morphological operations such as dilation and/or erosion etc.) to yield altered images that may be used as additional training data. Training may be performed in multiple rounds.

[0073] The inventors have found that creating altered training data by eroding nuclear boundaries can be particularly effective. For example, starting training of a CNN with annotated images (e,g, DHR images in which nuclei have been annotated by mask pixels, for example by human inspection of the DHR images) in the first round followed by multiple training rounds using ever more eroded versions of the masks are used as training data until only the center pixels remained was found to result in consistently successful nuclei center identification.

[0074] The inventors had access to training data sets showing normal cells, abnormal cells, and immune cells as well as training data sets depicting junk cells/debris/overlapping nuclei. These training data sets contain well over 2 million human annotated objects (See Ref. 22). Some of this training data was used in training the prototype system.

[0075] Generating training data for the prototype system was facilitated by in house developed tools which facilitate acquiring, reviewing and annotating DHR images. These tools include automated image cytometry systems for the scanning of cytology samples (Cervix, Lung, Oral)(see Refs 16-24). These systems are operable to automatically load slides and collect images of every object in best focus.

[0076] The tools also included software systems that are operable to analyze detected objects, automatically sort and classify them into: normal cells; abnormal cells; immune cells; junk cells; debris; and overlapping nuclei and present the detected objects for human review and classification. These tools are useful for building training data sets and also for final human review of any detected abnormal cells detected prior to signing off a slide as abnormal or normal (See Refs. 23,24). [0077] The tools include an interface that allows a user to trace boundaries of cell nuclei or other objects - for example by dragging a cursor along the boundary - as well as to annotate areas of overlap of different displayed objects (e.g. cell nuclei). [0078] The reproducibility of boundaries of cell nuclei drawn by different human annotators is not as high as one might expect. This disagreement likely comes from the difficulty in exactly tracing the boundary of a nucleus by hand, especially where a boundary being traced is for a nucleus in a complex cluster of nuclei. A training data set may be improved by including in training data training images in which the same cell nuclei have been traced by multiple different human reviewers and/or by refining traced images using software which adjusts the boundaries over a very limited range (e.g. 1or 2 pixels) to most closely follow the edge (rapid intensity change) of an object. For example, the drawn boundaries may be snapped to the strongest edge within a few pixels. These approaches or a combination of them may yield more consistent boundaries in training data sets and, in turn improved accuracy in a system trained using the training data sets.

[0079] Fig. 4 is a screen shot of a graphical user interface for an example tool useful for reviewing the segmentation/classification of 1000’s of nuclei rapidly. The controls along the bottom allow one to: page forward or back, jump to the end of images or the start, display all images or only those from a specific group, toggle the display of the nuclear boundaries and zoom in or zoom out.

[0080] The present technology may be integrated into a tool of the general type described for example in Refs 23 and 24 to provide automated segmentation of cell nuclei.

[0081] Figs 6A to 6C are respectively a section of a DHR image showing a cluster of cell nuclei, the section of the image marked to show boundaries of the cell nuclei in the cluster and a mask showing the boundaries only. The boundaries shown in Figs 6B and 6C may, for example be drawn by a human using a suitable tool (e.g. a tool as shown in Fig. 4). These boundaries may be included in training data for a CNN used to implement nuclear boundary finder 18. A loss function used in training the CNN may be based on differences between the boundaries shown in Fig. 6B and 6C from boundaries determined by the CNN used to implement nuclear center finder 18. [0082] Figs 7 A to 7C and 8A to 8C are similar to Figs. 6A to 6C but show different clusters of cell nuclei.

Example Results

[0083] Fig. 3A shows the results of a UNet trained to recognize the centers of nuclei. The light spots at the centers of the depicted nuclei are the geometric centers of the nuclei as determined by the UNet CNN. The success rate on a set of 28,000 training nuclei (with data augmentation) was found to be 95%-90% (depending upon human evaluator) which is much better than conventional ML algorithms on the same images. In the further developed prototype which was trained using a larger training data set segmentation accuracy on a 200K plus nuclei validation was 92% for all objects. Feature calculations and a binary classification tree were then applied to classify objects (as cell nuclei and other objects). After this classification, 99.2% of objects classified as nuclei were found to be well segmented cell nuclei regardless of size or shape. The classification tree weeded out (e.g. classified as not being nuclei) segmentation failures and objects too small and/or to light to be nuclei.

[0084] Fig. 3B shows example eroded nucleus masks as used to train the UNet to identify the geometric centers of nuclei depicted in DHR images. In Fig. 3B the nucleus masks are partially eroded. Training data may include images in which the nucleus masks are less eroded than as shown in Fig. 3B as well as images in which the nucleus masks are more eroded than as shown in Fig, 3B.

[0085] Figs. 3C and 3D show example results of the prototype system on two test sub images from much larger tissue microarray (TMA) images. These sub images include complex overlapping clusters of nuclei.

[0086] Fig. 3C shows a cluster of thirty cell nuclei all of which have similar absorption, pixel darkness). The prototype system correctly segmented all but three of the depicted nuclei. The first UNet (implementing nuclear center finder 16) missed two of the thirty nuclei indicated by arrows 31 and 32. The second UNet (implementing nuclear boundary finder 18) missed segments of the 15 ^th nucleus as indicated by arrow 33. The results in Fig. 3C demonstrate ~ 90% correct segmentation (~ as it is hard to tell if the nucleus labelled 12 is segmented correctly).

[0087] Fig 3D shows several clusters in which the absorption pixel darkness varies significantly over the depicted nuclei. In this image the prototype system also achieved ~ 90% accuracy in segmenting the nuclei.

[0088] Fig. 3D depicts approximately 97 nuclei. Of these, four nuclei as indicated by arrows (34A, 34B, 34C and 34D) were not correctly segmented. One nucleus was missed (indicated by arrow 35). Two false nuclear centers (indicated by arrows 36A and 36B) were incorrectly identified. Three objects that may not be cell nuclei (they are too small and/or too light in tone) were identified as indicated by arrows 37A, 37B and 37C.)

[0089] On a set 12,522 nuclei from TMA spots that were not included in training data used to train the UNet CNNs of the prototype the segmentation performed by the prototype had an accuracy in the range of 58-93% correct segmentation rate. The accuracy varied depending on the nature of the depicted nuclei. The segmentation accuracy for long thin stromal cell nuclei was about 58%. The segmentation accuracy for epithelial and immune cell nuclei was about 93%. The segmentation accuracy over the total set of TMA spots was about 84%. As discussed above, a subsequent prototype implementation in which much larger training sets were used to train nuclear center finder 16 and nuclear boundary finder 18 had even higher accuracy.

[0090] For cell types not well represented in the training set, the segmentation of new nuclei of that type was not as accurate as for cell types that were well represented in the training data.

[0091] Fig. 5 shows results of segmentation of cell nuclei in a DHR image of prostate tissue using the prototype system. The DHR image was of a thionin-stained prostate needle biopsy applied to an independent prostate section. The three inserts show some example errors. It is believed that the likelihood of such errors can be reduced by improved training of CNNs used to implement nuclear center finder 16 and/or nuclear boundary finder 18. In insert A the prototype system did not properly identify the entire boundary of a long thin nucleus. Errors of this type may be reduced or avoided by including sufficient examples of long thin nuclei in training data used to train CNNs in nuclear center finder 16 and/or nuclear boundary finder 18. Insert B shows over-segmentation of a cluster of nuclei. This can arise where nuclear center finder 16 over identifies a number of centers of nuclei in the nuclei cluster. In Fig. 5 the nuclear cluster associated with insert B is slightly out of focus. This out of focus condition may tend to make over segmentation more likely. Insert C demonstrates an error that is opposite to the error of insert B. In insert C nuclear center finder 16 is failing to detect some of nuclei present in the corresponding cluster. To make a system as described herein more resilient to errors of the types illustrated in inserts B and C, training data for nuclear center finder 16 and/or nuclear boundary finder 18 may include subsets of the training images that are highly enriched with annotated images similar to the errors observed that can be used as “finishing training sets” to improve the optimization of the networks’ weights for theses segmentation instances.

Example Applications

[0092] The present technology may be applied to determine locations of cells in tissue. In some embodiments this location information is combined with cell characterization information (e.g. morphologically based and/or immunohistochemistry (IHC) based characterization). This combination allows for cell type based cell-cell association quantification. This will enhance the fidelity of the intermediate representation (cell types, cell location, cell-cell associations) of the tissue. These intermediate representations can be used by subsequent ML/DL steps for tissue classification as well as for improving our understanding of the development of diseases such as cancers.

[0093] Some embodiments enable the creation of accurate spatial cell level representations of tissue based upon molecular single cell analysis of each cell in the tissue. This methodology may be scaled up to the entire tissue section level in a way that is amiable to high throughput clinical scale efforts.

[0094] Intermediate representations of the tissue can be based upon the cellular building blocks of the tissue itself and thus improve interpretability of the process. For example, nuclear center finder 16 operates to recognize the cells and structures within the cells that make up the tissue and where each is located. With this information one can then categorize the cells into various types based upon their morphological and molecular characteristics. These cell locations and characteristics can then be fed into a second CNN/ML algorithm to generate the final cell/tissue classification. This approach has the very large benefit that it allows for “Interpretability” of the intermediate representations used by the DL/CNN/ML process in the context of the large literature knowledge base of our current understanding of the neoplastic process and its interaction with the hosts tissues and immune system. [0095] The use of cell locations and characteristics allows for another large benefit the “Standardization” of the intermediate representation data (normalization of magnification, stain variation and other device specific effects using methods that have already exist in the digital pathology knowledge base) which should improve the generalizability of the DL results (6).

[0096] An advantage of at least some embodiments of the present technology is that intermediate representations created in processing DHR images may be more closely associated with physical properties of the tissue represented in the DHR images than is the case in other ML systems.

[0097] The foregoing examples have illustrated application of the present technology to segmenting cell nuclei in DHR images. It can be appreciated that the technology may be used for instance segmentation of other transparent or partially transparent overlapping objects of which cells and cell nuclei are examples. For example, the present technology may be applied to segmenting cells and/or cell nuclei in “medical images” which include: cytology images, cytopathology images, in vivo histology images (obtained by any modality) and histopathology images such as DHR images. [0098] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to herein, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e. , that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.

[0099] Embodiments of the invention may be implemented using specifically designed hardware, configurable hardware, programmable data processors configured by the provision of software (which may optionally comprise “firmware”) capable of executing on the data processors, special purpose computers or data processors that are specifically programmed, configured, or constructed to perform one or more steps in a method as explained in detail herein and/or combinations of two or more of these. Examples of specifically designed hardware are: logic circuits, application-specific integrated circuits (“ASICs”), large scale integrated circuits (“LSIs”), very large scale integrated circuits (“VLSIs”), and the like. Examples of configurable hardware are: one or more programmable logic devices such as programmable array logic (“PALs”), programmable logic arrays (“PLAs”), and field programmable gate arrays (“FPGAs”). Examples of programmable data processors are: microprocessors, digital signal processors (“DSPs”), embedded processors, graphics processors, math coprocessors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors in a control circuit for a device may implement methods as described herein by executing software instructions in a program memory accessible to the processors.

[0100] In typical applications, nuclear center finder 16 and nuclear boundary finder 18 are implemented in a programmed computer which includes a graphics processor unit (GPU) which is programmed by computer executable instructions to perform calculations for determining output from CNNs.

[0101] The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, non-transitory media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

Interpretation of Terms

[0102] Unless the context clearly requires otherwise, throughout the description and the claims:

• “comprise”, “comprising”, and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”;

• “connected”, “coupled”, or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof;

• “herein”, “above”, “below”, and words of similar import, when used to describe this specification, shall refer to this specification as a whole, and not to any particular portions of this specification;

• “or”, in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list;

• the singular forms “a”, “an”, and “the” also include the meaning of any appropriate plural forms. These terms (“a”, “an”, and “the”) mean one or more unless stated otherwise;

• “and/or” is used to indicate one or both stated cases may occur, for example A and/or B includes both (A and B) and (A or B);

• “approximately” when applied to a numerical value means the numerical value ± 10%;

• where a feature is described as being “optional” or “optionally” present or described as being present “in some embodiments” it is intended that the present disclosure encompasses embodiments where that feature is present and other embodiments where that feature is not necessarily present and other embodiments where that feature is excluded. Further, where any combination of features is described in this application this statement is intended to serve as antecedent basis for the use of exclusive terminology such as "solely," "only" and the like in relation to the combination of features as well as the use of "negative" limitation(s)” to exclude the presence of other features; and

• “first” and “second” are used for descriptive purposes and cannot be understood as indicating or implying relative importance or indicating the number of indicated technical features.

[0103] Words that indicate directions such as “vertical”, “transverse”, “horizontal”, “upward”, “downward”, “forward”, “backward”, “inward”, “outward”, “left”, “right”, “front”, “back”, “top”, “bottom”, “below”, “above”, “under”, and the like, used in this description and any accompanying claims (where present), depend on the specific orientation of the apparatus described and illustrated. The subject matter described herein may assume various alternative orientations. Accordingly, these directional terms are not strictly defined and should not be interpreted narrowly.

[0104] Where a range for a value is stated, the stated range includes all sub-ranges of the range. It is intended that the statement of a range supports the value being at an endpoint of the range as well as at any intervening value to the tenth of the unit of the lower limit of the range, as well as any subrange or sets of sub ranges of the range unless the context clearly dictates otherwise or any portion(s) of the stated range is specifically excluded. Where the stated range includes one or both endpoints of the range, ranges excluding either or both of those included endpoints are also included in the invention.

[0105] Certain numerical values described herein are preceded by "about". In this context, "about" provides literal support for the exact numerical value that it precedes, the exact numerical value ±5%, as well as all other numerical values that are near to or approximately equal to that numerical value. Unless otherwise indicated a particular numerical value is included in “about” a specifically recited numerical value where the particular numerical value provides the substantial equivalent of the specifically recited numerical value in the context in which the specifically recited numerical value is presented. For example, a statement that something has the numerical value of “about 10” is to be interpreted as: the set of statements:

• in some embodiments the numerical value is 10;

• in some embodiments the numerical value is in the range of 9.5 to 10.5; and if from the context the person of ordinary skill in the art would understand that values within a certain range are substantially equivalent to 10 because the values with the range would be understood to provide substantially the same result as the value 10 then “about 10” also includes:

• in some embodiments the numerical value is in the range of C to D where C and D are respectively lower and upper endpoints of the range that encompasses all of those values that provide a substantial equivalent to the value 10

[0106] Specific examples of systems, methods and apparatus have been described herein for purposes of illustration. These are only examples. The technology provided herein can be applied to systems other than the example systems described above. Many alterations, modifications, additions, omissions, and permutations are possible within the practice of this invention. This invention includes variations on described embodiments that would be apparent to the skilled addressee, including variations obtained by: replacing features, elements and/or acts with equivalent features, elements and/or acts; mixing and matching of features, elements and/or acts from different embodiments; combining features, elements and/or acts from embodiments as described herein with features, elements and/or acts of other technology; and/or omitting combining features, elements and/or acts from described embodiments. [0107] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any other described embodiment(s) without departing from the scope of the present invention.

[0108] Any aspects described above in reference to apparatus may also apply to methods and vice versa.

[0109] Any recited method can be carried out in the order of events recited or in any other order which is logically possible. For example, while processes or blocks are presented in a given order, alternative examples may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, simultaneously or at different times.

[0110] Various features are described herein as being present in “some embodiments”. Such features are not mandatory and may not be present in all embodiments. Embodiments of the invention may include zero, any one or any combination of two or more of such features. All possible combinations of such features are contemplated by this disclosure even where such features are shown in different drawings and/or described in different sections or paragraphs. This is limited only to the extent that certain ones of such features are incompatible with other ones of such features in the sense that it would be impossible for a person of ordinary skill in the art to construct a practical embodiment that combines such incompatible features. Consequently, the description that “some embodiments” possess feature A and “some embodiments” possess feature B should be interpreted as an express indication that the inventors also contemplate embodiments which combine features A and B (unless the description states otherwise or features A and B are fundamentally incompatible). This is the case even if features A and B are illustrated in different drawings and/or mentioned in different paragraphs, sections or sentences. [0111] It is therefore intended that the following appended claims and claims hereafter introduced are interpreted to include all such modifications, permutations, additions, omissions, and sub-combinations as may reasonably be inferred. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole.

References

1 . Evans AJ, Bauer TW, Bui MM, Cornish TC, Duncan H, Glassy EF, et al. US food and drug administration approval of whole slide imaging for primary diagnosis: A key milestone is reached and new questions are raised. Arch Pathol Lab Med. 2018 Apr 30;142(11):1383-1387.

2. Gertych A, Swiderska-Chadaj Z, Ma Z, Ing N, Markiewicz T, Cierniak S, et al. Convolutional neural networks can accurately distinguish four histologic growth patterns of lung adenocarcinoma in digital slides. Sci Rep. 2019 Feb 6;9(1):1483.

3. Chang HY, Jung CK, Woo JI, Lee S, Cho J, Kim SW, et al. Artificial intelligence in pathology. J Pathol Transl Med. 2019 Jan;53(1):1-12.

4. Kelly, C.J., Karthikesalingam, A., Suleyman, M. et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 17, 195 (2019). https://doi.Org/10.1186/s12916-019-1426-2

5. Holzinger, A., Biemann, C., Pattichis, C.S., Kell, D.B.: What do we need to build explainable Al systems for the medical domain? arXiv: 1712.09923 (2017)

6. Holzinger, A., et al.: Towards the augmented pathologist: challenges of explainable-AI in digital pathology. arXiv: 1712.06657 (2017)

7. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. Cham: Springer International Publishing; 2015. p. 234-241.

8. Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. UNet++: A Nested U-Net Architecture for Medical Image Segmentation. arXiv: 1807.10165 .

9. Abdolhoseini, M., Kluge, M.G., Walker, F.R. et al. Segmentation of Heavily Clustered Nuclei from Histopathological Images. Sc/ Rep 9, 4551 (2019). https://doi.Org/10.1038/S41598-019-38813-2 MacAulay C, Tezcan H, Palcic B. Adaptive color basis transformation. An aid in image segmentation. Anal Quant Cytol Histol. 1989 Feb;11 (1):53-58. MacAulay C, Palcic B. An edge relocation segmentation algorithm. Anal Quant Cytol Histol. 1990 Jun;12(3):165-171. Zarei N, Bakhtiari A, Gallagher P, Keys M, MacAulay C. Automated prostate glandular and nuclei detection using hyperspectral imaging. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). IEEE; 2017. p. 1028-1031. Zarei N, Bakhtiari A, Korbelik J, Carraro A, Keyes M, MacAulay C. Introducing an Interactive Method to Improve Digital Pathology Image Segmentation: Case Study on Prostate Cancer. Anal Quant Cytopathology Histpathol. 2017;39:1- 16. Enfield KSS, Martin SD, Marshall EA, Kung SHY, Gallagher P, Milne K, et al. Hyperspectral cell sociology reveals spatial tumor-immune cell interactions associated with lung cancer recurrence. J Immunother Cancer. 2019 Jan 16;7(1):13. Guerrero-Pena FA, Marrero Fernandez PD, Ing Ren T, Yui M, Rothenberg E, Cunha A. Multiclass weighted loss for instance segmentation of cluttered cells. 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE; 2018. p. 2451-2455. Palcic B, Garner DM, Beveridge J, Sun XR, Doudkine A, MacAulay C, et al. Increase of sensitivity of sputum cytology using high-resolution image cytometry: field study results. Cytometry. 2002 Jun 15;50(3): 168-176. Li G, Guillaud M, LeRiche J, McWilliams A, Gazdar A, Lam S, et al. Automated sputum cytometry for detection of intraepithelial neoplasias in the lung. Analytical Cellular Pathology. 2012;35(3):187-201. Macaulay C, Poh CF, Guillaud M, Michele Williams P, Laronde DM, Zhang L, et al. High throughput image cytometry for detection of suspicious lesions in the oral cavity. J Biomed Opt. 2012 Aug;17(8):086004-086001. Keyes M, Macaulay C, Hayes M, Korbelik J, Morris WJ, Palcic B. DNA ploidy measured on archived pretreatment biopsy material may correlate with prostate-specific antigen recurrence after prostate brachytherapy. I nt J Radiat Oncol Biol Phys. 2013 Aug 1 ;86(5):829-834. Chiu DHK, Guillaud M, Cox DD, Follen M, MacAulay C. [PDF] Quality Assurance System Using Statistical Process Control: An Implementation for Image Cytometry - Semantic Scholar, undefined. 2004; Ikeda N, MacAulay C, Lam S, LeRiche J, Payne P, Garner D, et al. Malignancy associated changes in bronchial epithelial cells and clinical application as a biomarker. Lung Cancer. 1998 Mar;19(3):161-166. Anderson G, Macaulay C, Matisic J, Garner D, Palcic B. The use of an automated image cytometer for screening and quantitative assessment of cervical lesions in the British Columbia Cervical Smear Screening Programme. Cytopathology. 1997 Oct;8(5):298-312. C Palcic B, Garner D, MacAulay C, Matisic J and Anderson D. Oncometrics Imaging Corp, and Xillix Technologies Corp.: Use of the Cyto-Savant in quantitative cytology. Acta Cytologica 40(1):67-72, 1996. Kamalov R, Haskins D, Guillaud M, Harrison A, Kemp R, Chiu D, Follen M, MacAulay C. A Java Application for Tissue Section Image Analysis. Computer Methods in Programs in Biomedicine, 77(2):99-113, 2005. Levenson R, Beechem J, McNamara G. Spectral imaging in preclinical research and clinical pathology. Stud Health Technol Inform. 2013;185:43-75. Erin A Marshall, Katey SS Enfield, Kevin W Ng, Martial Guillaud, Calum MacAulay, and Wan L Lam. Establishing a cell sociology platform for the assessment of targetable interactions to predict lung cancer outcome. IALC, 2019 World Lung conference, September 7-10, Barcelona, Spain. https://library.iaslc.org/conference- program?product_id=15&author=&category=&date=&am p;session_type=&session= &presentation=&keyword=Enfield&cme=undefined& ; E- Guillaud M, Zhang L, Poh C, Rosin M, MacAulay C. Potential use of Quantitative Tissue Phenotype to Predict Malignant Risk for Oral Premalignant Lesions. Cancer Research 68(9):3099-3107, 2008. MacAulay C, Keyes M, Hayes M, Lo A, Wang G, Guillaud M, Gleave M, Fazli L, Korbelik J, Collins C, Keyes S, Palcic B. Quantification of large scale DNA organization for predicting prostate cancer recurrence. Cytometry A. 2017 Dec;91 (12):1164-1174 Guillaud M, Ye Q, Leung S, Carraro A, Harrison A, Hayes M, Nichol A, Keyes M. Large-scale DNA organization is a prognostic marker of breast cancer survival. Med Oncol. 2017 Dec 6;35(1):9. doi: 10.1007/s12032-017-1068-1. PMID: 29214466. Varduhi Yeghiazaryan and Irina Voiculescu* . Family of boundary overlap metrics for the evaluation of medical image segmentation. J Med Imaging (Bellingham). 2018 Jan; 5(1): 015006. Published online 2018 Feb 19. doi: 10.1117/1.JMI.5.1.015006 Wang, Shidan, Ruichen Rong, Donghan M. Yang, Junya Fujimoto, Shirley Yan, Ling Cai, Lin Yang et al. "Computational staining of pathology images to study the tumor microenvironment in lung cancer." Cancer Research 80, no. 10 (2020): 2056-2066. http://lce.biohpc.swmed.edu/maskrcnn/ Aarno Oskar Vuola, Saad Ullah Akram, Juho Kannala, Mask-RCNN and U-net Ensembled for Nuclei Segmentation. arXiv: 1901.10170v1 [cs.CV] 29 Jan 2019