Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
SYSTEM FOR AUTOMATED AND QUANTITATIVE ASSESSMENT OF METASTATIC SPINAL STABILITY
Document Type and Number:
WIPO Patent Application WO/2024/098147
Kind Code:
A1
Abstract:
Computational tools for automated and quantitative assessment of metastatic spinal stability. Computational image analysis methods, that may include deep learning are used to automatically quantify spinal geometry/quality and disease burden from medical images (CT and MR) that can be used to predict patient outcomes of vertebral stability / risk of fracture. Two tools are provided: 1) An interpretable enhanced automated SINS (spinal instability neoplastic scoring) tool which calculates improved quantitative individual image-based parameters used in SINS and calculates stability through a similar scoring system; 2) A tool that directly estimates vertebral stability/fracture risk based on 3D imaging (with or without non-imaging patient specific clinical data). The automated tools may be used to improve clinical workflows (through automation) and accuracy, sensitivity, and specificity of vertebral stability prediction that can aid clinicians in better directing treatment to optimize patient outcomes. The tools are particularly useful in patients undergoing planning for stereotactic body radiation therapy (SBRT) because of the accessibility of clinical imaging and the high likelihood (14%) of vertebral compression fracture (VCF) and associated mechanical instability following this procedure.

Inventors:
HARDISTY MICHAEL RAYMOND (CA)
WHYNE CARI (CA)
SAHGAL ARJUN (CA)
KLEIN GEOFFREY (CA)
MARTEL ANNE (CA)
Application Number:
PCT/CA2023/051491
Publication Date:
May 16, 2024
Filing Date:
November 08, 2023
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
SUNNYBROOK RES INST (CA)
International Classes:
A61B5/00; A61B5/055; A61B6/03; G06T7/00; G16H30/40; G16H50/30
Attorney, Agent or Firm:
HILL & SCHUMACHER (CA)
Download PDF:
Claims:
THEREFORE WHAT IS CLAIMED IS: 1. A method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into A) a computational algorithm, or B) a machine learning algorithm trained on a dataset of spinal imaging, to determine mechanical stability and/or fracture risk, the algorithm is configured to i) calculate mechanical stability and/or fracture risk by the steps of calculating features from a feature extractor algorithm and/or image processing algorithm, in particular based on user input; ii) combining said features within a computational decision tool iii) combining said features in step ii) with non-imaging patient specific features to yield predictions on mechanical stability and/or fracture risk, using an optimization scheme based on said imaging data of the patient’s spine. 2. A method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into A) a computational algorithm, or B) a machine learning algorithm trained on a dataset of spinal imaging, to determine mechanical stability and/or fracture risk, the algorithm configured to i) calculate mechanical stability and/or fracture risk by the steps of calculating features from a feature extractor algorithm such as a feature extractor backbone network; ii) combining said features within a computational decision tool comprising convolutional layers and vertebrae specific features derived from convolutional arms; and/or extracting the latent features of each vertebrae; and iii) combining said features combined with said vertebrae specific features in step ii) and with non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on said imaging data of the patient’s spine. 3. The method according to claim 1 or 2, wherein said imaging data of the patient’s spine is CT and/or MR imaging data. 4. The method according to claims 1 or 2, wherein said dataset of spinal imaging is comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. 5. The method according to claims 1 or 2, wherein said feature extractor algorithm is a feature extractor backbone network, which is any one of a ResNet (Residual Network)+Feature Pyramid Network (FPN), Transformer, Inception, Convolutional Neural Network (CNN), Fully Connected Neural Network (FCNN), Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), GIST, U-Net, AlexNet, Visual Geometry Group Network (VGGNet), GoogLeNet, Radiomics, Wavelet Transforms, Fourier Transforms, Edge Detection Algorithms, Region-based Methods, Filter Banks, Texture Descriptors, Color Spaces Transformations, Geometric Descriptors, Binary Pattern Descriptors, Keypoint Descriptors, Fast Retina Keypoint (FREAK)), Ridge Detection, Scale-space Representation, Skeletonization, Principal Component Analysis (PCA), and Singular Value Decomposition (SVD). 6. The method according to claim 5, wherein said Edge Detection Algorithms are any one of Sobel Algorithms, Prewitt Algorithms and Canny Algorithms; said Region-based Methods are any one of Watershed algorithms and Superpixel Segmentation; said Filter Banks are Gabor filters); said Texture Descriptors are any one of Haralick Texture Descriptors, and Tamura Texture Descriptors; said Color Spaces Transformation are any one of Red, Green, Blue (RGB) to Hue, Saturation, Lightness (HSL) and Hue, Saturation, Value (HSV) Transformations; said Geometric Descriptors are any one of Zernike moments Geometric Descriptors and Fourier descriptors Geometric Descriptors); said Binary Pattern Descriptors and any one of Completed Local Binary Patterns (CLBP) and Dominant Local Binary Patterns (DLBP); said Keypoint Descriptors are any one of Binary Robust Invariant Scalable Keypoints (BRISK) and Fast Retina Keypoint (FREAK); said Ridge Detection uses eigen values of the Hessian matrix; and said Scale-space Representation is any one of Gaussian pyramids and Laplacian pyramids. 7. The method according to claims 1, or 2, wherein said feature extractor algorithm is a feature extractor backbone network, which is a ResNet 50 +FPN network. 8. The method according to claims 1 or 2, wherein the features calculated through a feature extractor algorithm and extracted latent features of each vertebrae are deep features. 9. A method for determining tumorous involvement of the posterolateral elements of the spine from imaging, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine involvement of the posterolateral elements of the spine; c) combining the segmented and localized vertebrae and calculating vertebra specific features from a feature extractor backbone network to determine posterolateral involvement or by retraining a feature exactor backbone network that is specific to classification of the posterolateral involvement, then using the features extracted to classify using any one of a classification branch, or a machine learning classifier, or a statistical classifier if there is posterolateral involvement, present or not, the extent (bilateral, unilateral), and the location (facet, costovertebral joint, pedicle), wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with labeling of posterolateral involvement. 10. The method according to claim 9 wherein the vertebra specific features calculated from a feature extractor backbone network are deep features. 11. A method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine mechanical stability and/or fracture risk, the machine learning algorithm configured to calculate spinal instability neoplastic score elements, said elements including bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement, c) wherein calculating each of the elements of bone lesions, spinal alignment, vertebral body collapse and posterolateral involvement includes segmenting and/or localizing the vertebrae by the steps of calculating features from a feature extractor backbone network and combining said features with convolutional layers, graph networks, and vertebrae specific features derived from convolutional arms and/or extracting the latent features, combining said features with said vertebrae specific features and non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with information about patient outcomes; wherein for calculating the element of bone lesions, using the segmented and localized vertebrae in a histogram-based analysis or segmenting the lesions using an automated approach to determine a volume and type of tumor tissue present within each metastatically involved vertebra; wherein for calculating the spinal alignment element, using the segmented and localized vertebrae to calculate Cobb angle in the coronal and sagittal planes; wherein for calculating the vertebral body collapse, combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral bodies in the 3D image, and c) uses intact vertebral volumes in adjacent levels, or d) uses a statistical model, machine learning model or neural network to estimate the percentage of vertebral body collapse; and wherein posterolateral involvement is determined by the method according to claim 8 12. The method for assessing mechanical stability and/or fracture risk in the metastatically involved spine according to claim 11, wherein the automated approach segmenting the lesions includes using any one of thresholding, clustering, region growing, level-set methods, active contour, variational methods, graph portioning methods, simulated annealing, watershed, model- based, classify features extracted from the image and neural network-based segmentations. 13. The method according to claims 10, 11 or 12, wherein said dataset of spinal imaging is comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. 14. The method according to claims 10, 11, 12 or 13, wherein said elements of bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with clinical pain and vertebral level to yield an automated spinal instability neoplastic score.

15. The method according to claim 11, wherein said elements of bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with clinical pain and other patient specific features and vertebral level to yield an Automated SINS score which predicts mechanical stability and/or fracture risk. 16. The method according to claim 11, wherein said elements bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with quantification of any one or all of musculoskeletal health, clinical pain and other patient specific features and vertebral level to yield an Automated Enhanced SINS type assessment which predicts mechanical stability and/or fracture risk better than the existing SINS. 17. The method according to claim 16, wherein musculoskeletal health is calculated by the steps of combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral body trabecular centrum in the 3D image, calculates the density of the vertebral body trabecular bone, the bone density distribution and further segments muscle volume with another convolutional network and calculates the volume, density and density distribution of the muscle. 18. The method according to claim 16, wherein the musculoskeletal health includes bone quality, bone mass, bone density, muscle quality, and muscle size.

19. The method according to claim 11, wherein the features calculated through a feature extractor algorithm and extracted latent features of each vertebrae are deep features.

Description:
SYSTEM FOR AUTOMATED AND QUANTITATIVE ASSESSMENT OF METASTATIC SPINAL STABILITY FIELD The present disclosure relates to a system that uses deep learning to automatically quantify spinal architecture (geometry/quality) and disease burden from 3D medical images (non limiting examples being CT and MR) that can be used to predict patient outcomes of vertebral stability / risk of fracture. More particularly, the present disclosure provides an interpretable automated spinal instability neoplastic scoring (SINS) tool which calculates improved individual image-based parameters used in SINS and calculates stability through a similar scoring system; and also provides an image-based tool that directly estimates vertebral stability/fracture risk. BACKGROUND Approximately 70% of cancer patients are found to have skeletal metastases during post-mortem examinations with the spine being the most common site [1]. These metastases cause pain, mobility issues, mechanical instability, and neurologic compromise with some lesions leading to vertebral compression fracture (VCF). With advances in image-guided external beam therapy and robotics, Stereotactic Body Radiation Therapy (SBRT) has emerged as a highly effective way to locally treat these tumours [1]. However, VCF remains a common complication following SBRT with rates ranging from 10% to 40% post-procedure [2]. VCF risk and mechanical stability assessment are currently evaluated through the manual review of imaging and combining this information with patient factors. The Spine Instability Neoplastic Score (SINS) is a standardized method used to assess mechanical instability, that has been widely adopted but shows limited efficacy (Hazard Ratio=0-5.6) [3] for predicting VCF. The existing scientific literature does not target computational medical image analysis with most studies that assess fracture risk and vertebral mechanical stability, using patient specific factors (Age, Primary Tumour location, BMI, etc.) and/or SINS assessment with limited fidelity. This invention describes an AI-enabled medical image analysis-based tool for the assessment of VCF risk and spinal mechanical stability. SUMMARY Disclosed herein is a method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining 3D imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine mechanical stability and/or fracture risk, the machine learning algorithm configured to i) calculate mechanical stability and/or fracture risk by the steps of calculating deep features from a feature extractor backbone network; ii) combining the deep features with convolutional layers, graph networks, and vertebrae specific features derived from convolutional arms; and/or extracting the latent deep features of each vertebrae; iii) combining the deep features combined with the vertebrae specific features in step ii) and with non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on the 3D imaging data of the patient’s spine. The 3D imaging data of the patient’s spine may be CT and/or MR imaging data. The dataset of spinal imaging may be comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. The feature extractor backbone network may be a ResNet network, or a transformer network or inception network or use fully connected layers, uses principal component analysis or radiomics or linear discriminant analysis, independent component analysis, t-distributed Stochastic Neighbor Embedding to mention some non-limiting examples. The feature extractor backbone network may be a ResNet 50 network. The present disclosure provides a method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining 3D imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine mechanical stability and/or fracture risk, the machine learning algorithm configured to calculate spinal instability neoplastic score elements, said elements including bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement; c) wherein calculating each of the elements of bone lesions, spinal alignment, vertebral body collapse and posterolateral involvement includes segmenting the vertebrae and localizing the vertebrae by the steps of calculating deep features from a feature extractor backbone network and combining the deep features with convolutional layers and vertebrae specific features derived from convolutional arms, combining the deep features with the vertebrae specific features and non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on the 3D imaging data of the patient’s spine combined with information about patient outcomes; wherein for calculating the element of bone lesions, using the segmented and localized vertebrae in a histogram-based analysis to determine a volume and type of tumor tissue present within each metastatically involved vertebra; wherein for calculating the spinal alignment element, using the segmented and localized vertebrae to calculate Cobb angle in the coronal and sagittal planes; wherein for calculating the vertebral body collapse, combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral bodies in the 3D image, and uses intact vertebral volumes in adjacent levels to estimate the percentage of vertebral body collapse; and wherein posterolateral involvement is determined by the steps of combining the segmented and localized vertebrae and calculating deep features from a feature extractor backbone network and combining said deep features with graph networks, convolutional layers, and vertebrae specific features derived from convolutional arms, combining said deep features with the vertebrae specific features with dense layers to determine posterolateral involvement, wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with labeling of posterolateral involvement. The dataset of spinal imaging may be comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. The elements bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement may be combined with clinical pain and vertebral level to yield an automated spinal instability neoplastic score. The elements bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with clinical pain and other patient specific features and vertebral level to yield an Automated SINS score which predicts mechanical stability and/or fracture risk. A further understanding of the functional and advantageous aspects of the disclosure can be realized by reference to the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the AI-enabled medical image analysis-based tools for the assessment of VCF risk and spinal mechanical stability will now be described, by way of example only, with reference to the drawings, in which: FIGURE 1 shows an overview diagram for the system for automated and quantitative assessment of metastatic spinal stability disclosed herein. FIGURE 2 is an expanded view of the non-limiting VertDetect model 14 of FIGURE 1. DETAILED DESCRIPTION Various embodiments and aspects of the disclosure will be described with reference to details discussed below. The following description and drawings are illustrative of the disclosure and are not to be construed as limiting the disclosure. The drawings are not necessarily to scale. Numerous specific details are described to provide a thorough understanding of various embodiments of the present disclosure. However, in certain instances, well- known or conventional details are not described in order to provide a concise discussion of embodiments of the present disclosure. As used herein, the terms, “comprises” and “comprising” are to be construed as being inclusive and open ended, and not exclusive. Specifically, when used in this specification including claims, the terms, “comprises” and “comprising” and variations thereof mean the specified features, steps or components are included. These terms are not to be interpreted to exclude the presence of other features, steps or components. As used herein, the term “exemplary” means “serving as an example, instance, or illustration,” and should not be construed as preferred or advantageous over other configurations disclosed herein. As used herein, the terms “about” and “approximately”, when used in conjunction with ranges of dimensions of particles, compositions of mixtures or other physical properties or characteristics, are meant to cover slight variations that may exist in the upper and lower limits of the ranges of dimensions so as to not exclude embodiments where on average most of the dimensions are satisfied but where statistically dimensions may exist outside this region. Unless otherwise specified, the terms “about” and “approximately” mean plus or minus 25 percent or less. It is not the intention to exclude embodiments such as these from the present disclosure. It is to be understood that unless otherwise specified, any specified range or group is as a shorthand way of referring to each and every member of a range or group individually, as well as each and every possible sub-range or sub-group encompassed therein and similarly with respect to any sub-ranges or sub-groups therein. Unless otherwise specified, the present disclosure relates to and explicitly incorporates each and every specific member and combination of sub-ranges or sub-groups. As used herein, the term "on the order of", when used in conjunction with a quantity or parameter, refers to a range spanning approximately one tenth to ten times the stated quantity or parameter. Definitions As used herein, the phrase “machine learning algorithm” refers to algorithms that ‘learn’ how to perform their task by being ‘trained’ on datasets to perform the task by optimizing the performance by changing internal elements that affect the output of the algorithms. The internal elements are varied such that the output generated fits what is seen in the dataset. Here we use machine learning algorithms that learn how to represent imaging data with feature extracting units and how to use these latent representations, combined with patient specific factors to make predictions that are consistent with the training dataset. As used herein, the phrase “deep features” refers to quantitative values that are extracted from an image that are the result of deep learning network architectures. Network architectures with greater than one (1) layer are referred to as deep learning architectures. The deep learning networks that used herein have multiple layers. The networks used herein learn how to represent images. A representation of an image which contains its features is referred to as a feature representation. In this disclosure deep learning networks are used to learn a feature representation. Because we learn the representation with deep learning, we refer to the features within as deep features. As used herein, the phrase “feature extractor backbone network” refers to a network or part of a network where latent features are used, or shared, between different subsequent tasks. This can refer to a network that after training generates a feature representation for the entire image volume input that is useful for the tasks required. We use the features extracted by this network as the backbone for the predictions and determinations of the entire network. As used herein, the phrase “convolutional layers” refers to structures that perform a series of filtering operations on the input data, producing an output. Convolutional layers perform discrete convolutions operations on some input where the weights are the kernel to be learned. Convolutional layers can also have a bias which are learned weights. Many convolutional layers can create filters that derive output from an entire image or only small aspects of it. As used herein, the phrase “convolutional arms” refers to a series of convolutional, pooling and fully connected layers. Pooling layers reduce the dimensionality or number of voxels/pixels output, combining information from many input voxels/pixels to produce outputs with fewer voxel/pixels. Fully connected layers have connections and weights from each input node (pixel/voxel, prior layer output) to every node in the fully connected layer. The use of arms denotes that there are several arms that do computation in series and are specifically trained to perform specific tasks. As used herein, the phrase “vertebrae specific features” refers to features or aspects that characterize individual vertebrae in the imaging data. This is in contrast to features or aspects that describe the whole image, whole spine, or the patient. The features can be human understandable, such as vertebral size, vertebral bone density, or they may be the result of machine learning computations such as those through the convolutional arms that produce features that are descriptive of specific vertebrae but are not intuitively understandable. As used herein, the phrase “non-imaging patient specific features” refers to patient demographic data including age, sex, body mass index, primary tumor location and histology, prior treatments (e.g., systemic drug therapies and/or local treatments (such as radiation and dose received to give some non- limiting examples), number of vertebral metastases, presence of other metastases in the body, existing fractures, presence of pain. As used herein, the phrase “training is performed by an optimizer” means that we adjust the weights within the machine learning algorithm during a phase called training where the input data has been labeled specifying its output state. The weights within the machine learning algorithm are adjusted in order to fit the desired output. An optimizer varies the weights in an iterative fashion by changing the weights then evaluates the output against the desired output from the training dataset. The optimizer continues varying the weights until the output from the machine learning algorithm matches the output in the training dataset. This disclosure describes AI enabled tools for automated and quantitative assessment of metastatic spinal stability. In this, we use deep learning to automatically quantify spinal geometry/quality and disease burden from medical images (CT and MR) that can be used to predict patient outcomes of vertebral stability / risk of fracture. Two tools have been developed: 1) An interpretable automated SINS (spinal instability neoplastic scoring) tool which calculates improved individual image-based parameters used in SINS and calculates stability through a similar scoring system; 2) A tool that directly estimates vertebral stability/fracture risk based on 3D imaging (with or without non-imaging patient specific clinical data). The automated tools may be used to improve clinical workflows (through automation) and accuracy, sensitivity, and specificity of vertebral stability prediction that can aid clinicians in better directing treatment to optimize patient outcomes. The tools are particularly useful in patients undergoing planning for stereotactic body radiation therapy (SBRT) because of the accessibility of clinical imaging and the high likelihood (14%) of vertebral compression fracture (VCF) and associated mechanical instability following this procedure. As such, the present inventors have created a data set of imaging and associated clinical data from patients who have undergone SBRT for spinal metastases at the Sunnybrook Odette Cancer Centre, and have trained our algorithms against this retrospective data. An additional clinical research dataset was also used consisting of temporal CT imaging data of patients with spinal metastases, as well as an open dataset (VerSe [12]). The technology and trained algorithms would be of use to clinicians planning treatment for patients with spinal metastases, and companies that create technology for planning SBRT procedures. The algorithms are novel in that they quantify bio- markers from clinical imaging in an automated quantitative manner using innovative multitask architectures. They replace currently used clinical manual qualitative scoring which is poor at predicting fracture risk, with automated and improved predictions. A non-limiting exemplary implementation of the system for automated and quantitative assessment of metastatic spinal stability disclosed herein will now be described. Part Numbers for features in FIGURES 1 and 2. 10 - Overall flow diagram 12, 50 - Spinal imaging 14 - VertDetect model which includes three main branches; the detection, classification, and segmentation branches. The detection branch detects each vertebra in a 3D CT scan by determining both its vertebral body centroid location and placing a bounding box around the whole vertebra. The classification branch utilizes shared information between each neighbouring vertebra to determine what vertebra are present in the input CT scan. The segmentation branch semantically segments vertebra that are positively detected from the classification and the detection branches. 16 - Segmented and localized vertebrae 18 - Level of vertebra to be evaluated 20 - Bone Lesion Element: histogram-based calculation of tumor type and volume 22 - Spinal alignment Element: Cobb angles calculated in the coronal and sagittal planes 24 - Vertebral Collapse Element 26 - Posterolateral Involvement Element 28 - deep learning features directly generated by the machine learning network 30 - Patient specific non-imaging-based factors (i.e. age, sex, BMI, current medications, systemic and local therapies etc.) 36 - AutoSINS score 38 - Fracture risk /vertebral stability score 52 - Backbone architecture consisting of a 3D ResNet-50 and Feature Pyramid Network (FPN). P1-P5 refer to feature maps generated from the convolution backbone. Feature maps generated from this backbone are then used in further downstream tasks. 54, 56, 60-65 - 3x3x3 Convolutions + Batch Normalization +ReLU 58 - 1x1x1 Convolutions + Batch Normalization +ReLU 67-69, 128 – 1x1x1 Convolutions 70 - Offset calculation 72 - Bounding Box sizes 74 – Gausian Heatmap 76, 114 - Concatenations 80 - Bounding boxes 82 - Initial Bounding Box processing 84 - Binary Classification block 86 - Binary Vertebra Classification 88 – Cropped regions of interest for each individual vertebra 100,102, 108, 110, 120, 122, 124, 126 - 3x3x3 Convolutions + Instance Normalization +ReLU 104, 122 - 1x1x1 Convolutions + Instance Normalization +ReLU 106 - 2x2x2 Convolution Transpose + Instance Normalization 130 - Segmentation of each individual vertebrae within the image 130 – Gaussian heat maps used to localize each vertebrae Full 3D end-to-end vertebrae instance segmentation model VertDetect is a novel architecture inspired by previous work by Yi et al. [7], Mask R-CNN [3] and FCOS [5][6], and modifies layers for 3D use. VertDetect takes as input in Figure 2 a 3D image 50 containing a depiction of vertebrae. It adjusts first layers of ResNet backbone 52 in Figure 2 for improved high-resolution feature map. The architecture uses Gaussian heatmaps 74 shown in Figure 2 for the segmentation branch to help identify which vertebrae to segment. It applies Graph Convolutional Network (GCN) layers for vertebrae classification, allowing vertebrae identification to be influenced by neighboring predictions. It uses linear scheduling aids in the gradient descent for localization stages by combining a variant focal loss from CornerNet with mean-square-error (MSE) loss. Detection Branch The detection branch utilizes an anchorless approach. The largest resolution feature map from the convolution backbone (P1) is convoluted by three convolutions, the first two having kernels 54 and 56 in Figure 2 of 3x3x3, and the last kernal 58 having 1x1x1 shown in Figure 2. The first of these three convolutions 54 and 56 also compress the P1 feature maps to 128 features to reduce the memory impact. The resulting feature map (after 54, 56 and 58) is then sent to three separate blocks of convolutions: 1) for the heatmap, 62, 65 and 69 shown in Figure 2, 2) for bounding box sizes, 61, 64 and 68, and 3) for offset predictions, 60, 63 and 67. Heatmap 74 in Figure 2 The heatmaps have C channels, where each channel corresponds to a vertebra (i.e., channel 0 is C1, channel 1 is C2, etc.). Therefore, each centroid and bounding box is implicitly defined in the heatmap predictions. The maxima of each channel’s heatmap is used to provide the centroid predictions for each vertebra. Ground truth 3D Gaussian heatmaps are generated using ground truth centroid points in a down-sampled space. Offset Sizes 70 in Figure 2 The heatmaps, and therefore the predicted centroids, are in a downsampled state. Following the work by Yi et al. [7] a predicted offset coordinate is used to shift the predicted centroids to compensate for potential differences during upsampling. Bounding Box Sizes 72 in Figure 2 Both the heatmap outputs and the offsets are used to determine the centroid of an object in the full resolution image. The bounding boxes sizes are used to determine the bounding box surrounding the centroid point. Classification Branch Each heatmap channel corresponds to an individual vertebra, the information is combined with the offset and bounding box size to create bounding box 80 (Figure 2) candidates for each vertebrae. However, these heatmaps do not consider neighboring vertebra. The purpose of the classification branch is to leverage the information between neighboring vertebra to improve overall classification and detection. Initial bounding box processing in step 82 (Figure 2) uses centroid locations predicted from the heat maps and then combines these predictions with features maps from the convolutional backbone within the binary classification step 84 in Figure 2. In the binary classification block step the features from the convolutional backbone are cropped to the regions centered about the heatmap predicted centroids. The features from the convolutional backbone are then sent through two (2) convolution layers and then sent through a graph convolutional network layer that uses shared information between vertebrae to make the best binary classification of vertebrae 88 (see Figure 2). Segmentation Branch The segmentation branch semantically segments positively detected vertebra. Before extracting the positively detected regions found in the previous branches, unnormalized Gaussian heatmaps 131 in Figure 2 are concatenated in step 76 in Figure 2 with the full resolution input image. This Gaussian is centered about the predicted full-resolution centroid locations with a standard deviation of four (4). As the vertebra segmentation step is for the full vertebra, bounding boxes contain neighboring vertebra due to the necessity to include posterior elements. The bounding boxes are defined and used to crop regions of interest with step 86 in Figure 2 from the input image and feature maps from the convolutional backbone. The Gaussian heatmap ensures that the model focuses on the correct vertebra during semantic segmentation. The features derived from the convolutional backbone and Gaussian input image, after being convolved as in steps 100, 102, 106, 108, 110 of Figure 2 and unsampled in steps 104 and 112 which are merged in step 114 of Figure 2. Subsequently, the obtained feature map is passed through the concluding convolutional layers in steps 120, 122, 124, 126 and 128 shown in Figure, to generate the segmentation predictions 130. The tool that directly estimates vertebral stability/fracture risk based on 3D imaging provides a method for assessing mechanical stability and fracture risk in the metastatically involved spine. The method comprises a) obtaining 3D imaging data of a patient’s spine 12 shown in Figure 1; b) inputting the imaging data into a machine learning algorithm in step 14 of Figure 1 to determine mechanical stability and fracture risk, the machine learning algorithm configured to calculate mechanical stability and fracture risk by the steps of i) calculating deep features from a feature extractor backbone network 28 shown in Figure 1; ii) combining these deep features with convolutional layers, graph networks and vertebrae specific features derived from convolutional arms; and/or extracting the latent deep features of each vertebrae; and iii) combining the deep features in step ii) and with non-imaging patient specific features 30 in Figure 1 with dense layers to yield predictions on mechanical stability and fracture risk, wherein training is performed by an optimizer based on the 3D imaging data of the patient’s spine. Referring again to Figure 1, the 3D imaging data 12 may be acquired by imaging systems such as computed tomography (CT) or magnetic resonance imaging (MRI). Figure 1 also shows the tool that provides an interpretable automated SINS (spinal instability neoplastic scoring) tool 36 which calculates improved individual image-based parameters used in SINS and calculates stability through a similar scoring system and provides a method for assessing mechanical stability and fracture risk in the metastatically involved spine, comprising: a) obtaining 3D imaging data of a patient’s spine 12; b) inputting the imaging data into a machine learning algorithm to determine mechanical stability and fracture risk in step 38 in Figure 1, the machine learning algorithm configured to calculate spinal instability neoplastic score elements, said elements including step 20 in Figure 1 that quantifies bone lesions involvement, spinal alignment in step 22 in Figure 1, vertebral body collapse in step 24 in Figure 1 and, posterolateral involvement in step 26 in Figure 1; c) wherein calculating each of the elements of bone lesions, spinal alignment, vertebral body collapse and posterolateral involvement includes segmenting the vertebrae, 16 in Figure 1 and localizing the vertebrae, 18 in Figure 1 in step 14 of Figure 1 and further elaborated on in all of Figure 2 by the steps of calculating deep features for vertebrae (which may from a feature extractor backbone network and combining said deep features with convolutional layers and vertebrae specific features derived from convolutional arms) 14, combining said deep features 28 with said vertebrae specific features and non-imaging patient specific features 30 with dense layers to yield predictions on mechanical stability and fracture risk, wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with information about patient outcomes; wherein for calculating the element of bone lesions, the segmented and localized vertebrae 16 are used in a histogram-based analysis step 20 to determine a volume and type of tumor tissue present within each metastatically involved vertebra; wherein for calculating the spinal alignment element step 22, using the segmented and localized vertebrae to calculate Cobb angle in the coronal and sagittal planes; wherein for calculating the vertebral body collapse step 24, combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral bodies in the 3D image, and uses intact vertebral volumes in adjacent levels to estimate the percentage of vertebral body collapse; and wherein posterolateral involvement step 26 is determined by the steps of extracting deep features specific to a vertebra from the feature extractor backbone network and predicting posterolateral involvement, wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine and posterolateral involved labels. Datasets used in training Prospective Cohort Study Data was prospectively collected, including patient factors and serial CT imaging (every 4 months) with a consistent imaging protocol from T4-L5 was collected from 100 patients with vertebral metastases over 1 year. 1000 vertebrae within the CT imaging were semi-automatically segmented (vertebral centrum, osteoblastic, mixed and osteolytic tumour burden). Sunnybrook SBRT Spine Patients Five hundred (500) patients (1500 lesions) treated with Spine SBRT for metastatic vertebral lesions at Sunnybrook hospital were prospectively added to a clinical database (tumour location, SINS assessment, primary tumour location, receptor status, previous treatments, etc.) with retrospective collection and association of clinical imaging (MRI (T1 and T2), CT) and SBRT related planning data (RT struct, RT dose, RT Plan, MRI to CT registration object). VerSe Dataset [12] Open dataset of 141 training, 120 validation and 113 testing samples of 3D CT images. Each scan includes full vertebrae segmentations, including vertebral body level labelling, and vertebral body centroid locations. Dataset consists of C1 to L5 vertebrae and includes rare T13 and L6 transitional vertebrae. Dataset does not include images with implants or diseases. Step 1: Vertebral Detection, Vertebral Segmentation and Feature Extraction Input: Computed tomography imaging Output: Deep Learning Features, Vertebral location, Vertebral identity, Vertebral segmentation The VertDetect model 14 of Figure 1 used herein can be broken down into three main branches; the detection, classification, and segmentation branches. The detection branch detects each vertebra in a 3D image by determining both its vertebral body centroid location and placing a bounding box around the whole vertebra. The model predicts a location heatmap, offset (sub voxel location adjustment to heatmap location), and bounding box for each vertebrae in the CT volume. The classification branch utilizes shared information between each neighbouring vertebra to determine what vertebra are present in the input CT scan (vertebral identity). The segmentation branch semantically segments vertebra that are positively detected from the classification and the detection branches. The overall architecture of VertDetect 14 of Figure 1 is shown in Figure 2. A 3D ResNet-50 [9] and Feature Pyramid Network (FPN) [10] shown at 52 in Figure 2 act as a backbone architecture. Feature maps from this backbone are then used in further downstream tasks. The ResNet-50 +FPN is used to extract features from the CT image that are used for the vertebral detection, classification, segmentation tasks in this step, and for tasks in the later steps. The network was trained using data from both the 2019 and 2020 VerSe datasets [12]. The model was trained with a composite loss. Heatmap The heatmaps have C channels, where each channel corresponds to a vertebra (i.e., channel 0 is C1, channel 1 is C2, etc.). Therefore, each centroid and bounding box is implicitly defined in the heatmap predictions. The maxima of each channel’s heatmap is used to provide the centroid predictions for each vertebra. Ground truth 3D Gaussian heatmaps are generated using ground truth centroid points in a down-sampled space. A downsampled space was used to be consistent with the P1 output of the FPN (2x downsampling from input image size). The ground truth Gaussian distributions were constructed ( ^^^^)^^^^^^ ^ ^^ ^(^^^ )^ such that the peak max value was 1.0, ^ = ^ ^ ^ ^^^ where cx, cy and cz are the ground truth centroid coordinates, σ is the standard deviation of the distribution, and x, y, and z are points in space. In each channel of the heatmap, there is a single centroid point of interest and the rest is the background. To account for the large imbalance between foreground and background a variant focal loss was used as [4][8][7]: 1 ^ ^^^^^ = − ^ (1 − ^^)^ log(^^) , if ^^ = 1 ^ (1 − ^^)^^^ ^ log(1 − ^^) , else where i is the i-th index, p is the predicted heatmap, y is the ground truth, and N is the number of centroids. This differs from the original focal loss [4] by reducing the impact of predicted centroids that are close to the ground truth compared to predictions that are further when yi^[0, 1), with the (1-yi) β term. It was found that a 2x down-sampled heatmap showed superior performance compared to the 4x used in [7] likely due to better separation between neighbouring vertebrae. However, due to the size of the heatmaps and the class imbalance convergence of the variant focal loss was difficult. To improve the rate of convergence the variant focal loss was combined with mean square error (MSE) loss ^ ^^^^ = ^^ ^^^^^ + ^^ ^^^ where a and b are such that ^ = ^ ^^ ^ and ^ = ^^^^ ^^ given some epoch ^ and some threshold epoch ^’. This epoch when using the combined MSE and variant focal loss to using only the variant focal loss. The constant λ is a scaling term to address the large numerical difference between the variant focal loss and MSE functions and is = ^^^ ^ ^^^^^ ^ ^ ^^ ^ + ^ ^ ^^ , and γ=1e-4. After epoch ^’, ^ = ^ ^^^^^ . The overall ^^^^ , when ^ = 0 ^ ^ ^ ^^ ^ . During channel. This differs for validation as is discussed later in the Post Processing section. Offset Sizes The heatmaps, and therefore the predicted centroids, are in a downsampled state. Even if the predicted centroids match the ground truth centroids in the downsampled state they may not match in the full resolution after upsampling due to rounding errors which can occur if the ground truth centroid is not divisible by the downsampling amount. Following the work by Yi et al. [7] a predicted offset coordinate is used to shift the predicted centroids to compensate for potential differences during upsampling: ^ ^ = ^ ^^,^ ^ − ^ ^^,^ ^ ^ , ^^,^ ^ − ^ ^^,^ ^ ^ , ^^,^ ^ − ^ ^^,^ ^ ^^ where i is the i-th for the i-th centroids. The brackets ⌊ ⌋ are the floor operation and n is the downsampling size. A smooth-L1 loss is used to regress the offsets: ^ ^^^^^^ = ^^^^^^ℎ ^^ (^ ^ − ^ ^ ) , where ^ ^ and ^ ^ are Bounding Box Sizes Both the heatmap outputs and the offsets are used to determine the centroid of an object in the full resolution image. The bounding boxes sizes are used to determine the bounding box surrounding the centroid point. The coordinates of the bounding box for the i-th vertebra (bbi) is: ^^ ^ = [^ ^ , ^ ^ ,^ ^ ,^ ^ , ^ ^ , ^ ^ ] ^ ^ = ^ ^,^ − ^ ^ ^ ^ = ^ ^,^ + ^ ^ ^ ^ = ^ ^,^ − ^ ^ ^ ^ = ^ ^,^ + ^ ^ where ci is the full-scale for vertebra i, and s is the sizes. The subscripts for the sizes s correspond to left (l), right (r), posterior (p), anterior (a), inferior (i), and superior (s). All six are necessary as the centroids defined here are vertebral body centers and not centers of the object’s encompassing bounding box so mirrored sizes (ie, left and right) are not symmetric. Bounding box sizes are regressed using a log Intersection-over-Union (IoU) loss function [5][6][11]: ^ ^^ = − log(^^^) 1 ^ ^ ^^ = − log ( ^ ∩ ^ where ^ ^ is the predicted ground truth. This loss was used as opposed to mean-absolute-error (MAE) or smooth-L1 as this can allow for box sizes to be modified slightly if the centroid prediction is slightly shifted (ie, larger left than right if the centroid prediction is slightly offset). Classification Branch Each heatmap channel corresponds to an individual vertebra. However, these heatmaps do not consider neighbouring vertebra. If a channel that corresponds to a T3 vertebra shows a high probability of existing in the scan, then the probability for neighbouring T2 and T1 should reflect it. The purpose of the classification branch is to leverage the information between neighbouring vertebra to improve overall classification and detection. A RoiAlign [3] generates C feature maps of size 7x7x7 from P1 cropping and resampling regions focused on the centroid locations. The resampled feature maps are then sent through a 7x7x7 followed by a 1x1x1 convolution, both with ReLU activation. To leverage the shared information of each vertebra Graph Convolutional Network (GCN) layers are used. The resulting features are then sent through three GCN layers resulting in Cx1 logits that correspond to a vertebra being present in the scan or not. As the class of each vertebra is implicitly defined the heatmap’s channel a binary classification is used to determine if that channel and predicted centroid correspond to a positive detection. A vertebra is positively detected if sigmoid(^ ^ ) > 0.5 where ^ ^ is the logit score for the i-th vertebra of the classification branch output. classification branch is trained using binary cross entropy, ^ ^^^^^ = ^^^ 1 ^ ^^^^^ = − ^^ log(^ ) + (1 − ^ ) log(1 − ^ ) ^ ^ ^ ^ ^ where ^ ^ is the binary value specifying if the i-th vertebra is active, ^ ^ is the predicted probability vertebra i being active, and N is all possible vertebra. Classification of the vertebrae is further refined by a graph pos- processing step. The predicted heatmaps can have multiple local clusters and these local clusters can incorrectly correspond to the neighbouring vertebrae. To address the local maxima that can occur in the heatmap predictions, a post- processing method is used to determine which maxima from the local clusters are correct. A non-maximum suppression (NMS) is first used through a max- pooling layer to select the top k candidates from each channel of the heatmap predictions. These k candidates are then filtered by Euclidean distances to ensure no neighbours from the same local cluster exist resulting in k′ candidates for each heatmap channel. The logits for each k′ candidate from the heatmap predictions are then averaged with the logits from the classification branch for each corresponding vertebrae to scale each based on the probability of that particular vertebra existing in the scan. The resulting k′ candidates are constructed in a graph as seen in 5 based on two rules. The first rule is that the axial position of the node above must be greater than the node below to ensure the correct vertebrae ordering is enforced. The second is that the Euclidean distance between any two connected nodes must be greater than 3 voxels to ensure that no two nodes from the same physical locations are used. The weights for each node are taken as the averaged logits. The centroid location of each vertebra is then determined by solving the graph from T (top) to B (bottom) by determining the longest path and therefore the path with the highest sum of averaged logits; this process is also solved from B to T and the path with the largest sum taken as the correct path. The model was first trained with only the heatmap output for 500 epochs and will be referred to as the self-initialization. After the self-initialization, all outputs were predicted and all loss functions were used. During the first 100 epochs after the self-initialization (between epochs 501 and 600 from the overall start) the model was trained using the ground truth bounding boxes for the segmentation task. This was done so to ensure the model learned to accurately segment vertebrae without being negatively affected by inaccurate bounding box predictions as the model was converging. After epoch 100 after the self-initialization (epoch 600 from the overall start) and when ^ ^^^^ < 1.0 the model would transition to using the predicted bounding boxes. This would continue until model completion. Overall, the model was trained for 1500 epochs. Step 2: SINS Element Bone Lesion Input: Vertebral Segmentation from Step 1, Computed tomography imaging. Output: osteoblastic lesion volume, osteolytic lesion volume, tumour classification (osteolytic, osteoblastic, mixed). The bone lesion element is quantified and automated by calculating the amount of osteolytic and osteoblastic tissue in the vertebrae using the vertebral body segmentation. A vertebra is cropped based on the vertebral body centroid location predictions from Step 1 and a secondary convolutional network is used to segment the vertebral body. The vertebral detection model described in Step 1 was combined with another deep learning network (a U-Net, convolutional neural network (CNN)) yielding a vertebral body segmentation model of the trabecular centrum. A histogram-based analysis is used to define osteolytic and osteoblastic tumour involvement. Specifically, it calculates the distribution of normal bone tissue from trabecular centrum of the vertebral bodies in the non- diseased vertebrae in the spinal image. It then isolates osteolytic disease within the vertebral bodies using the threshold mean-standard deviation, and isolates osteoblastic disease using mean + 2 x standard deviation. The vertebra is then classified as osteolytic, osteoblastic or mixed based on the presence of osteolytic or osteoblastic disease found. The volume of osteoblastic and osteolytic tissue delineated is used to quantify the tumour involvement. The algorithm was developed and tested against the patient cohort study dataset and the Sunnybrook Spine SBRT Spine patient dataset. Step 3: SINS Element Spinal Alignment The vertebral localization and segmentation were used from Step 1. The Cobb angle was calculated in both the coronal and sagittal planes from the gradient of a spline curve generated through the vertebral body centroids. The angles calculated were validated against manual measurements of cobb angles made in the sagittal and coronal planes. Manual Cobb angle measurement (local to the vertebrae of interest and over the whole spine region (cervical, thoracic, lumbar) were taken on a subset of the Sunnybrook Spine SBRT treated patients with clinical indication of malalignment. Measurements were made by an orthopedic spine surgeon receiving fellowship training with consultation from a fellowship trained staff spine surgeon based on the CT imaging available for Spine SBRT planning. Step 4: SINS Element Vertebral Body Collapse Vertebral body collapse was calculated using a combination of deep learning-based methods. The vertebral detection model described in Step 2 was combined with another deep learning network (a U-Net, convolutional neural network (CNN)) for segmentation of the vertebral body centrums. In this, the CNN was trained to segment the vertebral body with unfractured vertebrae from metastatically involved spines from the patient cohort study dataset. Vertebral body collapse was quantified utilizing the CNN for vertebral body centrum segmentation developed in Step 1. Using this algorithm, segmentations were generated for each collapsed vertebra of interest and for the 4 adjacent non-collapsed vertebrae (2 proximal and 2 distal). Locations of all vertebrae were determined automatically using the vertebral detection model described in Step 1. An estimate of each fractured vertebra’s intact volume was estimated using a polynomial fit of the 4 adjacent vertebral body volumes. Comparison of the estimated intact volume with the volume of the fractured vertebral body of interest yielded the % collapse. A linear model was used to evaluate collapse in vertebrae located at the edges of the image; this was most often the case for the 5 th lumbar vertebra (the most inferior vertebra in the spinal column). The automated module was found to yield results consistent with semi-quantitative clinical assessment of vertebral collapse. Step 5: SINS Element Posterolateral Involvement The neural network from Step 1 was retrained to detect posterolateral tumour involvement. Features maps from the ResNet 50 +FPN backbone were combined with additional convolutional layers trained to classify vertebrae. The complete network for this step was pretrained to identify the tumour involvement in the entire vertebrae, it was retrained to identify CT scans with unilateral or bilateral involvement of the posterior and lateral elements of the tumour involved vertebrae. Training of the network was done using the Spine SBRT patient dataset with CT scored by radiation oncology fellows and staff as having no involvement, unilateral, or bilateral involvement of the posterolateral spine elements. Step 7: Automate Quantitative calculation of musculoskeletal health biomarkers. The vertebral localization and segmentation were used from Step 1. Lumbar vertebrae (L1-L5) vertebral bodies were segmented using the vertebral detection model described in Step 2 and combined with another deep learning network (a U-Net, convolutional neural network (CNN)) for segmentation of the vertebral body centrums. A psoas muscle region from the midpoints between the L2/L3 and L4/L5 intervertebral discs was isolated based on the vertebrae locations and curve of the spine found in Step 1. The psoas muscle region was segmented by using another deep learning network (a U-Net, convolutional neural network (CNN)) for segmentation. derived from the vertebrae locations from Step 1. Specific biomarkers extracted were bone density, bone density distribution of lumbar vertebral bodies and the volume and density of the major psoas muscles. Muscles segmented for use in biomarker calculations will be defined by the region of the spine imaged and may include: Iliocostalis lumborum, Iliocostalis thoracis, Iliocostalis cervicis, Longissimus thoracis, Longissimus cervicis, Longissimus capitis, Spinalis thoracis, Spinalis cervicis, Spinalis capitis, Semispinalis thoracis, Semispinalis cervicis, Semispinalis capitis, Multifidus, Rotatores brevis, Rotatores longus, Interspinales, Intertransversarii, Levatores costarum, Serratus posterior superior, Serratus posterior inferior, Quadratus lumborum, Psoas major, Psoas minor, Rectus capitis posterior major, Rectus capitis posterior minor, Obliquus capitis superior, Obliquus capitis inferior. Step 8: Automated Quantitative SINS The current standard for assessing mechanical stability in vertebrae with tumour involvement is the Spine Instability Neoplastic Score (SINS). SINS is a qualitative score based on six criteria: mechanical pain of the patient, vertebral level, lesion type (osteolytic, osteoblastic, or mixed), existing vertebral body collapse, malalignment, and presence of posterior element involvement. Total scores are binned into three categories: 0-6 (stable), 7-12 (possible instability), and 13-18 (instability) [6]. This work automates and enhances SINS by using the quantitative aspects of the image-based biomarkers allowing finer distinctions. For example, current SINS scores Cobb angle of 15° and 30° as having deformity, and 15% or 45% vertebral body collapse as the equivalent. The approach disclosed herein allows variables to be treated as continuous, potentially improving their predictive power. Step 9: Vertebral Fracture Risk Predictive models (random forest classifiers, support vector machines) combined patient factors (sex, age, presence of pain), treatment factors (dose and fractions), and the image-based quantitative biomarkers from the previous steps were utilized to predict vertebral fractures secondary to SBRT. The new prediction model performance was evaluated against SINS with generalizability assessed using 5-fold cross-validation and demonstrated that quantitative imaging-based biomarkers improve the accuracy to predict fractures. The present inventors have also used the deep learning model developed in Step 1 to directly predict vertebral fracture risk from images without calculating the intermediate image-based biomarkers. The detection model described above shares imaging features between its different sub-tasks and this framework was leveraged similarly to extract features regarding the spine’s metastatic involvement. Furthermore, the detection model’s implementation utilizes information from the vertebrae treated and the entire spine at once. These shared features were used to make predictions about fracture risk while using both vertebrae and spine-specific features. To summarize, the in an embodiment the present disclosure provides a method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into A) a computational algorithm, or B) a machine learning algorithm trained on a dataset of spinal imaging, to determine mechanical stability and/or fracture risk, the algorithm is configured to i) calculate mechanical stability and/or fracture risk by the steps of calculating features from a feature extractor algorithm and/or image processing algorithm, in particular based on user input; ii) combining said features within a computational decision tool iii) combining said features in step ii) with non-imaging patient specific features to yield predictions on mechanical stability and/or fracture risk, using an optimization scheme based on said imaging data of the patient’s spine. In an embodiment there is provided a method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into A) a computational algorithm, or B) a machine learning algorithm trained on a dataset of spinal imaging, to determine mechanical stability and/or fracture risk, the algorithm configured to i) calculate mechanical stability and/or fracture risk by the steps of calculating features from a feature extractor algorithm such as a feature extractor backbone network; ii) combining said features within a computational decision tool comprising convolutional layers and vertebrae specific features derived from convolutional arms; and/or extracting the latent features of each vertebrae; and iii) combining said features combined with said vertebrae specific features in step ii) and with non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on said imaging data of the patient’s spine. In an embodiment the imaging data of the patient’s spine is CT and/or MR imaging data. In an embodiment the dataset of spinal imaging is comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. In an embodiment the feature extractor algorithm is a feature extractor backbone network, which is any one of a ResNet (Residual Network)+Feature Pyramid Network (FPN), Transformer, Inception, Convolutional Neural Network (CNN), Fully Connected Neural Network (FCNN), Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), GIST, U-Net, AlexNet, Visual Geometry Group Network (VGGNet), GoogLeNet, Radiomics, Wavelet Transforms, Fourier Transforms, Edge Detection Algorithms, Region-based Methods, Filter Banks, Texture Descriptors, Color Spaces Transformations, Geometric Descriptors, Binary Pattern Descriptors, Keypoint Descriptors, Fast Retina Keypoint (FREAK)), Ridge Detection, Scale-space Representation, Skeletonization, Principal Component Analysis (PCA), and Singular Value Decomposition (SVD). In an embodiment the Edge Detection Algorithms are any one of Sobel Algorithms, Prewitt Algorithms and Canny Algorithms; the Region-based Methods are any one of Watershed algorithms and Superpixel Segmentation; the Filter Banks are Gabor filters; the Texture Descriptors are any one of Haralick Texture Descriptors, and Tamura Texture Descriptors; the Color Spaces Transformation are any one of Red, Green, Blue (RGB) to Hue, Saturation, Lightness (HSL) and Hue, Saturation, Value (HSV) Transformations; the Geometric Descriptors are any one of Zernike moments Geometric Descriptors and Fourier descriptors Geometric Descriptors); the Binary Pattern Descriptors and any one of Completed Local Binary Patterns (CLBP) and Dominant Local Binary Patterns (DLBP); the Keypoint Descriptors are any one of Binary Robust Invariant Scalable Keypoints (BRISK) and Fast Retina Keypoint (FREAK); the Ridge Detection uses eigen values of the Hessian matrix; and the Scale-space Representation is any one of Gaussian pyramids and Laplacian pyramids. In an embodiment the feature extractor algorithm is a feature extractor backbone network, which is a ResNet 50 +FPN network. In an embodiment the features calculated through a feature extractor algorithm and extracted latent features of each vertebrae are deep features. In an embodiment there is provided a method for determining tumorous involvement of the posterolateral elements of the spine from imaging, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine involvement of the posterolateral elements of the spine; c) combining the segmented and localized vertebrae and calculating vertebra specific features from a feature extractor backbone network to determine posterolateral involvement or by retraining a feature exactor backbone network that is specific to classification of the posterolateral involvement, then using the features extracted to classify using any one of a classification branch, or a machine learning classifier, or a statistical classifier if there is posterolateral involvement, present or not, the extent (bilateral, unilateral), and the location (facet, costovertebral joint, pedicle), wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with labeling of posterolateral involvement. In this embodiment the vertebra specific features calculated from a feature extractor backbone network are deep features. In an embodiment there is provided a method for assessing mechanical stability and/or fracture risk in the metastatically involved spine, comprising: a) obtaining imaging data of a patient’s spine; b) inputting the imaging data into a machine learning algorithm trained on a dataset of spinal imaging to determine mechanical stability and/or fracture risk, the machine learning algorithm configured to calculate spinal instability neoplastic score elements, said elements including bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement, c) wherein calculating each of the elements of bone lesions, spinal alignment, vertebral body collapse and posterolateral involvement includes segmenting and/or localizing the vertebrae by the steps of calculating features from a feature extractor backbone network and combining said features with convolutional layers, graph networks, and vertebrae specific features derived from convolutional arms and/or extracting the latent features, combining said features with said vertebrae specific features and non-imaging patient specific features with dense layers to yield predictions on mechanical stability and/or fracture risk, wherein training is performed by an optimizer based on said 3D imaging data of the patient’s spine combined with information about patient outcomes; wherein for calculating the element of bone lesions, using the segmented and localized vertebrae in a histogram-based analysis or segmenting the lesions using an automated approach to determine a volume and type of tumor tissue present within each metastatically involved vertebra; wherein for calculating the spinal alignment element, using the segmented and localized vertebrae to calculate Cobb angle in the coronal and sagittal planes; wherein for calculating the vertebral body collapse, combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral bodies in the 3D image, and a) uses intact vertebral volumes in adjacent levels, or b) uses a statistical model, machine learning model or neural network to estimate the percentage of vertebral body collapse; and wherein posterolateral involvement is determined by the method according to claim 8 In this embodiment the automated approach segmenting the lesions includes using any one of thresholding, clustering, region growing, level-set methods, active contour, variational methods, graph portioning methods, simulated annealing, watershed, model-based, classify features extracted from the image and neural network-based segmentations. In this embodiment the dataset of spinal imaging is comprised of CT and/or MR imaging that includes a clinical cohort of patients with spinal metastases. In this embodiment the elements of bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with clinical pain and vertebral level to yield an automated spinal instability neoplastic score. In this embodiment the elements of bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with clinical pain and other patient specific features and vertebral level to yield an Automated SINS score which predicts mechanical stability and/or fracture risk. In this embodiment the elements bone lesions, spinal alignment, vertebral body collapse, posterolateral involvement are combined with quantification of any one or all of musculoskeletal health, clinical pain and other patient specific features and vertebral level to yield an Automated Enhanced SINS type assessment which predicts mechanical stability and/or fracture risk better than the existing SINS. The musculoskeletal health is calculated by the steps of combining the segmented and localized vertebrae with another convolutional network that segments and calculates the volumes of the vertebral body trabecular centrum in the 3D image, calculates the density of the vertebral body trabecular bone, the bone density distribution and further segments muscle volume with another convolutional network and calculates the volume, density and density distribution of the muscle. The musculoskeletal health includes bone quality, bone mass, bone density, muscle quality, and muscle size. The features calculated through a feature extractor algorithm and extracted latent features of each vertebrae are deep features.

REFERENCES [1] C.-L. Tseng et al., “Spine Stereotactic Body Radiotherapy: Indications, Outcomes, and Points of Caution.,” Glob. spine J., vol.7, no.2, pp.179–197, Apr.2017, doi: 10.1177/2192568217694016. [2] G. Maccauro, M. S. Spinelli, S. Mauro, C. Perisano, C. Graci, and M. A. Rosa, “Physiopathology of Spine Metastasis,” Int. J. Surg. Oncol., vol.2011, pp.1–8, 2011, doi: 10.1155/2011/107969. [3] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in Proceedings of the IEEE International Conference on Computer Vision, Oct. 2017, pp.2980–2988, doi: 10.1109/ICCV.2017.322. [4] T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal Loss for Dense Object Detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol.42, no.2, pp. 318–327, 2020, doi: 10.1109/TPAMI.2018.2858826. [5] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional one-stage object detection,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, vol.2019-Octob, pp.9626–9635, doi: 10.1109/ICCV.2019.00972. [6] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: A Simple and Strong Anchor- free Object Detector,” IEEE Trans. Pattern Anal. Mach. Intell., pp.1–1, 2020, doi: 10.1109/tpami.2020.3032166. [7] J. Yi et al., “Object-Guided Instance Segmentation with Auxiliary Feature Refinement for Biological Images,” IEEE Trans. Med. Imaging, vol.40, no.9, pp.2403–2414, 2021, doi: 10.1109/TMI.2021.3077285. [8] H. Law and J. Deng, “CornerNet: Detecting Objects as Paired Keypoints,” Int. J. Comput. Vis., vol.128, no.3, pp.642–656, 2020, doi: 10.1007/s11263- 019-01204-1. [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp.770–778, doi: 10.1109/CVPR.2016.90. [10] T.-Y. Lin et al., “(FPN) Feature Pyramid Networks for Object Detection,” Cvpr,2017, doi: 10.1109/CVPR.2017.106. [11] J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, “UnitBox: An advanced object detection network,” in MM 2016 - Proceedings of the 2016 ACM Multimedia Conference, 2016, pp.516–520, doi: 10.1145/2964284.2967274. [12] A. Sekuboyina et al., “VERSE: A Vertebrae labelling and segmentation benchmark for multi-detector CT images,” Medical Image Analysis, vol.73. 2021, doi: 10.1016/j.media.2021.102166.