System and Method of Diagnostics from Hand-Drawn Image Analysis

INVENTOR

David H. Nguyen

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application takes priority from Provisional App. No. 63/373,028, filed on August 19, 2022, which is incorporated herein by reference.

BACKGROUND

[0002] The human hand and the human visual system are very complex; thus, many neurological diseases and conditions have an effect on tasks requiring hand-eye coordination. Hand drawings and handwriting analysis are frequently used to diagnose medical conditions such as Parkinson’s disease, multiple sclerosis, stroke, Lewy Body dementia, traumatic brain injury, developmental coordination disorder, or other conditions affecting the visual or motor system. These same tests may also be used to identify a normal state of health within the spectrum of normality - for example, visuospatial memory, visuospatial constructional ability, processing speed, fine motor skills, and so on. For purposes of the present disclosure, to simplify the discussion, “medical status” will mean either a medical condition or a normal variation in functioning.

[0003] For example, a clock drawing test (i.e. simply asking the patient to draw a clock face) may be used to diagnose Alzheimer’s disease or Parkinson’s disease. Fig. 1 shows the difference between clocks drawn by a healthy patient, a patient with Alzheimer’s disease, and a patient with Parkinson’s disease. A spiral drawing test, as shown in Fig. 2, can be used to diagnose Parkinson’s disease, essential tremor, or dystonia. A graphic sequence test, as shown in Fig. 3, can be used to diagnose frontal lobe dysfunction or damage. Other drawing tests include a pentagon drawing test, and the Rey Osterrieth Complex Figure Test, as shown in Fig. 4, which is used for assessing visuospatial recall memory, visuospatial recognition memory, response bias, processing speed, and visuospatial constructional ability. [0004] Handwriting can also be used for diagnostic purposes. For example, analyzing the shape of individual letters can be used to assess children for autism. Analyzing the overall size of the handwriting can be used to diagnose Parkinson’s Disease, since it involves micrographia (decrease in handwriting size and letter spacing).

[0005] Another area in which an image may be useful for diagnostic purposes is in analyzing eye gaze patterns. For example, there arc significant differences in the eye gaze patterns of people with autism versus neurotypical people when looking at a human face. A recorded eye gaze pattern may be useful to diagnose autism. Also, eye gaze patterns may be useful for diagnosis of eye movement disorders such as gaze palsies or nystagmus.

[0006] Historically, a medical professional would diagnose the patient by simply looking at the drawing or writing. However, this method is highly subjective and requires the medical professional to have a lot of experience. Thus, machine learning is increasingly used to assist a medical professional in making the diagnosis. However, it is very resource-intensive to analyze the drawing directly.

[0007] A need exists for processing a drawing in order to make machine -learning analysis more efficient, accurate, and less resource-intensive.

SUMMARY OF THE INVENTION

[0008] In brief, the method of the present invention comprises receiving a graphical image from a patient, and inputting the graphical image into a computing device comprising a processor and memory. The computing device then overlays a coordinate grid over the graphical image, wherein the coordinate grid comprises a reference entity and at least one gridline. In an embodiment, the coordinate grid is a linear grid and the reference entity is a line; in another embodiment, the coordinate grid is a radial grid (extending over 360 degrees, 180 degrees, or any other range) and the reference entity is a point. Tire computing device then detennines all the intersection points between each gridline and the graphical image, and measures the distance from each intersection point to the reference entity. The computing device then creates a transform function, wherein the first coordinate is the index of a gridline and the second coordinate is the distance from the reference entity to the intersection point on that gridline.

[0009] Once the transform function is created, a decomposition transform is performed on the transform function. The decomposition transform can be a Fast Fourier Transform, a Fourier Transfonn, a Laplace Transfonn, a Z-transfonn, a Slant Transform, fractal analysis, or any other transform that can create a cluster of components based on a function. The cluster of components can be a frequency spectrum or any other group of numerical components. The cluster of components can later be processed to obtain processed data. A trained algorithm can then analyze the processed data to diagnose the patient with a medical status, and the diagnostic information is then displayed on a display device.

[0010] The graphical image can be a spiral drawn by a patient, a patient’s handwriting, a clock face drawn by a patient, a composite of lines, polygons, and curves drawn by a patient, or an image generated by a patient’s eye gaze pattern or body movement pattern. In an embodiment, the graphical image is processed before being analyzed, wherein the processing can be removing part of the image, removing any smudges or stray marks, or extracting part of the image. For a spiral, the processing step may involve dividing the spiral into sectors and removing any unneeded sectors of the spiral, or dividing the spiral into sectors, removing any unneeded sectors, and then removing all except one of the curved lines remaining in the sector. For a clock face, the processing step may involve isolating the clock edge, hands, and numbers apart from each other.

[0011] The medical status that can be diagnosed with the present invention includes normal variations in functioning as well as medical conditions. In an embodiment, the analysis of eye gaze pattern can be used to diagnose autism, gaze palsy, or nystagmus. In an embodiment, the analysis of body movement may be used to diagnose stroke or epilepsy. Tire analysis of images drawn by the patient can also be used to diagnose essential tremor, stroke, heart attack, multiple sclerosis, Parkinson’s disease, Lewy body dementia, frontotemporal dementia, vascular dementia, nerve injury, traumatic brain injury, epilepsy, and psychosis. It may also be used to diagnose normal variations in function such as visuospatial recall memory, visuospatial recognition memory, response bias, processing speed, and visuospatial constructional ability.

[0012] In an embodiment, once the cluster of components is obtained, it is further processed to create a numerical index or to separate it into at least two components or to obtain other processed data.

[0013] After the processed data is obtained, it is then fed into a trained algorithm to analyze it and determine whether it correlates to a medical status. The algorithm may be specifically trained on a training data set that includes similar processed data and known outcomes. The algorithm may be an artificial neural network, a decision tree, a support vector machine, a regression analysis, a Bayesian network, logistic regression, or any other similar module. The algorithm may deliver a binary diagnosis or the likelihood of a medical status existing.

LIST OF FIGURES

[0014] Fig. 1 shows an illustration of the clock-drawing test.

[0015] Fig. 2 shows an illustration of the spiral drawing test.

[0016] Fig. 3 shows an illustration of the graphic sequence test.

[0017] Fig. 4 shows an illustration of the Rey Osterrieth Complex Figure Test.

[0018] Fig. 5 is ahigh-level flowchart of an embodiment of the method of the present invention.

[0019] Fig. 6 shows a spiral used as a sample image for demonstrating an embodiment of the method of the present invention.

[0020] Fig. 7A shows a linear grid overlaid over the sample image in an embodiment of the present invention.

[0021] Fig. 7B shows a radial grid overlaid over the sample image in an embodiment of the present invention.

[0022] Fig. 8 shows the distances between the intersection points and the reference line on a linear grid overlaid over the sample image in an embodiment of the present invention. [0023] Fig. 9 shows a plot of the transform function in an embodiment of the present invention.

[0024] Fig. 10 shows the input and output of a Fast Fourier Transform on a sample image.

[0025] Fig. 11 shows two sample spiral drawings and a plot of the Fast Fourier Transform results for each one in an embodiment of the present invention.

[0026] Fig. 12 shows a sample handwriting analysis and a plot of the decomposition transform for autism diagnosis in an embodiment of the present invention.

[0027] Fig. 13 shows a sample handwriting analysis and a determination of the average size of the handwriting in an embodiment of the present invention.

[0028] Fig. 14 shows a sample handwriting analysis and a determination of letter spacing in the handwriting in an embodiment of the present invention.

[0029] Fig. 15 shows an analysis of an eye-tracking pattern in an embodiment of the present invention for diagnosing autism.

[0030] Fig. 16 shows an analysis of body movement tracking in an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0031] The following disclosure focuses on the analysis of an image consisting of straight or curved lines. The lines may be connected or disconnected. While the image does not have to be exclusively composed of straight or curved lines, for the present invention to work, it has to intersect a straight gridline at distinct points. The image may be generated by a patient bydrawing or writing, or may be generated by eye gaze tracking or motion tracking. A non-limiting list of examples of images that may be analyzed by the present invention is: spirals, clocks, polygons, circles, handwriting, eye gaze patterns, motion tracking patterns.

[0032] The system and method of the present invention are used for identifying a patient’s medical status. As discussed above, “medical status” incorporates medical conditions as well as normal variations in functioning. Medical conditions that may be diagnosed by using the present invention include, but are not limited to, essential tremor, stroke, heart attack, multiple sclerosis, Parkinson’s disease, Lewy body dementia, Frontotemporal dementia, vascular dementia, nerve injury, traumatic brain injury, epilepsy, and psychosis. The present invention may also be applied to eye gaze patterns to diagnose autism, gaze palsy, or nystagmus, and to body movement patterns to diagnose stroke or epilepsy. Finally, the present invention may be used for evaluating normal variations in functioning, for example in assessing visuospatial recall memory, visuospatial recognition memory, response bias, processing speed, and visuospatial constructional ability.

[0033] The method of the present invention requires a computing device that comprises a processor, a memory, an input device, and a display device; in an embodiment, it also requires a server to which the computing device is connected via a communication module. The input device is preferably a scanner for scanning an image into the computing device. The processor and memory are configured to process the image and perform the analysis steps on the image - overlaying a grid on top of the image, perform a Linearized Compressed Polar Coordinates Transform to obtain a LCPC function, and perform a decomposition transform on the resulting function to obtain a cluster of components. The cluster of components may be processed further to obtain an index number, or may be analyzed as is. Either the processor and memory or the server may then use machine learning to analyze the cluster of components or the index to determine if it meets certain diagnostic criteria.

[0034] An embodiment of the method of the present invention is described below. The image provided is used solely as an illustrative example and is not meant to limit the disclosure in any way. The method of the present invention is applicable to any image.

[0035] A high-level flowchart of an embodiment of the method of the present invention is shown as Fig. 5. Initially, an image is received from the patient 700. The image may be drawn by the patient and then scanned into the computing device, may be directly inputted into the computing device via a stylus and tablet or a similar input device, or may be generated by cyc-gazc tracking or motion tracking. Eye gaze tracking may be performed via smart glasses, smartphones, or eye- tracking cameras such as the ones used on computers or car/truck dashboards. Motion tracking may be performed via smartwatches, smart rings, smart bracelets, or smart body braces or other wearable devices. Regardless of how the image is obtained, the end result is an image comprising straight or curved linear features. Fig. 6 shows a piece of a spiral used as a sample image to demonstrate the method of the present invention.

[0036] As an optional step, the image may be processed 710 before the analysis is performed. The processing is performed by the processor and memory on the computing device. For example, for a spiral drawing, the spiral may be split in half to enable easier analysis, or only one turn of the spiral may be isolated, or only a sector of the spiral may be isolated. The sector may be a quarter spiral (i.e. a 90° angle), or any other angle. As another example, a part of the image may be extracted and the rest of the image deleted. As another example, ink smudges or other artifacts may be removed.

[0037] A grid is then overlaid on the image 720, as shown in Figs. 7A and 7B. The grid may be linear or radial. In the example shown in Fig. 7A, the grid is linear and gridlines are spaced every 5 pixels. A reference line 900 is preferably placed at the edge of the image, as shown. In the example shown in Fig. 7B, the grid is radial and gridlines are spaced every 5 degrees. The center point of the grid 910 is used as a reference point. In an embodiment, a radial grid only extends 180 degrees rather than 360. It is to be understood that while only a linear and radial grid are shown in the Figures, the present invention may be practiced with any form of grid and any form of grid spacing.

[0038] The placement of the grid may depend on the exact shape of the image. For example, for a spiral, a radial grid may be used and the reference point may be placed at the center of the spiral. If a spiral is divided in half, a 180° radial grid may be used and aligned with the halfspiral. For a linear grid, the reference line may be placed at the bottom of the image or the left side of the image. In an embodiment, the location and spacing of the grid can be optimized by a software module to produce the best location and spacing for identifying different features. [0039] Then, the system performs a Linearized Compressed Polar Coordinate (LCPC) transform on the image. To do this, the system finds all the intersection points between gridlines and the image 730, and for each intersection point, a distance is calculated between the intersection point and the reference line or reference point 740, as shown in Fig. 8. Then, a transform function is created 750, where the X coordinate is the gridline position and the Y coordinate is the distance between the intersection and the reference point, as shown in Fig. 9. In an embodiment, if the image intersects a given gridline multiple times, the intersection distances for that gridline are summed up as a “compression” step.

[0040] Once a transform function is created, the system then performs a decomposition transform 760 on the transform function to obtain a cluster of components. The purpose of the decomposition transform is to produce a cluster of components that can more easily be analyzed by machine learning methods. Several types of decomposition transform will be discussed below, and all of them are consistent with the present invention. The cluster of components may be a frequency spectrum, as in the case of a Fast Fourier Transform, or may be another group of numbers, as in the case of other decomposition transforms.

[0041] Once the cluster of components is obtained - whether it is a spectrum or not - it is then inputted 770 into a machine learning or neural networking module in order to characterize the image and predict the medical status of the patient who generated the image. The machine learning module may be configured with instructions to analyze a large dataset of information (i.e. LCPC transforms) and its correlation to various medical statuses. The machine learning module may comprise an artificial neural network, a decision tree, a support vector machine, a regression analysis, a Bayesian network, logistic regression, or any other similar module.

[0042] The machine learning module may be trained using a training data set in which input data includes both the desired input clusters of components as well as known outcomes (e.g. clusters of components that correlate with Parkinson’s Disease). The system may “learn” from the exemplary' training data set. The objective of the training process may be to approximate the function /between the input and the output in order to later use the model to predict output values with high accuracy. The machine learning model may be trained on the raw cluster data, or on numerical index values that are generated from the cluster of components. Cluster data from nonaffected individuals or affected individuals may serve as the ground truth categories on which the model may be trained. For example, clock drawings by patients with Alzheimer’s Disease and clock drawings by healthy patients may be used in the training data set.

[0043] When there are enough examples in the system, the machine learning model can be used to predict the output value (i.e. medical status) based on training with prior patient data. The model may be configured to report categorical outputs (“normal” or “affected”), or to report continuous outputs (“percent risk of disease”.) The results are then displayed 780 for the user. LCPC Transform

[0044] The Linearized Compressed Polar Coordinates (LCPC) Transform is used to objectively translate complex two-dimensional shapes into a transform function on which a decomposition transform can be performed, a cluster of components is then obtained. To perform the LCPC transform, a grid is overlaid on top of the image; the grid may be linear or radial. In the case of a radial grid, the grid may extend over 180° or 360°, or any other angle. For a linear grid, one of the gridlines is considered a reference line; for a radial grid, the center is the reference point. Once the grid is overlaid over the image, the system determines all the intersection points between a gridline and the image, and finds the distance between each intersection point and the reference point or reference line. In an embodiment, if the image intersects the gridline multiple times, all the distances are summed up into one value; this summing step may be referred to as “compressing.” This results in ordered pairs that can form a function, where the X-coordinate represents the numerical order of the lines in the grid system, and the Y-coordinate represents the distance value associated with each line (or the compressed distance value). Tire transform function may then be subjected to a decomposition transform, such as a Fast Fourier Transform, to obtain a cluster of components. Once the cluster of components is obtained, it may be processed in different ways to turn it into an index number or a set of numbers, or may be analyzed as is. Since these components are precise quantities, the LCPC transform may be highly sensitive in measuring shapes and slight variations in shapes, lending itself well to machine learning analysis of these shapes. Because the difference between drawings produced by patients may be very subtle, it is important to have a more sensitive method of measuring the properties of these drawings objectively, especially when it comes to features like shakiness produced by a tremor. The LCPC transform provides a much more sensitive way of doing so than simply doing a machine-learning analysis on the shapes directly.

[0045] The grid spacing can be any spacing; closer-spaced gridlines will provide a more accurate analysis of the image, but will require more computational power to process. In an embodiment of the present invention, the grid spacing is every 5 pixels for a linear grid; in other embodiments, the grid spacing can be every 2 pixels, every 10 pixels, every 15 pixels, or any other spacing. For a radial grid, the grid spacing can be every 5°; in other embodiments, the grid spacing can be 1°, 10°, 15°, or any other spacing. In an embodiment, the grid spacing is subjected to an optimization process, and is optimized for maximum accuracy.

Decomposition Transform

[0046] The decomposition transform described in this disclosure may be a Fast Fourier Transform, which transforms the transform function into a cluster of frequency components. In other embodiments, other decomposition transforms may be used, such as the Fourier Transform, Laplace Transform, Z-transform, Slant Transform, and Fractal Analysis. Any transform that can decompose a complex function into composite components may be used for the present invention. The decomposition transform will be discussed in more detail below.

[0047] One type of decomposition transform that can be used for the present invention is the Fast Fourier Transfonn, which can transform a 2D sinusoidal function into a cluster of frequency components. Fig. 10 shows the input and output of the Fast Fourier Transform on a sample image. For this embodiment, the cluster of components is a frequency spectrum that may be summed up into different “bins” for easier analysis. Since each unique shape produces a unique frequency spectrum, this is a sensitive and objective way of analyzing shapes that lends itself well to machine learning.

[0048] If the sinusoidal function is continuous rather than discrete, a Fourier Transform may be used. That, too, will produce a cluster of components that looks like a frequency spectrum, and may be analyzed the same way.

[0049] In an embodiment, a Laplace Transform may be used. This is especially helpful if the function to be transformed is 3 -dimensional rather than 2-dimensional.

[0050] In an embodiment, a Z-transform is used. This is used for discrete 3D sinusoidal waves, and is essentially a discrete version of the Laplace transform.

[0051] While some of the decomposition transforms used for the present invention produce a frequency spectrum, not all of them do. For example, a Slant Transform may be used to produce a cluster of components by arranging all the pixels of an image in a line, turning a 2-dimensional image into a 1 -dimensional row of pixels, and then using a transform on that function.

[0052] Similarly, a fractal transform may be used to analyze the shape. Fractal analysis describes 2D shapes based on 3 numerical components: the macro-scale features, the meso-scale features, and the micro-scale features. Those three numbers can be extracted from the transform function in varying ways. For example, the object can be divided up into small straight line segments; then the number of straight line segments, the average length of the line segment, and the ratio between the number and the length can be a cluster of components.

Sample Use Cases

[0053] The following paragraphs describe some potential use cases for the method and system of the present invention. It is to be understood that the below description is not meant to be limiting and that the present invention may be used for analyzing other images and diagnosing other types of medical status. The use eases arc given here solely for illustration. [0054] Fig. 11 shows two sample spiral drawings and a plot of the FFT results for each one. For

Spiral 1, the patient’s tremor is severe early on. For Spiral 2, the patient’s tremor worsens with prolonged concentration. Having the cluster of components for each one makes the difference between the two plots very obvious, and makes it much easier for the machine learning system to do a differential diagnosis. Tire system of the present invention can also be used to measure improvements in tremor, for example in response to medication.

[0055] In an embodiment, the present invention can be used to tell the difference between two mirrored images - for example, a left and right hand, or a left or right handed spiral. Using standard measures of area or perimeter would not be helpful in distinguishing right or left handed features; the present invention makes this easy.

[0056] Fig. 12 shows a sample handwriting analysis use case for diagnosing autism in children. It is known that children with autism show specific handwriting impairments that can be used for diagnosis. As is shown in the Figure, each cursive letter L has a distinct cluster of components associated with it; machine learning can then be used to predict the likelihood of autism or categorize the degree of affect.

[0057] As is shown in Fig. 13, it is also easy to determine the average size of handwriting (which is useful for diagnosis of Parkinson’s Disease, which often involves micrographia), simply by summing all the bins.

[0058] As is shown in Fig. 14, the system of the present invention makes it very easy to determine letter spacing in handwriting by determining the total number of bins. More bins would correlate to greater spacing. In Parkinson’s disease, the letters are very close together; in autism, letter spacing is uneven. The present invention can make it easy to diagnose each condition.

[0059] Fig. 15 shows an application of the present invention to eye-tracking. Eye-tracking data may be collected by smart glasses, a smartphone, or any other device with a camera, such as a computer, tablet, or even a dashboard camera on a car or truck. Eye-tracking may be used to diagnose nystagmus, epilepsy, gaze palsies, or autism. Fig. 15 shows an eye-tracking pattern of a person’s gaze over a human face; the left side shows a person without autism, and the right side shows a person with autism. Since people with autism do not focus on a face in the same way neurotypical people do (in particular, they do not look at a person’s eyes as much as a neurotypical person would), an eye-tracking pattern may be used to diagnose autism.

[0060] Fig. 16 shows an application of the present invention to body movement tracking. Body movement tracking data may be collected by a smartwatch, smart ring, fitness tracker, or a smart body brace. Fig. 16 shows the body movement patterns of a person with a neurological condition compared to the movement patterns of a healthy patient. Once the pattern is recorded, the system and method of the present invention may be used on it to diagnose the patient.

[0061] Exemplary embodiments are described above, ft will be understood that the system and method of the present invention encompass other elements and embodiments that fit the limitations of the appended claims.