System and method of assessing a movement pattern

FIELD OF THE INVENTION

The invention refers to a method of assessing a movement pattern, a system for assessing a movement pattern using the method and to uses of the system for training of a movement and for a motor impairment rehabilitation therapy.

BACKGROUND OF THE INVENTION

Stroke is the most prominent cause of permanent disability in the industrialized countries. One of the most prominent disabilities stroke survivors suffer from is half sided paralysis of the upper limbs. Rehabilitation exercises are known to be efficient in regaining motor control, provided the training is intense and the patient is guided in the therapy. Technical solutions for unsupervised home stroke rehabilitation require the use of sensors for acquiring the patient's posture during exercises with computer-based movement training systems. A key ingredient for ensuring high motivation and compliance of the patients is timely automated feedback to their performance. From US 5 078 152, a method for diagnosis and training of a muscle and joint system of a human patient is known. An exercise system and an arrangement for controlling parameters of an exercise movement is used. A patient performance goal is displayed during the patient exercise motion and the actual patient performance relative to the performance goal is tracked and displayed. It is a drawback of the described method that it does not distinguish individual movement patterns including hesitations during the exercises from an attempt to execute the exercise.

SUMMARY OF THE INVENTION

It is therefore an objective of the invention to provide a system and a method of assessing a movement pattern that takes a wider variability of motor impairments and individual movement patterns into account.

The above objective is achieved by a method of assessing a movement pattern, comprising the steps of: continuously monitoring sensor data which characterize a movement,

recognizing the movement pattern by comparing the sensor data to an actual template, comparing the recognized movement pattern to a target template. It is an advantage of the method according to the invention that complex three- dimensional movements can be processed and recognized, even if the movement pattern under assessment is pathological. The actual template which may as well be referred to as a personal template is, in the sense of the invention, a variable representation of the actual movement pattern under assessment, whereas the target template is a fixed reference, a representation of a desired movement pattern, like for example, a healthy movement pattern. The actual and the target template are preferably stored as data for beneficially easy comparison to the sensor data. However, an attempt to execute a movement under the condition of a motor impairment is hardly recognizable by comparing it to a healthy movement pattern. By comparing the sensor data to the actual template instead, movements which strongly deviate from the target template may advantageously be recognized as an attempt.

The method is advantageously used in rehabilitation exercises. Further, the method is applicable in any kind of training situation referring to certain movement patterns, like for example sports. A movement pattern in the sense of the invention is a certain exercise, which may be a comparably simple movement, like raising an arm or bending a knee or elbow, or a more complex motion sequence, like throwing an object or performing a certain kind of jump. According to the method, the movement pattern under assessment may also be detected in a longer motion sequence. Preferably, motion recognition algorithms are applied, which will be described in more detail later. The sensor data are preferably obtained using body-worn sensor technology to capture a person's motions. For example, commercially available inertial sensors and piezo -resistive strain sensors are used to track motions. The sensor data may advantageously be transferred wirelessly.

According to a preferred embodiment, the method further comprises the step of updating the actual template, using the recognized movement pattern. Thus advantageously, movements can be processed and recognized over a longer period of treatment, even if the movement pattern under assessment is changing over the course of the treatment. In the best case, regarding a rehabilitation situation, the actual template changes from a pathological to a 'normal' or healthy movement pattern.

Preferably, a feedback regarding the recognized movement pattern and/or the actual template is generated. The feedback may be given in any suitable manner, in particular

visually and/or acoustically. A person whose movement is assessed using the method according to the invention may advantageously be given the training target represented by the target template, as well as their own movement pattern as a timely feedback and encouragement in a suitable manner. The person whose movement is assessed may, for example be a patient under rehabilitation therapy or an athlete in a training session.

According to another preferred embodiment, the method further comprises the step of monitoring an evolution of the actual template with respect to the target template. Preferably, a distance of the recognized movement pattern and/or the actual template to the target template is calculated. More preferably, a feedback comprises a visualized display of the distance.

In still a further embodiment a person performing the movement under assessment is instructed according to the target template, the instructions preferably being presented on a display by an electronic data processing means. The instruction can be given as written or spoken text, as video or as a rendered movie. According to a further preferred embodiment, an algorithm for recognizing the movement pattern comprises Dynamic Time Warping (DTW), the target template preferably being represented by a multidimensional time series of data. Dynamic Time Warping is an algorithm which may be used for determining a similarity between two sequences which may vary in time or speed. For instance, similarities in movement patterns may advantageously be detected, even if the movement in a first sequence is executed slowly and in another sequence it is executed more quickly, or even if there were accelerations and decelerations during the execution of the monitored movement. DTW is advantageously applicable to any data which can be turned into a linear representation, including video and audio data. Generally speaking, DTW is a method that allows a computer to find a match between two given sequences, as for example time series of sensor data, with certain restrictions. The sequences are "warped" non- linearly in the time dimension to determine a measure of their similarity independent of certain non- linear variations in the time dimension. Continuity is less important in DTW than in other pattern matching algorithms which are advantageously appropriate for the recognition of pathological movement. DTW is an algorithm which is advantageously suited to matching sequences with missing information. The optimization process is performed using dynamic programming.

According to still a further preferred embodiment, an algorithm for recognizing the movement pattern comprises Hidden Markov Models (HMM), the target template preferably being encoded in form of a set of model parameters for a Markov chain.

The Hidden Markov Model (HMM) is known in the art to be a statistical model in which the system being modeled is assumed to be a Markov process with unknown parameters. The hidden parameters are advantageously determined from the observable parameters. The extracted model parameters can then be used to perform further analysis, i.e. in this case for the recognition of the movement pattern.

For the continuous monitoring of the movement, preferably one or more sensors are used which are appropriate for motion detection. The sensors are applied to a limb of a person and the sensor data is fed into an electronic data processing means. According to the invention the target template is a representation of a reference movement pattern. The target template may be chosen from a set of standardized movement patterns. Preferably the target template is recorded by a reference person, using sensors which are appropriate for motion detection and an electronic data processing means. The reference person may advantageously be a person who is capable of executing the respective movement or exercise in a desired way, for example a therapist or a trainer. According to still a further preferred embodiment, the actual template is recorded in an initial session by the person performing the movement under assessment, preferably under the instruction of a reference person, for example the therapist or trainer.

Another object of the invention is a system for assessing a movement pattern using a method according to the invention as described in here before. The system comprises: - one or more sensors which are appropriate for detecting a movement, an electronic data processing means, adapted to process algorithms for recognizing the movement pattern and for comparing the recognized movement pattern to the target template, a storage for storing at least the actual template and the target template. The system offers the advantage that complex three-dimensional movements are recognized, even if the assessed movement pattern shows yet no similarity to the pattern according to the target template. The system may beneficially be applied in rehabilitation and/or training exercises.

Preferably, the electronic data processing means is adapted to process an algorithm for updating the actual template using a recognized movement pattern. Movements are thus reliably recognized, advantageously also over a longer period of treatment, wherein the movement pattern under assessment is changing over the course of the treatment. The system preferably comprises output means for providing feedback and/or instructions. The feedback and/or instructions may be given as written or spoken text, as video or as a rendered

movie. Preferably the system comprises a visual and/or an audible output.

Another object of the invention is a use of the system according to the invention for training of one or more movements of a person.

Another object of the invention is a use of the system according to the invention for a motor impairment rehabilitation therapy.

Personalized motion analysis algorithms take the changing movement patterns of a person due to training or treatment into account and allows real-time feedback and an easy-to -understand performance measure for the patient. The algorithms can be generically used for a variety of sensor types, implementation details and for all applications where training or recovery of motor abilities is essential.

These and other characteristics, features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention. The description is given for the sake of example only, without limiting the scope of the invention. The reference Figures quoted below refer to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 illustrates schematically an embodiment of the system according to the present invention. Fig. 2 illustrates in a diagram an example of generating a target template from examples for a method according to the present invention. Fig. 3 illustrates embodiments of the method according to the invention in a flowchart.

Figs. 4a, 4b, 4c and 4d illustrate an evolution of an actual template in schematic diagrams.

Fig. 5 illustrates an actual template compared to a target template in a schematic diagram.

DETAILED DESCRIPTION OF THE EMBODIMENTS The present invention will be described with respect to particular embodiments and with reference to certain drawings but the invention is not limited thereto but only by the claims. The drawings described are only schematic and are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for illustrative purposes.

Where an indefinite or definite article is used when referring to a singular noun, e.g. "a", "an", "the", this includes a plural of that noun unless something else is specifically stated.

Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein. Moreover, the terms top, bottom, over, under and the like in the description and the claims are used for descriptive purposes and not necessarily for describing relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other orientations than described or illustrated herein. It is to be noticed that the term "comprising", used in the present description and claims, should not be interpreted as being restricted to the means listed thereafter; it does not exclude other elements or steps. Thus, the scope of the expression "a device comprising means A and B" should not be limited to devices consisting only of components A and B. It means that with respect to the present invention, the only relevant components of the device are A and B.

Fig. 1 refers to a system 1 for assessing a movement pattern according to an embodiment of the invention. The system 1 comprises an electronic data processing means 10, a storage 12 and a display screen 11. The transfer of data between the electronic data processing means 10 and the storage 12 is illustrated by first arrows 13 and to the display screen 11 by a second arrow 14. One or more motion detecting sensors 20 are used which capture the movements of a person P. The sensors 20 can be e.g. inertial sensors, marker- based or marker-less camera systems for motion acquisition, wearable strain sensors or others. A stream of sensor data from the sensor 20 to the electronic data processing means 10 is illustrated by a third arrow 15. An instruction to the person P to perform an exercise is given via the display screen 11 which may as well comprise loudspeakers. The instruction can be given as written or spoken text, as video or as rendered movie. A target template 23 (cf. Fig. 4a), a representation of a reference movement pattern when executing the exercise is recorded, using the sensors 20, processing the data in the electronic data processing means 10 and storing it by means of the storage 12. A representation of the typical movement pattern of

the person P is recorded and stored in the same way when doing the exercise as an actual template 24 (cf. Fig. 4b). A first algorithm is executed on the electronic data processing means 10, to recognize the movement pattern 25 (cf. Fig. 4c) of the exercise in the stream of sensor data generated from the person's movement. For the recognition, this first algorithm compares the sensor data to the actual template. A second algorithm to update the actual template is running on the electronic data processing means 10, such that the actual template reflects the change of movement patterns over the course of treatment. A third algorithm to compare the personal template to the target template is running on the electronic data processing means 10. Feedback on the deviation of the actual template and/or the latest recognized movement pattern is output via the display screen 11.

Before the person P starts training with the system 1 , a reference person, for example a therapist or trainer defines the exercise and the target template. The reference person wears the sensors 20 to capture reference movement patterns by executing the exercise several times, while the electronic data processing means 10 records the corresponding sensor data. From this sensor data, the target template is generated.

Referring now to Fig. 2, the generation of the target template 23 is illustrated in a diagram, showing sensor data values on the ordinate axis 21 and a number of sampling points on the axis of abscissae N. Sensor data 22 are recorded during multiple executions of the same exercise (the thin lines). The target template 23 is then generated from the sensor data 22 of the past exercises. A representation of the target template 23 can either be a time series of the sensor data 22, or in a more abstract form, depending on the chosen movement recognition algorithm. Possible algorithms include Dynamic Time Warping (DTW), where the target template 23 is given by a multidimensional time-series of the sensor data 22, or Hidden Markov Models (HMM), where the target template 23 is encoded in a more abstract way in form of a set of model parameters for a Markov chain. In Fig. 2, the target template 23 for a drinking motion is generated from examples 22. Likewise, the actual template (not shown) of the person is generated in an initial session. Later on, the person can use the system 1 alone. While attempting to execute the requested exercise, the sensor data is analyzed by the movement recognition algorithm, which searches the stream of sensor data for the pattern stored in the actual template. The target template 23 and the actual template may largely differ, for example if a heavily impaired person is executing the exercise. Thus, any algorithm is looking for the target template 23 instead of the actual template would fail to recognize the attempts to execute the exercise, which might lead to a low compliance of the person. Upon recognizing the actual template, the movement recognition algorithm triggers a

feedback for the person, e.g. in form of a jingle or on-screen feedback. Furthermore, the recognized movement pattern is used to update the actual template (e.g. by generating the actual template from the last five or the last ten repetitions of the exercise, as illustrated in Fig. 2) to ensure that the movement recognition algorithm faithfully reflects the changing capabilities of the person.

In Fig. 3, a flow chart illustrating the method and system according to the invention is depicted. According to a preferred embodiment, feedback on the person's movement pattern is given, using the actual template. In step 100 the target template and the actual template are loaded from the storage 12 (Fig. 1). Step 101 represents the loading and display of instructions regarding the exercise and from step 102 on, the sensor data is continuously monitored. During monitoring the sensor data, the actual template is used to recognize the movement pattern of the exercise (step 104). Any time an attempt to execute the exercise is recognized, a visual or audible feedback is given in step 103 and the actual template is updated by the recognized movement pattern in step 105. The updated actual pattern is now used by the recognition algorithm in steps 102 and 104. A deviation of the updated actual pattern from the target pattern is calculated and given as a feedback in step 106. In Figs. 4a, 4b, 4c and 4d, an evolution of the actual template 24 is illustrated. The target template 23 (Fig. 4a) and the actual template 24 (Fig. 4b) show a marked difference. Once the recognition algorithm has recognized an execution attempt, the data of the recognized movement pattern 25 (Fig. 4c) is used to update the actual template 24 from the old actual template (Fig. 4d, lower curve) to the new actual template (Fig. 4d, top curve), the new actual template 24 coming closer to the target template 23.

A possible update mechanism can be training the actual template 24 according to algorithms used for DTW or HMM, where only the recorded sensor data of the last executions (e.g. last 5 or 10) of the exercise are used. Finally, the person can receive feedback on the deviation from the desired reference movement by calculating and visualizing the distance between the target template 23, representing the reference movement pattern and the actual template 24, representing the actual movement pattern of the person. Visualization can be achieved by showing two rendered Figures where one shows the motion encoded in the target template 23 and the other in the actual template 24, or by an alignment of time series in case of DTW (see Fig. 5).

In Fig. 5, a time series graph is depicted, comparing the actual template 24 or recognized movement pattern data 25 to the target template 23. In this case, the differences are minimal.