Technical Field

The invention relates to methods and devices for determining a physical state of a subject based on sensor data provided via a wearable device worn by the subject.

Background Art

Today, wearable devices, such as smart watches, are commonly used for registering number of steps, heart beats per minutes, distance run, etc. In addition to capturing sensor data, it is also known to use this data for providing training reminders, alerts, e.g. if the heart beat is at or above a critical level, and other notifications that may be of interest for a user of the wearable device. To be able to see training progress over time as well as more detailed statistics, many of the wearable devices are made to interact with computers or other devices suitable for presenting statistics and other information on a user interface made for a larger screen than provided by e.g. the smart watch.

In parallel with the development of the wearable devices used by persons enjoying running or generally interested in keeping track of their physical activity, tests have been carried out to see if customized accelerometer devices attached to limbs of persons suffering from Parkinson’s disease can be used for determining level of bradykinesia at different points of time. By being able to monitor the level of bradykinesia, it is made possible to more accurately determine status of the disease. By being able to continuously capture data and process this data and not rely on physical meetings with a doctor or other healthcare personnel, the status of the disease can be determined more quickly. An effect of having better understanding of the status of the disease is in turn that the administration of Levodopa can be administered more accurately, thereby more efficiently reducing the impact of the disease. In EP 2 306 899 B1 by Amygdala Pty Ltd and Golgi Pty Ltd, it is disclosed how hypokinetic states can be determined by using an accelerometer data logger.

Even though tests have been made and methods and devices have been described on how accelerometer devices can be used for determining hypokinetic states, there are still challenges to overcome. One such challenge is how to more accurately determine the status of the disease such that e.g. administration of medicine can be improved. Summary

It is an objective of the invention to at least partly overcome one or more of the above-identified limitations of the prior art. In particular, it is an object to provide a method for determining a physical state, such as hypokinetic state, by both taking into account individual characteristics of the subject - the person wearing the device - as well as take into account learnings made from other subjects.

A further objective is to provide appropriate dosing of medicine by processing the movement data captured via the wearable device.

Still an objective is to determine effects caused by non-medicine substances, such as energy supplement bars, either taken alone or in combination with the medicine, by processing the sensor data captured from the wearable device.

Yet another objective is to determine effects caused by interventions with the subject, such as physiotherapy and/or speech therapy.

In addition, it is an objective to determine effects caused by medicine, nonmedicine substances and/or interventions in combination.

By applying a combination of a first and a second model, the first model relying on data from multiple subjects and the second model being specifically adapted to the subject in question, it is made possible to use data captured from a large pool of subjects suffering from similar diseases and conditions, but still be able to take into account that the different stages or states of a disease may come across in different ways for different subjects. For instance, subjects suffering from Parkinson’s disease will share some characteristics, while other will only apply to a few. For instance, a transition phase from an ON-state to an OFF-state may have at least partly different characteristics for different subjects. In addition to that the symptoms for a particular disease may to some extent be different for different subjects, the symptoms may also be experienced differently for the different subjects. As a result, asking subjects about the symptoms in interview form comes with the drawback that information provided by the subjects are to at least some degree subjective. For instance, what is seen as a problem by one subject may not be seen as a problem for another subject. An effect of this subjectivity is that combining information from interviews with different subjects may result in unreliable conclusions or that detailed models cannot be achieved.

Using the two-model approach suggested herein comes with a number of advantages and overcomes, or at least mitigates, the drawbacks presented above with respect to current approaches. First, by having the second model in situations when the subject in question may deviate from a general movement pattern, it is made possible to take this into account by the second model. In this way, the risk of having incorrect or at least non-optimal decisions made by the first model can be remedied by the second model. Second, by having the two-model approach the need for custom- built wearable devices may be reduced. By having the second model configured to remedy non-optimal decisions made in the first model may namely also result in that a general-purpose smart watch may be sufficient. Third, an effect of being able to reliably determine the physical state is that a dosing regimen for the subject may be adapted accurately, providing for that both the risk of under-medicating and over-medicating can be reduced. Fourth, since symptoms also are experienced differently, the second model also provides for that symptoms experienced as problems for the subject in question, and as an effect influences the movement pattern of this subject, can be recognized and acted upon. Fifth, by having the two-model approach, the first model, trained based on movement data from multiple subjects, can be used for identifying categories of movement reliably, e.g. walking, standing, sitting down. By knowing the category of movement, it is in turn possible for the second model, which is subjectspecific, to take this into account when identifying the physical state, which in turn provides for that the physical state can be identified more reliable.

Throughout this document, the expressions below are used. To avoid any confusion, please find below how these expressions are to be understood in the context of this document:

“State of disease” describes in the context of Parkinson’s disease whether the disease is in early stage or late stage, and whether there is wearing off (where the changing from on- to off-state starts to be so fast that the medication does not work properly anymore). Thus, state of disease describes the stage of the disease in longer run.

“State of symptom(s)”, in the context of Parkinson’s disease, describes e.g. whether the patient is in on- or off-state or in a dyskinetic state. State of symptom(s) describes the patient state in a shorter term than state of disease. The state of symptoms may be reflected as motor and non-motor symptoms.

According to a first aspect, it is provided a method for determining a physical state of a subject by using movement data obtained via a wearable device worn by the subject, said method comprising identifying a movement representation among a number of movement representations based on the movement data using a first model, wherein the first model is a generative model using data from multiple subjects, retrieving weights linked to the movement representation, weighting the movement data using the weights, thereby forming a weighted movement representation, and identifying the physical state among a number of physical states based on the weighted movement representation using a second model, wherein the second model is subject-specific.

By having the first model using information gathered from a large pool of subjects such that a movement representation can be identified, and combining this with the second model, which is subject specific, it has been found that the physical state can be determined reliably. For instance, it can be reliably determined that the subject being in the physical state should eat, drink, rest, etc to avoid an unwanted situation, such as fainting.

The first model can be a latent representation forming model. Put differently, chaotic movement data can be received by the first model such that this model is trained to identify the movement representation among the number of movement representations.

The physical state may be linked to a state of disease and/or a state of symptom.

The state of disease and/or the state of symptom may be a disease and/or a state associated with a movement disorder.

The disease and/or state may be selected from a list consisting of Parkinson’s disease, epilepsy, Multiple sclerosis (MS), Alzheimer’s disease, dementia, chronic or acute musculoskeletal pain and myopathy. In addition to the diseases and states listed herein, generally speaking, the method can be used for any illness affecting the musculoskeletal system, either directly or indirectly.

The disease may be Parkinson’s disease and the state is on state or off state.

The method may further comprise requesting symptom data from the subject, receiving the symptom data from the subject, and linking the symptom data with the movement data.

By having the symptom data linked to the movement data, the movement data is labelled and a more correct identification of the movement data can be achieved. Put differently, the fact that the subject can input symptom data via a mobile phone or the wrist watch is made use of.

The method may further comprise comparing the movement data with trigger data, in case of match , triggering the step of requesting the symptom data from the subject.

By being able to control when to ask the subject for symptom data, e.g. to fill gaps in the first model, symptom data can be captured in a more structured manner and as an effect a more reliable second model can be achieved.

The symptom data may comprise motor symptom data related to tremor, rigidity, bradykinesia, dyskinesia and/or balance, and/or non-motor symptoms related to sleeping disturbances, anxiety, dizziness, hallucination, changes in ability to smell or taste, urinating, digestion, pain, fatigue and/or depression.

The method may further comprise requesting substance intake data from the subject, receiving the substance intake data from the subject, and linking the substance intake data with the movement data.

By requesting substance intake data from the subject, the second model may be based on a combination of movement data and substance intake data or a combination of movement data, symptom data and substance intake data. The substance may be medicine, but it may also be different kinds of food, water etc.

The symptom data and/or the substance intake data may be requested by transmitting a symptom data request and/or a substance intake data request to a mobile phone assigned to the subject, wherein the symptom data and/or the substance intake data is received via the mobile phone.

An advantage of using a mobile phone is that the symptom data and/or the substance intake data may be requested by using both audio information and/or visual information. It is also made possible for the subject to be connected to assisting personal via the mobile phone if needed. By having the mobile phone assigned to the subject, using the mobile phone as an interface with the subject, also provides for that a risk that the information is input by someone else than the subject is reduced. If having mobile phones provided with user identity verification functionality, e.g. fingerprint or face recognition, this will further reduce this risk.

The physical state may either be a state with positive response to substance intake and reduced symptoms, determined based on the movement data registered after the substance intake data is received, or a state with no or negative response to the substance intake and symptoms remained or worsened, determined based on the movement data registered after the substance intake data is received. This is advantageous for instance in case the subject is suffering from Parkinson’s disease. This disease is namely known to have two states; an ON state and an OFF state. Depending on which of these states the subject is in, different response to the substance intake will be achieved. Thus, by taking into account substance intake data, more reliable determination of the physical state can be achieved.

In addition to the example of ON and OFF states above, in another example the subject may also be or not be in a dyskinetic state. By being able to determine whether or not the subject is in this dyskinetic state, it is made possible to determine if the subject is in the physical state with positive response to substance intake or in the physical state with no or negative response to the substance intake.

The wearable device may be a limb-worn device, such as wrist-worn device, leg-worn device or finger-worn device, provided with an accelerometer, a gyroscope and/or a biosignal sensor.

The first model may be chosen from a group comprising of an autoencoder (AE) model, a principal component analysis (PCA) model and a variational autoencoder (VAE), and the second model may be a regression model or a classification model, such as a logistical regression model, a convolutional neural network (CNN) or a random forest model.

The method may further comprise determining a suggested time for the substance intake, a suggested amount for the substance intake and/or a suggested type of substance for the substance intake based on the physical state.

An advantage with being able to provide when, how much and what to consume is that it is made possible for the subject to act such that an unwanted condition may be avoided.

According to a second aspect it is provided a data processing apparatus configured to determine a physical state of a subject using movement data said apparatus comprises a movement representation identifying module configured to identify a movement representation among a number of movement representations based on the movement data using a first model, wherein the first model is a generative model using data from multiple subjects, a movement data weighting module configured to weight the movement data using weights linked to the movement representation, thereby forming a weighted movement representation, and a physical state identifying module configured to identify a physical state among a number of physical states based on the weighted movement representation using a second model, wherein the second model is subject-specific.

The same features and advantages as presented above with respect to the first aspect also applies to this second aspect.

According to a third aspect, it is provided a system comprising a wearable device arranged to be worn by a subject, said wearable device is configured to generate movement data based on movements of the subject, and a data processing apparatus according to the second aspect communicatively connected to the wearable device and configured to receive the movement data from the wearable device.

The system may further comprise a mobile phone communicatively connected to the data processing apparatus, wherein the mobile phone is configured to upon a symptom data request transmitted from the data processing apparatus provide symptom data to the data processing apparatus, wherein the data processing apparatus is further configured to receive the symptom data and link the symptom data to the movement data.

The mobile phone may be configured to upon a substance intake data request transmitted from the data processing apparatus provide substance intake data to the data processing apparatus, wherein the data processing apparatus is further configured to receive the substance intake data and link the substance intake data to the movement data.

According to a fourth aspect, it is provided a computer program comprising instructions which, when the program is executed on a computer, cause the computer to carry out the steps of the first aspect.

Still other objectives, features, aspects and advantages of the invention will appear from the following detailed description as well as from the drawings.

Brief Description of the Drawings

Embodiments of the invention will now be described, by way of example, with reference to the accompanying schematic drawings, in which Fig. 1 generally illustrates a system comprising a number of subjects equipped with wearable devices sharing data with a data processing apparatus.

Fig. 2 illustrates an example of how a physical state of one of the subjects can be determined based on movement data captured by the wearable device of this subject.

Fig. 3 illustrate an example of how the physical state of one of the subjects can be determined based on the movement data in combination with symptom data.

Fig. 4 is a flowchart illustrating a method for determining the physical state based on the movement data.

Fig. 5 illustrates the data processing apparatus in further detail.

Fig. 6 illustrates effects on bradykinetic accelerometer scores according to some examples.

Fig. 7 illustrates an example on how the data analysis of for example symptom data may be performed after the identification of a movement representation.

Fig. 8 is a flowchart illustrating a method for determining the physical state based on the movement data.

Detailed Description

Fig. 1 illustrates by way of example a system 100 to which three persons, also referred to as subjects 102a-c, are linked. As illustrated, each subject 102a-c may be equipped with a wearable device 104a-c, such as a watch worn around a wrist. The wearable device 104a-c may be provided with one or several sensors, such as accelerometer, gyroscope, infrared sensors for measuring pulse, etc. By using the wearable device 104a-c, movement data 106a-c from each of the subjects 102a-c may be captured. Even though illustrated as one wearable device per subject, several wearable devices may be used for each subject. By way of example, in addition to the watch also a mobile phone 108a-c belonging to the subject 102a-c may be used for capturing the movement data 106a. The movement data may be data directly related to movement, such as accelerometer data, but it may also be data indirectly related to movement, such as pulse, skin conductance, heart rate variability, etc.

The mobile phone 108a-c, or any other device provided with a user interface, may also be used for receiving symptom data 110a-c from the subject 102a-c. The symptom data 110a-c may be retrieved as user interactions with the user interface in response to direct questions posed to the user 102a-c via the user interface. For instance, “Are you currently suffering from dizziness (yes/no)?” and in case the user responds affirmatively provide a follow-up question such as “Please rate the level of dizziness from 1 to 5”. The subject may be asked to indicate e.g. the starting time and ending time of the symptom and/or the nature of the symptom via the user interface. The symptom data 110a-c may also be captured indirectly by asking the subject 102a-c to perform different tests or tasks, and based on the results of these estimate the level of symptoms experienced by the subject 102a-c. In addition, a combination of direct and indirect methods for receiving the symptom data 110a-c may also be used. For each subject, general information, such as age, gender, known diseases and ongoing medication, may be input to the system 100. The general information may be stored centrally and/or it may be stored locally in the mobile phone 108a-c and/or the wearable device 104a-c. The information may be provided by the subject herself or himself, but it may also be provided indirectly via e.g. a doctor or other medical staff. The information may be provided for instance via an app running on a mobile phone.

In this example, after having input the symptom data 110a-c to the mobile phone 108a-c, and the movement data 106a-c captured via the wearable device 104a- c is transferred to the mobile phone 108a-c, the movement data 104a-c and the symptom data 110a-c can be combined into subject data 112a-c and transferred via a data communications network 114 to a data processing apparatus 116, e.g. a server. As will be described in further detail below, the subject data 112a-c, which may comprise the movement data 106a-c and optionally also the symptom data 110a-c and/or the general information of the subject 102a-c, can be processed in the data processing apparatus 116 such that result data 118a-c for each of the subjects 102a-c is provided. The result data 118a-c may comprise information on a physical state of the subject 102a-c. Even if illustrated that the result data 118a-c is transferred to the subject 102a-c, it is also possible to have this data transferred to a caregiver or other instance providing for the subject 102a-c.

Having information about the physical state provided to the subject 102a-c, and possibly others as well, is that this information may assist the subject 102a-c in taking appropriate actions to avoid ending up in unwanted physical states. For instance, by providing information about the physical state to the subject 102a-c it is made possible for the subject 102a-c to take actions in form of resting, eating, drinking, medicate etc.

The result data 118a-c may in addition to, or instead of, comprise information about the physical state of the subject 102a-c comprise information about when and how the symptom data 110a-c should be input. For instance, different events identified in the movement data 106a-c may be used for triggering a request for the symptom data 110a-c from the subject 102a-c. These events may be different for different subjects 102a-c and for different physical states and also for different diseases. The result data 118a-c may also comprise information about which questions to ask or which tests to be performed by the subject 102a-c when capturing the symptom data 110a-c.

As will be described more in detail below, by having several subjects 102a-c participating, the movement data 106a and optionally the symptom data 110a-c may be linked to the disease the subject is suffering from. In such a situation, by knowing the physical state it may as an effect be possible for the subject to take appropriate actions based on this information. For instance, it may be that the movement data 106a-c alone, or in combination with the symptom data 110a-c, indicates that the physical state of the subject 102a is a state that requires a different dosing regimen, i.e. the way the medicine is to be taken, including formulation, route of administration, dose, dosing interval and treatment duration. Based on the physical state, a notification indicating e.g. changed dosing regimen can be provided.

Even though three subjects 102a-c are illustrated in fig. 1 , the number of subjects may be significantly higher. An advantage with having a vast amount of subjects involved is that models used in the data processing apparatus 116 are exposed to information from a wide range of subjects. Based on this information, the models used for determining the physical state can be continuously improved, e.g. by using artificial intelligence (Al), machine learnings (ML) and/or statistical models.

Fig. 2 illustrates by way of example how the physical state 210 can be determined based on the movement data 106a captured by the wearable device 104a worn by the subject 102a as illustrated in fig. 1.

As illustrated, the movement data 106a, e.g. accelerometer data, can be fed to a first model 200. This first model 200 serves the purpose of linking the movement data 106a to a movement representation 202. More specifically, the movement data 106a can be linked to one of a number of movement representations formed based on information gathered from a large number of subjects. For instance, one of the number of movement representations may be associated to a subject suffering from Parkinson’s disease that is in a transition phase from an ON state to an OFF state. In addition, the different movement representations may be linked to different actions, such as different dosing regimen, different treatments, etc, and outcomes of these actions. For instance, for the subject being in the transition phase exemplified above, the movement representation 202 may be linked to a specific dosing regimen known from the historical data to be successful in reducing the impact of the disease if followed.

Using data collected from a large base has the advantage that a large number of different situations are made available to the first model 200 via the data. However, since movement of limbs may vary from subject to subject, using the movement representation 202 in isolation may result in that subjects having atypical movement data may be provided with an incorrect physical state, and as an effect possibly a non- optimal dosing regimen. To avoid or at least reduce such risk, a weighting model 204 can be used for generating a weighted movement representation 206. The weighting model 204 can be provided with the movement representation 202, output from the first model 200, and the movement data 106a. By having access to the movement representation 202, based on the first model 200 being based on data from a plurality of subjects, and the movement data 106, based solely on the subject 102a in question, the weighting model 204 may be used for adjusting the movement data 106a such that findings made by the first model 200 can be made available to the second model 208. For instance, in case it is found by the first model 200 that the movement data 106a is linked to a specific stage of a specific disease, the movement data 106a may be modified such that this is reflected in the data being sent to the second model, herein referred to as the weighted movement representation 206. The movement representation 202 is however not restricted to states of the disease, but could also involve everyday life states such as walking, standing, sleeping, sitting etc. This may in turn be further divided into sub-categories such as walking outside in cold weather, sitting down and occupied with tasks involving fine motor skills, e.g. repairing a watch. The movement representation 202 does not have to relate to concretely definable movement (such as “running”) but can relate e.g. to movement of “class A” without a concrete semantic definition.

In contrast to the first model 200 in which the plurality of subjects is used for forming a reliable way of interpreting the movement data 106a and to link this data to one of a number of predefined representations, the second model 208 is subjectspecific. The second model 208 may be trained by requesting data from the symptom data 110a from the subject 102a, either by posing direct questions, but also in asking the subject 102a to perform certain physical tests or exercises. By having access to both movement data linked to these tests and also symptom data linked to these tests labelled data is achieved, which may be used for training the second model 208. Medication data, such as medicine intake data, e.g. when and how much medicine that is taken, may also be input by the subject 102a and taken into account when establishing or updating the second model 208. In addition to medicine, information on other substances, such as food products, consumed by the subject 102a may also be taken into account and provided to the second model 208. It is also possible to request and take into account information on exercises performed, sleep and other parameters affecting the physical state. After having assessed the weighted movement representation in the second model 208, the physical state 210 can be output. As described above, this information may in turn result in that notifications, e.g. changed dosing regimen, instruction to rest, diet instructions (nutritional composition, protein composition, eating schedule etc), exercise instructions and sleep instructions, are transmitted to the subject 102a and/or other party admitted access to this information. In case the subject is administered e.g. deep brain simulation therapy, the information of the physical state 210 can be used to determine deep brain stimulation operational parameters in coming therapy sessions.

In an alternative embodiment, even though not illustrated, the movement representation 202 may be transmitted directly to the second model 208, i.e. the second model 208 may be configured such that the weighting model 204 is forming part of the second model 208.

Even though the second model 208 is subject-specific, during a start-up phase, i.e. before the second model 208 has been trained based on the specific movement patterns of the subject, starting values that are non-subject specific may be used.

Fig. 3 illustrates, in line with fig. 2, how the physical state 210 can be determined based on the movement data 106a generated by the wearable device 104a. As in the example illustrated in fig. 2, the movement data 106a may be fed into the first model 200, which can be trained based on the movement data 106a-c from multiple subjects 102a-c and also optionally the symptom data 110a-c from the multiple subjects 102a-c. The movement representation 202 may be output from the first model 200 and fed into the weighting model 204 in line with the example illustrated in fig. 2. From the weighting model 204, the weighted movement representation 206 may be fed to the second model 208.

In addition to the weighted movement representation 206, the symptom data 110a may be fed into the second model 208. The symptom data 110a may as illustrated be obtained via the mobile phone 108a. Even though exemplified by the mobile phone 108a other means for obtaining the symptom data 110a are also possible. For instance, it is also possible to obtain the symptom data 110a via the wearable device 104a even though not illustrated.

When and what kind of symptom data 110a to request may depend on data captured by the wearable device 104a. For instance, in case a new combination of sensor data captured by the wearable device 104a is detected, e.g. pulse above a certain threshold and acceleration data in a certain interval, symptom data may be requested from the subject 102a to assure that a proper basis for determining the physical state 210 is provided to the second model 208.

In addition to or instead of the symptom data 110a, substance intake data 300 may be obtained via the mobile phone 108a, or other device, such as the wearable device 104a, and fed into the second model 208. As for the symptom data 110a, a substance intake data request may be sent from the wearable device 104a to the mobile phone 108a based on the movement data 106a registered. The request for the symptom data 110a and/or the request for the substance intake data 300 may also be sent from the data processing apparatus 116.

An advantage with having the symptom data 110a and/or the substance intake data 300 provided to the second model 208 is that in case movement data 106a is deviating from previously received movement data, it is possible, in order to make a more reliable physical state determination, to request the symptom data 110a and/or the substance intake data 300. In addition, it may also be used for filling in gaps of the second model 208.

Based on the physical state 210, a dose determination model 302 may be used for determining a dosing regimen, i.e. when and how much of a medicine or other substance that is to recommend for the subject 102a given the physical state 210 of the subject 102a. This information may be provided to the subject 102a by that dose data 304 is sent from the dose determination model 302 to the mobile phone 108a.

Trigger data 306 may also be sent to the mobile phone 108a from external devices, e.g. hospital personnel. This trigger data 306 may trigger that the symptom data 110a is requested and/or that the substance intake data 300 is requested via the mobile phone 108a. It may be that this data is requested promptly, but it may also be a request for having this information at a future point of time.

Regarding when and how to request symptom data, there are a number of situations when this may be beneficial. By way of example, symptom data may be requested in the following situations: 1. In the beginning of a treatment

To assure that a baseline is provided, the subject may be asked to input the symptom data in the beginning of a treatment.

2. Based on calendar

To make sure that the initial settings still apply, the subject may be asked to input the symptom data at predetermined points of time.

3. Insufficient inaccuracy

In case the model is not able to recognize the state, e.g. an uncertainty level is above a pre-set threshold, the subject may be asked to input symptom data to readjust the system e.g. start reguesting symptom data for a period of e.g. one week.

4. Confirmation

The system may ask the subject to confirm. For instance, when the physical state 210 determined is linked to a certain symptom, the subject can be asked to confirm that he or she is suffering from this symptom. It may also be possible to ask the subject, in case he or she is suffering from Parkinson’s disease, if he or she is shifting to off-state.

It is also possible to reguest symptom data such that a reinforcement loop is formed, that is, the symptom data 110a is reguested up until a certain accuracy is achieved. In such a situation, it may be that the movement data 106a is used as a trigger, but that the reinforcement loop is using comparisons between estimated symptoms and actual symptoms up until the physical state is determined with a pre-set accuracy. Put differently, the system tests the correspondence of the state produced by the second model and the symptom data given by the patient. This correspondence can be tested and synced until the accuracy is good enough and certain guality criteria are fulfilled. Reinforcement learning (either supervised or unsupervised) is used to test the correspondence.

The symptom data may comprise motor symptom data related to tremor, rigidity, bradykinesia, dyskinesia and/or balance, walking/gait, falls, stiffness, dizziness, freezing, muscle cramps, dystonia, wearing off, and motor fluctuation. In addition or instead, the symptom data may comprise non-motor symptoms related to fatigue, anxiety, depression, communication capabilities, speech effort, pain, sleep, skin and sweating, low blood pressure, restless legs, eating, swallowing and saliva control, mild memory and thinking problems, dementia, hallucinations and delusions, bladder and bowel problems, eye problems, foot care, dental health, impulse control disorders (ICD), hyposmia, decrease in quality of life, and sexual problems.

It has been found that in some situations, the symptom data can be limited to the motor symptom data, but still produce reliable results.

Further, the non-motor symptoms can be linked to the motor symptoms. For instance, anxiety, skin conductance data and heart rate variability data may be correlated to one or several of the motor symptoms. Thus, by receiving information about the non-motor symptoms, the one or several motor symptoms can be determined.

Fig. 4 is a flowchart illustrating a method 400 for determining the physical state of the subject. In a first step 402, a movement representation is identified among a number of movement representations based on the movement data using a first model. The first model can be a generative model using data from multiple subjects. For instance, after this step it may be identified that the subject is “walking”.

In a second step 404, weights linked to the movement representation is received. For instance, in the case the subject is identified to be walking, weights linked to this movement representation may be retrieved. The weights may be retrieved from a server shared by several subjects or the weights may be stored locally in the wearable device 104a-c or the mobile phone 108a-c.

After the weights have been retrieved, in a third step 406, the movement data 106a-c may be weighted. Put differently, the weights retrieved and the movement data 106a-c may be combined into the weighted movement representation 206.

In a fourth step 408, the physical state can be identified among a number of physical states based on the weighted movement representation using a second model. As described above, the second model may be subject-specific. For instance, after the subject is identified to be walking and the movement data captured from the wearable device is adjusted to reflect that the subject is walking, the weighted movement representation may be used for subject-specific assessment. In this assessment, the individual characteristics linked to the subject may be taken into account. By way of example, in case the subject is known to sweat and walk at a slower pace during the physical state this is taken into account in this step.

Optionally, in a fifth step 410, the symptom data 110a-c may be requested. In a sixth step 412, the symptom data 110a-c can be received, and once received, in a seventh step 414, the symptom data 110a-c can be linked to the movement data 106a- c. By having the symptom data 110a-c linked to the movement data 106a-c, it is made possible to also take into account the symptoms perceived by the subject 102a-c.

The request for symptom data 106a-c may be triggered by pre-set conditions, e.g. once a week, but it may also be triggered based on the movement data 106a-c. In an eighth step 416, as illustrated, the movement data can be compared with the trigger data 306. In case of match, in a ninth step 418, the symptom data 110a-c can be requested. In having this approach, the symptom data 110a-c can be dynamically requested from the subject 102a-c, that is, in case a reason to ask for input from the subject 102a-c is found, such information is requested, e.g. in that software application running on the mobile phone 108a-c is pushing a request to the subject to input information related to symptoms to the physical state.

Optionally, in a tenth step 420, the substance intake data 300 can be requested, and in an eleventh step 422 the substance intake data 300 can be received. In a twelfth step 424, the substance intake data 300 can be linked with the movement data 106a-c. In line with the symptom data 110a-c, the substance intake data 300 can be used as another piece of information for even more reliably identifying the physical state 210. For instance, by knowing what, when and how much of that substance, e.g. medicine, that has been used, the movement data 106a-c can be analyzed more reliably. The symptom data 110a-c and the substance intake data 300 can be provided in isolation or in combination. Further, even though not illustrated, the substance intake data request may also be triggered by that the movement data 106a-c matches trigger data related to the substance intake data.

Optionally, in a thirteenth step 426, once the physical state 210 is identified, the suggested time for the substance intake, a suggested amount for the substance intake and/or a suggested type of substance for the substance intake can be determined.

Fig. 5 illustrates the data processing apparatus 116 described above and also illustrated in fig. 1 more in detail. As illustrated, the apparatus 116 may comprise a movement representation identifying module 502 configured to identify the movement representation 202 among a number of movement representations based on the movement data 106a-c using the first model 200. A movement data weighting module 504 may be provided and this may be configured to weight the movement data 106a-c using the weights linked to the movement representation 202, thereby forming the weighted movement representation 206. A physical state identifying module 506 configured to identify the physical state 210 among the number of physical states based on the weighted movement representation 206 using the second model 208 may also be provided.

According to some examples, the first model may be based on a variational autoencoder, VAE. The use of the VAE allows for use of more complex generative models. The use of VAE further allows for fitting with generative models for larger sets of data.

To this end, the second model may be a regression model or a classification model, such as a logistical regression model, a convolutional neural network, CNN or a random forest model.

By way of example, the VAE may be trained on accelerometer data. The accelerometer data may pertain to movement registrations by a wearable device. The wearable device may be a smart watch. An advantage being that a sole wearable device worn by the subject may be used. The movement registration may comprise accelerometer data.

The wearable device may be worn by the subject. Put differently, the wearable device may be attached to a limb of a patient or person under study. The limb may, for example, be an arm of the subject. The wearable device may be worn on the wrist of the subject. Alternatively, the limb may be a leg or a finger of the subject. The wearable device may comprise an accelerometer for obtaining the accelerometer data.

Advantageously, accelerometer data pertain from a three-axis accelerometer. By way of example, three accelerometer channels (x, y, z) may be used for obtaining the accelerometer data. A more accurate determining of movement data may thereby be obtained.

The data analysis of movement data may be based on raw data obtained from the accelerometer. The accelerometer data may be subject to spectral processing prior to being used as by the VAE. The spectral processing may, by way of example, comprise Fast Fourier Transform, FFT, and subsequent calculation of logarithmic, dB- scale power spectra with varying bin sizes which may be logarithmically spaced.

The acceleration data may be aligned to an estimate of gravity, g, and to its temporal differentials, which are orthogonal to gravity. The estimate of gravity may be obtained by low-pass filtering. A moving coordinate system may thereby be constructed for the measurements of motion data may thereby be obtained.

As described above, an advantage of having a two-model approach, is that the first model may be trained based on movement data from multiple subjects such that it can be used for identifying categories of movement reliably, e.g. movement representations such as walking, standing, sitting down etc. By knowing the category of movement, it is in turn possible for the second model, which is subject-specific, to take this into account when identifying the physical state, which in turn provides for that the physical state can be identified more reliable.

According to some examples, the VAE may be trained to identify one or more movement representations prior to being used to determine symptoms linked to such a movement representation. Put differently, with the use of descriptors for the classification of different movement representations the analysis of the symptoms from the movement data may be simplified. In other words, by utilizing a classification pertaining for instance to a given activity for a subject, a more efficient detection of a symptom, for instance tremor, from the movement data may be made. Data having a lower signal-to-noise may moreover be utilized. A more robust method may thereby be provided.

By way of example, the movement states may without any limitation to these, pertain to a task such as: standing, sitting, lying, walking, moving limb, performing coordination movement or fine motor skills.

According to some examples, a VAE model may be used to determine a movement representation from movement data pertaining to accelerometer data. According to some examples, training data comprising accelerometer data and associated movement representations that may be used to train the VAE. Put differently, the trained VAE may be used to create representations for accelerometer data with the task of predicting which of the predetermined movement representations such a given tasks performed by the subject.

By way of example, the implementation of VAE may be done using PyTorch. The encoder of the VAE may be of a two layer type fully connected neural network. A symmetric decoder may further be used.

Thus, a pre-detection of these states with a VAE from accelerometer data with adequate signal-to-noise ratio, and subsequent quantification of for example Parkinson's disease symptoms may be advantageous.

Fig. 6 illustrates effects on bradykinetic accelerometer scores according to some examples. In more detail, the experimental data in Fig. 6 illustrates effects of Levodopa infusion for subjects. The Levodopa infusion may be referred to as “drug state”.

The experimental data pertains to accelerometer measurements from subjects, with and without Levodopa infusion, the subjects performing different given tasks, and some repetitions within the given task. Tremor, bradykinesia, and dyskinesia occurring in the experiment where scored by external observers onto an ordinal scale. The experiments showed that effect of the drug state on symptoms can be seen both in observer and in accelerometer scores, and with expected polarity, which means dyskinesia is amplified, and bradykinesia and tremor attenuated by the levodopa treatment.

By way of example, drug effects on bradykinetic accelerometer scores are shown in the Fig. 6. In the following, automatic detection of motor states will be exemplified. The automatic training was based on unsupervised deep-learning models, variational autoencoders, VAEs, on accelerometer signals. By way of example, the use of a commercially available wearable device together with a mobile application on a smartphone allows for capturing of data that enable reliable assessment of motor fluctuations and dyskinesia in Parkinson's disease, PD, patients in a cost-effective manner. Both the wearable device and the mobile app were set to use the same clock server to ensure accurate labelling of the data for a subject.

With reference to Fig. 6, the variational autoencoders, VAE, were trained on both on measured accelerometer data and on data from databases to detect movement states. In more detail, the data was based on continuous accelerometer data from the wearable device for different task. The experimental data was acquired for participants wearing a wearable device in the form of a smartwatch. The smartwatch comprising an acceleration sensor for registering acceleration data pertaining to movement data for each of the participants.

To get scalar measures of symptoms for replicates as seen in the accelerometer data, trained ordinal (linear) regression models for each symptom, on spectral features of the accelerometer data were utilized. The models were L2- regularized with the regularization coefficients cross-validated. Thus, three separate models, one for each symptom type (tremor, bradykinesia, dyskinesia) where developed. The results were then scored with the models to get an indicator of symptoms, parallel to the observer scores, that is a function of accelerometer data only. With the observer and accelerometer scores available, symptoms, as defined by those scores, were studied in relation to drug state and task, with the subject-level variation controlled.

Fig. 6 illustrates experimental data obtained using the model as described above. Information pertaining to the drug effect (Levodopa infusion) was obtained for the subjects participating in the experiment. The Levodopa treatment intake and subjective symptom data were collected with a mobile device application from subjects with Parkinson's disease, PD. The graph 600 illustrates drug effect on the symptom severity as predicted from accelerometer data (x-axis), and relative strength of the effect (y-axis) at different tasks are illustrated. Put differently, the x-axis shows the difference from the drug state (i.e. Levodopa infusion), and the y-axis the strength of the symptom signal as seen by the linear spectral model. Scales on the axes are commensurable but otherwise artificial.

The effect of Levodopa (x) shows the effect of a well-controlled manipulation in a constant movement state. Differences across different tasks may be observed such as limb movement 602 , e.g. arm movement, and coordination movement 604, e.g. finger/hand movement. Also different types can be identified such as standing still 606, walking 608, or fine motor skills 610.

From the analysis of the experimental data analysis of Levodopa response it is observable that movement representation may affect the visibility of PD symptoms. Additionally, it is observable that unsupervised variational autoencoders, VAEs are efficient in determining and discriminate between different movement representations.

As discussed above, the use of a two-model approach as suggested herein comes with a number of advantages. An advantage of the having the two-model approach, the first model, trained based on movement data from multiple subjects, can be used for identifying movement representation reliably, e.g. categories of movement such as walking, standing, sitting down.

By way of example, the movement representation may without any limitation to these, pertain to a task such as: standing, sitting, lying, walking, moving limb, performing coordination movement or fine motor skills.

The wording movement representation may in some examples be referred to as a category of movement. By knowing the category of movement, it is in turn possible for the second model, which is subject-specific, to take this into account when identifying the physical state, which in turn provides for that the physical state can be identified more reliable.

Fig.7 illustrates an example on how the data analysis of for example symptom data may be performed after the identification of a movement representation. Fig. 7 illustrates by way of example how the physical state 210 can be determined based on the movement data 106a captured by the wearable device 104a worn by the subject 102a as illustrated in Fig. 1. Put differently, Fig. 7 illustrates one potential high-level scheme 700 on how data analysis of, for example, Parkinson's disease symptoms, may be performed in an efficient manner after the detection of a movement representation according to some examples.

In Fig. 7, two different paths 702 and 704 of the data analysis are illustrated. The first path 702 corresponds to a situation when the movement representation 202 is known for a given set of movement data 106a. The movement data 106 may be accelerometer data pertaining to a movement of a subject. The movement data 106a may pertain to accelerometer data from the wearable device 104a. Put differently, the category of movement or type of movement of the subject may be known. The movement representation 202 may thereby be linked 706 to the movement data 106a. The movement representation 202 may, for example, be defined by the subject. By way of example, the subject may use a mobile phone, not sown, or any other device provided with a user interface to define the movement representation 202. The movement representation 202 may, for example, be selected among a list of movement representation in a user interface. The movement representation 202 may alternatively be defined by an activity in a scheduling program such as a calendar. The movement representation 202 may alternatively be defined by an instruction given to the subject to engage in a task such as walking, standing up lying down, drinking from a glass or cup etc.. In other examples, the movement representation 202 may be known from other sensor data such as pulse, skin conductance, heart rate variability, etc. Put differently, when the subject is doing a specific and known physical task, the movement representation 202 may be known and there is no need to analyze the movement data 106 by the first model 200. Thus the movement data 106a may be analyzed by the second model 208. A weighting model 204 may optionally be used for generating a weighted movement representation as described above. The movement representation and/or the weighted movement representation may be introduced into the second model 208. It may be noted that the movement representation 202 may be sent to the second model 208 without utilizing the weighting model as further discussed below. The second model 208 may be trained by requesting data from the symptom data from the subject, either by posing direct questions, but also in asking the subject to perform certain physical tests or exercises, as discussed above. By having access to both movement data 106a linked to these tests and also symptom data linked to these tests labelled data is achieved, which may be used for training the second model 208.

The second path 704 illustrates a situation when the movement data 106a is not associated with a known movement representation. Such a situation may for instance occur as no information to the movement representation has been gathered. Alternatively, since movement of limbs may vary from subject to subject, using the movement representation 202 in isolation may result in that subjects having atypical movement data may be provided with an incorrect physical state, and as an effect possibly a non- optimal dosing regimen. In more detail, in the second path 704, the first model 200 serves the purpose of linking the movement data 106a to a movement representation 202. More specifically, the movement data 106a can be linked to one of a number of movement representations 202 formed based on information gathered from a large number of subjects. The movement representation 202 may thereafter be sent to the second model 208. A weighting model 204 may optionally be used for generating a weighted movement representation as described above. If so, the weighting model 204 can be provided with the movement representation 202, output from the first model 200, and the movement data 106a. By having access to the movement representation 202, based on the first model 200 being based on data from a plurality of subjects, and the movement data 106a, based solely on the subject 102a in question, a weighting model 204 may be used for adjusting the movement data 106a such that findings made by the first model 200 can be made available to the second model 208. After having assessed a weighted movement representation or the movement representation 202 in the second model 208, the physical state 210 may be output. As described above, this information may in turn result in that notifications, e.g. changed dosing regimen, instruction to rest, diet instructions (nutritional composition, protein composition, eating schedule etc.), exercise instructions and sleep instructions, are transmitted to the subject 102a and/or other party admitted access to this information.

Fig. 8 is a flowchart illustrating a method for determining the physical state based on the movement data. With reference to Fig. 8, a method 400 for determining a physical state of a subject by using movement data 106a-c obtained via a wearable device 104a-c worn by the subject 102a-c may be provided according to some embodiments. The method 400 comprising: identifying 402 a movement representation 202 among a number of movement representations based on the movement data 106a-c using a first model 200, wherein the first model is a generative model using data from multiple subjects, and identifying 408 the physical state among a number of physical states based on the movement representation 202 using a second model 208, wherein the second model 208 is subject-specific. The movement representation 202 may pertain to an activity by the subject, the activity may be one or more selected from the list of: standing, sitting, lying, walking, moving limb, performing coordination movement, and performing fine motor skills.

The movement data may pertain to acceleration data.

The accelerometer data may pertain to a movement of a limb of the subject.

The accelerometer data may comprise data pertaining from a three-axis accelerometer.

The wearable device may be a smartwatch. The wearable device may be worn by the subject. The wearable device may be worn on the wrist of the subject.

The accelerometer data may pertain to a movement registration by a wearable device.

The physical state may be linked to a state of disease and/or a state of symptom.

The first model may be a variational autoencoder, VAE.

The accelerometer data may be subject to spectral processing prior to being used as training data for the VAE.

The state of disease and/or the state of symptom may be a disease and/or a state associated with a movement disorder.

The disease and/or state may be selected from a list consisting of Parkinson’s disease, epilepsy, Multiple sclerosis (MS), Alzheimer’s disease, dementia, chronic or acute musculoskeletal pain and myopathy.

The disease may be Parkinson’s disease and the state is on state or off state.

The method may further comprise: requesting 410 symptom data 110a-c from the subject 102a-c, receiving 412 the symptom data 110a-c from the subject 102a-c, and linking 414 the symptom data 110a-c with the movement data 106a-c.

The method may further comprise: comparing 416 the movement data 106a-c with trigger data 306, in case of match 418, triggering the step of requesting the symptom data 110a-c from the subject 102a-c.

The symptom data 110a-c may comprise motor symptom data related to tremor, rigidity, bradykinesia, dyskinesia and/or balance, and/or non-motor symptoms related to sleeping disturbances, anxiety, dizziness, hallucination, changes in ability to smell or taste, urinating, digestion, pain, fatigue and/or depression.

The method may further comprise: requesting 420 substance intake data 300 from the subject 102a-c, receiving 422 the substance intake data 300 from the subject 102a-c, and linking 424 the substance intake data 300 with the movement data 106a-c.

The symptom data 110a-c and/or the substance intake data 300 may be requested by transmitting a symptom data request and/or a substance intake data request to a mobile phone 108a-c assigned to the subject, and wherein the symptom data and/or the substance intake data is received via the mobile phone 108a-c.

The physical state may either be a state with positive response to substance intake and reduced symptoms, determined based on the movement data 106a-c registered after the substance intake data 300 is received, or a state with no or negative response to the substance intake and symptoms remained or worsened, determined based on the movement data 106a-c registered after the substance intake data 300 is received.

The second model may be a regression model or a classification model, such as a logistical regression model, a convolutional neural network (CNN) or a random forest model.

The method may further comprise: determining 426 a suggested time for the substance intake, a suggested amount for the substance intake and/or a suggested type of substance for the substance intake based on the physical state 210.

From the description above follows that, although various embodiments of the invention have been described and shown, the invention is not restricted thereto, but may also be embodied in other ways within the scope of the subject-matter defined in the following claims.