DESCRIPTION

Field of the invention

The present invention has its application within the information and communications technology sector, more specifically, relates to the deployment of tools which use data measurements from users’ portable devices (e.g., applications in mobile user terminals such as smartphones or tablets, and wearables user devices such as smartwatches, etc.) for detecting users’ deviant behaviour.

More particularly, the present invention refers to a system and method for extracting behaviour routines based on the processing of sensors data and active data and using topic modelling, and generating alerts based on deviations of actuals from the extracted routines. The system can work systematically with a therapist and/or a user/patient in improving his/her mental health.

Background of the invention

People's routines can vary significantly in terms of movement, socializing and locations visited and in general they are much influenced by their working shifts and hours.

In existing systems that track people's routines for mental health applications using smartphones, basic statistical information derived from the distribution of mobile sensing data is used to determine these routines. While this was a great start to obtain routines passively from smartphone data, without the use of direct user inputs such as diaries, they are too simplified to obtain a complex user entity such as routines for the detection of deviant behaviours. Recent methods use contexts, such as time of the day and also extract more complex features such as circadian rhythms and regularity indices to approximate real world behaviours and contexts through which better routines can be obtained. However, they fail to use the wide array of formidable mathematical tools that can help in the closer and more accurate approximation of a user’s routine.

Moreover, understanding when an alert should be provided by observing deviant behaviours was done in a simplistic manner. Methods established a threshold based on statistics and checked when an extracted feature exceeded this value significantly.

An example of a method and apparatus for modeling behavior using a probability distribution function is disclosed by US7202791B2. In this system, behaviours (in particular, activities) of an individual are modelled using the probability distribution functions of sensing modalities. Using basic thresholds, obtained from the statistics of the distribution function, the system flags if the user is engaged in a deviant behaviour that’s away from their consistent/normal state of behaviour. An alert is established when a deviant behaviour occurs. This system uses basic statistics to establish regular routines of behaviours. Routines are significantly complex behaviours that are dependent on many factors, and basic statistics of sensing measurements cannot represent human routines accurately. This system’s estimation of deivation behaviours is also simplistic and replies on the same statistical information.

US8606497B2 discloses a method for detecting and monitoring human activity utilizing location data. This is a basic method that is used to infer a person’s engagement in an activity based on their location and time context. The method extracts a user’s activity pattern over space and time using location data obtained from a personal communication device (mobile phones). This method obtains features from the GPS data to establish a location and time context used to derive a person’s engagement in a certain activity, but it does not establish any routine that the user follows or how their behaviours can deviate.

“Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis” by Canzian, L, et al. (Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing, pp. 1293- 1304, ACM, September, 2015.) describes how smartphones can be used to unobtrusively monitor a user’s depressive states. It obtains various features engineered from smartphone sensing data related to this purpose. One of these features is called routine index and is extracted from location data. This feature compares the number of locations visited by a user during a given time instance against the number of features visited by the same user during other time instances of the same interval. This is a clever way of establishing how regular the user is, in his/her location patterns, and thus creates a routine of sorts. However, this uses only location data and cannot be applied to other features or location together with other sensing data. Canzian, L, et al. do not provide the ability to use multiple multiple sensing modalities. Canzian, L., et al. use a single index that defines the user’s routine but cannot obtain more in-depth features.

“Sensing behavioral change over time: Using within-person variability features from mobile sensing to predict personality traits” by Wang, W., et al. (Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3), page 141 , 2018) also uses time slots and obtains features within each slot to create a single feature like the regularity index. This paper looks at differences in sensing data within each person to create features (standard deviation, regularity index) that relate to one’s routines. The regularity index is obtained using the cross-correlation of hourly sensing data over two-day pairs. This is then aggregated to obtain how regular the aggregate sensing features of a user are - describing how regularly a user behaves.

“The Social Rhythm Metric: An instrument to quantify the daily rhythms of life” by Monk, T. K., et al. (Journal of Nervous and Mental Disease, 1990) describes a psychometric instrument to obtain the routines and rhythms of a person’s life, using a self reported questionnaire. This method has been used by the Psychological community to help users tackle behaviours that are against their routines, however such methods are not popular with users. While accurate, questionnaires like these are extremely tedious for a user to complete everyday. The pervasiveness of smartphones allows us to tackle this problem head on and not be a nuisance to users.

Therefore, there is a need in the state of the art for providing a personalized detection of abnormal behaviour related to clinical conditions using measurements from the users’ smartphones and/or other portable and/or wearable devices. Summary of the invention

The present invention solves the aforementioned problems and overcomes previously explained state-of-art work limitations by providing a method that extracts the user’s routine automatically from measurements obtained by the phone and wearables for each individual in a way that is personalised to the users’ work shifts. The method defines ranges of low, medium and high for each measurement and generates documents where each word represents the time slot and sensor-based feature levels for that time slot. The method obtains personalized time slots for each user based on their past mobile usage patterns and active data information Further, through topic modelling, the most frequent topics are computed and thus the routines are automatically extracted, based on previously defined time slots and measurements’ levels. Furthermore, the system computes deviations from the routine and alerts when these deviations are sufficiently high, suggesting that the user has deviated from their expected routine. This alert could be used by a clinician as an indication that a check up is due.

The present invention can be applied to clinical systems so that the clinician can be aided by a dashboard of the routine of the user and their historical alerts, receiving automated alerts without the need of continuously monitoring and visualising dashboards of users’ data. Furthermore, the present invention can be applied to (sub- clinical envisioned systems where stress, depression and/or anxiety are being measured, and the routines extracted from the users’ data can shed some light on why the users’ level is particularly high or low depending on their extracted routine.

The present invention takes into account various contexts about the users, including when they are working or not working, whether they are at home or not, whether the sensing data was obtained during the week or the weekend, etc., to name but a few examples. Using these contexts, a smart data multiplexing method is proposed to extract sensor/feature levels in these time slots, and construct words and documents. Subsequently, topic modelling, which is a robust method for context estimation and data mining, is used to extract routines.

Routines: The proposed method first automatically extracts routines from users’ data through the use of topic modelling for extracting most frequent topics and keywords from a set of words constructed artificially based on the measurements’ levels. None of the existing systems consider forming words with measurements’ levels. Moreover, this method goes beyond the standard state of the art where topic modelling has been applied to location for mobility patterns, as it constructs novel keywords; where a combination of personalized time slot allocation and different sensor-based feature levels, tailored to the users’ distribution of past values, are captured. Moreover, the method goes beyond aggregated measures such as mean and standard deviation routine indexes that are most commonly found in the literature by extracting the routine automatically through most frequent sensors level detection within a time slot defined personal to the user.

Alerting: The proposed method sends alerts when a significant deviation occurs from the routine extracted in the current data. This deviation takes into account, through weights, the correlation of the measurements with the target feature (e.g., depression level) and also the distance from the routine level (e.g., noise level currently is high compared to low in the routine). None of the existing systems take into consideration such computations in the alert mechanisms. Moreover, multiple weights vectors can be added, allowing the alert mechanism to have personalized and tuned alerts for each condition monitored (e.g., depression, anxiety, wellbeing measures, etc.).

A first aspect of the present invention refers to a computer program implemented method for detecting human abnormal behaviour, comprising the following steps: data preprocessing which comprises: defining time intervals for a user by extracting time intervals based on historical data from raw sensors and personalizing the extracted time intervals based on active data input by the user or inferred data from the raw sensors; constructing words for each defined time interval, wherein a word is constructed as a vector comprising the defined time interval and a plurality of sensor-based feature levels, a sensor-based feature level being mapped to a range of values of a sensor-based feature extracted from historical and real raw sensors data; constructing a text document with the constructed words based on the defined time intervals and the mapped sensor-based feature levels; extracting routines for each defined time interval by extracting the most frequent words from the constructed documents using topic modelling to obtain a routine vector; and alert generation comprising triggering an alert (428, 605) if a deviation is detected between the sensor-based features extracted in a current time interval and a past routine extracted for a past time interval previous to the current time interval.

A second aspect of the present invention refers to a processing system configured to implement the method described before.

The method and system in accordance with the above described aspects of the invention has a number of advantages with respect to the aforementioned prior art, which can be summarized as follows:

None of the existing techniques consider combining measurements’ levels with topic modelling for deriving new insights into users’ routines and for alerting when deviations from these routines are significant, making the proposed method of deviant behaviour detection more robust and allowing the possibility to tune the alerting mechanism for multiple conditions.

While existing techniques use basic statistics from sensing data, along with basic time contextualisation, and create alerts using simple estimations based on distribution statistics, the proposed in-depth method for extracting routines significantly improves over the state of the art as a complex understanding of routines is established. Also, the method goes beyond the state of the art in triggering alarms by understanding how much each sensing information is related to the output variable, as it uses a combination of weighted distances from the calculated routines to trigger alarms.

Personalisation is a key advantage of the invention since knowing the user’s routines can help a clinician to obtain relevant information about the user’s clinical conditions as follows: o the right time to intervene with targeted interventions: i) based on location (at home, work or outdoor); ii) based on deviations (e.g., a check-up from the clinician when the deviation exceeds the normal amount); o Insights to the user, coupled with their stress, depression and/or anxiety levels, observing which routines are helpful for reducing their symptoms.

The proposed method/system does not rely on raw sensor values, but levels, which enables it to be ported to different products with minor adjustments.

These and other advantages will be apparent in the light of the detailed description of the invention.

Description of the drawings

For the purpose of aiding the understanding of the characteristics of the invention, according to a preferred practical embodiment thereof and in order to complement this description, the following Figures are attached as an integral part thereof, having an illustrative and non-limiting character:

Figure 1 shows a block diagram of the system architecture, according to a possible embodiment of the invention.

Figure 2 shows a schematic representation of an output dashboard of results provided by the system.

Figure 3 shows an example of a word constructed by the system based on time slot and sensor feature levels.

Figure 4 shows a flow diagram of the system, according to a possible embodiment of the invention.

Figure 5 shows an example of a word constructed by the system based on raw values of the sensor features and levels of the sensor features. Figure 6 shows an example of an alarm generation by the system based on a routine extracted as one keyword for a specific time slot.

Preferred embodiment of the invention

The embodiments of the invention can be implemented in a variety of architectural platforms, operating and server systems, devices, systems, or applications. Any particular architectural layout or implementation presented herein is provided for purposes of illustration and comprehension only and is not intended to limit aspects of the invention.

Figure 1 shows an overview of the system architecture for detecting abnormal behaviour, according to a preferred embodiment, where the main blocks are: data preprocessing (110), routines extraction (120) and alert generation (130).

The system uses raw sensors data (101) coming from different input data sources:

Mobile phone sensors (e.g., screen unlock, accelerometer, light, noise, steps, GPS, battery etc.) from which additional measurements can be derived (e.g., number of steps, number of locations visited, noise level exposed to, light level, battery consumed, etc.). Some of these can be proxies for health-related measurements, for instance time spent at home is considered a proxy for social isolation looking at social anxiety and other clinical models.

Wearables sensors (e.g., PPG, EDA, thermometer), from which physiological measurements and biomarkers can be derived (e.g., heart rate, heart rate variability, blood pressure, temperature etc.)

The measurements extracted from raw sensors data can be also referred to as sensor-based features or even simpler features, a common term used in machine learning for referring to input and target variables.

In addition, the system active data (102) can use as input; i.e. , data that the user inputs manually, such as what is their work shift, personality through active forms/questionnaires. In further detail, the system blocks depicted in Figure 1 perform the following steps of a method for detecting abnormal behaviour by:

Data Preprocessing (110): a. Interval definition (111): Firstly, the method extracts relevant time intervals or slots taking into consideration typical country routines or specific user routines in a way that is personalised to the user. For personalization, a) user’s input active data (102) can be used through a questionnaire to specify their working hours and typical home hours, or b) a machine learning-based approach to capture raw sensors data (101) related to the work, outdoor activities, home (i.e., location categories) can be used to infer how the user spends their time and thus define the relevant time intervals in a personalised manner. b. Word construction (112): Further, data is split based on the time slots extracted from the previous step of interval definition (111), either using active data (102) input by the user or inferred from the raw sensor data (101).

Based on the raw sensor data in a defined time slot, the method computes aggregations of sensor-based features (e.g., steps sum) and, based on their distributions, the method maps ranges of values into different levels for each sensor-based feature, for example: low, medium and high levels.

The sensor feature levels can be expanded in more categories, depending on the quality of the raw sensor data.

Furthermore, based on the sensor feature level, a word is constructed as a tuple <time slot, feature_1_level, ... , feature_n_level>; feature_i_level being in {H, M, L}, where L stands for Low, M for Medium, and H for High, and i = 1, 2,... ,n, where n represents the number of features built based on the sensors raw values.

An example of such a word (200) is shown in Figure 2. Figure 2 illustrates an example of the word (200) constructed as a tuple based on time slot (t _x ) and the three distinguished sensor feature levels: high level (H), medium level (M) and low level (L). The words follow the same order (e.g., in the example shown in Figure 2, the second letter always represents the level of the noise). c. Document construction (113): Furthermore, based on the sensor feature levels and time intervals defined, text documents are built based on the words (200) constructed in the previous step Routine extraction (120): The time-slot routine is then captured using topic modelling by extracting the most frequent words from the constructed documents.

- Alert generation (130): Deviations from time-slot routines then can easily be computed by observing current pattern for the time-slot versus the routine calculated. Thus, the method then detects when there is a deviation from the routine captured before and alerts when there are irregular patterns. The alert mechanism takes into account both the distance from the routine extracted, and also the weights vector which represents correlations observed between the target feature (e.g., anxiety level, depression level, well-being score) and the sensor-based features.

A system dashboard (300) presenting the results output by the proposed system is depicted in Figure 3, where the alerts (310), history charts (320), routines (330) can be visualized, and different time slots separations can be set or configured (340) by the end user. Moreover, the information extracted on a user’s routine can be constantly provided to the user through the display on his/her portable devices (smartphones, tablet, smartwatch etc.) and so the user can also monitor his/her daily behaviours. This personal monitorization can persuade the users to change their behaviours so that they can stay in range of their regular routine behaviours. Additionaly, alerts can be provided by beeps on their portable (and wearable) devices or change in regular user interface during exceptional situations, or using any other (visible, audible,...) means for notifying/presenting the alerts to the end user. This alert presentation on the users’ devices trigger them to alter their behaviours significantly.

The core blocks of the system architecture, shown in Figure 1 , are described in detail below and illustrated in the flow diagram of the system depicted in Figure 4 as follows:

Data Preprocessing (110, 400): The data preprocessing is applied on a per user basis. The data of each user is processed applying the aforementioned three steps, a) interval definition (111), b) word construction (112), and c) document construction (113), further detailed below:

Intervals definition (111, 403):

For user U _k , k — 1, . .. , K a number of types of days, Num_type_days

, is defined. Each type of day ( TD ) is divided into N _TD non-overlapping intervals. Those intervals are personalised to the users in a way that they are meaningful to them (e.g. taking into account different work- shifts). In a possible embodiment, there are three ways to define the intervals: i) Based on the country where the user lives (and equal for all users within the country): the typical routines followed in the country where the user is located, obtained from historiacal raw sensors data (401) are used to define the intervals. For example, each weekday can be divided into three intervals: Morning 7h00-13h00, Afternoon 13h00-22h00 and Night 22h00-7h00. ii) Personalized per user with active feedback from user: The definition of the time intervals also use the user’s active data (402) input through a questionnaire in which the user fills how a normal day could be divided; e.g. for user : Monday to Friday, y ₁ does Oh-

8h at home, 8h-9h commuting, 9h-17h working, 17h-18h commuting, 18h-24h home; on weekends, U _t usual splits is 0h-1 Oh sleeping, 10h-13h shopping and housework, 13h-15h lunch, 15h- 20h afternoon at home, 20h-0h going out. iii) Personalized per user with inferred intervals: Use a machine learning algorithm to derive the usual routines of the user, e.g., based on location by capturing when the user is normally static at the same area (applying a clustering technique based on time and location) and when does their typical transitions between areas. This approach can be used to define automatically relevant intervals of time that are user specific. Word construction (112):

The word construction process consists of two main steps: 1) feature computation (404) and 2) discretization by assigning levels (405) to the computed features for the construction (406) of the words and the documents with these words.

1) Feature computation (404):

Once the specific intervals are defined for each user, for user k, the system computes S features per interval and day as aggregations of the sensors data, denoted by

( ^F k, i, d, 1. ^F k, i, d,s) ^{for interval} ' ^{and da} y ^d - Examples of aggregated features are: percentage of time spent in a certain location, battery consumed per minute, number of steps per minute, average light level, etc.

2) Word construction by Discretization:

Then, the system assigns or extracts (405) a level

^F k i d s· f° ^{r eac} * ¹ i ^nterva l i _> day d and features, based on the value of the feature during that interval and how it compares with the values at the rest of the intervals. For example, the levels can be Low (L), Medium (M) or High (H) and the method to determine the mapping of the feature value to one of these three levels can be as follows:

For each feature j ^ _s , the empirical distribution of its value is obtained using the values of all the intervals.

Then, ^F k i d s ' ^s below the 33rd percentile,

Lp, . , = Low. If the value is between the 33rd and 66th k, i, d,s percentile, L _F . , = Medium . If the value is above the k, i, d, s

66th percentile, For user k, a given time interval t, and day d, the construction (502) of the word (500) for the raw sensor feature values (501) is defined as:

^{Where Lp} k, i, d,s ^{e {ίΐ} ’ A ^s . ^1l ’ ^{s} and F} k, i, d, 1. ^F k, i, d,S ^{are the} S ^{feature values} for the time interval i and day d.

The result of this process is a word vector (500) with the time of the interval (t _k ) and a level {H, M, L} for each of the features, as shown in the example of Figure 5, following the three levels mapping of the example described above; i.e., {H, M, L}, where L stands for Low, M for Medium, and H for High. Figure 5 shows how the word is constructed by the system based on both the raw values of the sensor features and the extrated levels of the sensor features. The order of the features is preserved for each defined interval, so that in the next step of the process the word vectors can be comparable.

Document construction (113):

The constructed in the previous step for user k are then grouped together to form N documents, denoted by Dk = {¾ i 4 2'···'4 w)· The documents can be created by grouping the words, according to a possible embodiment, in at least one of the following three different ways:

• Each document is the sequence of words of a certain day; i.e., a day would be “<ti, L, M, M, H, .., M>, 2, L, M, M, H, .., L>, ... , <t _m , M, H, ... , L>”.

• Each document is the sequence of words of a certain day of the week during a certain time frame. For example, a document is the sequence of words of all the Mondays during January 2018, another document is the sequence of words of all the Wednesdays during April 2017. • Each document is the sequence of words of a certain type of day during a certain time frame. For example, a document is the sequence of words of all the weekdays during the week 34 of 2019.

Routines extraction (120, 410) through topic modelling:

Once the words and documents are obtained for user k, the time-slot routines are captured using topic modelling by extracting the most frequent words ^w k i d documents D^. The output of the topic modelling step is given in terms of a relationship between topics and words in the dictionary, e.g. through a conditional probability distribution.

This can be done with standard topic modelling methods such as Latent Dirichlet Allocation, LDA; e.g., LDA using Bag of Words or LDA using TF-IDF (Term Frequency-Inverse Document Frequency). Topic modelling involves a preprocessing of the data contained in the documents. In a possible embodiment, the Routines extraction (120, 410) is performed by applying LDA with bag of words. The application of this LDA method involves these three steps, 1) dictionary creation, 2) bag of Words encoding and 3) the application of LDA itself, as follows:

1) Dictionary creation (411):

This step involves indexing the unique words in the documents. For example, it can be done by listing the set of unique words in all documents and indexing them consecutively to create a table as the Table 1 below:

Table 1: Dictionary creation for document d*

2) Bag of Words encoding:

For each document, the words within the document are mapped to the index using the dictionary and the word count or order within the document as [wordjdx, word_count]. Alternatively, the TF-IDF score can be considered.

For example, for the document d _k in Table 1 : d

^¾ = {<ti, L, M, M, H, M>,<t_2, L, M, M, H, L>, < ti, L, M, M, H, M>, ... , < t _m , M, H, ... , L>} the following bag of words is encoded:

B o V _a

= { [0, 2] , [2, 1] . [N_w, 1]}

BoW _a where denotes the bag of words encoding for the set of pairs [wordjdx, word_count] for each word in a document d.

3) Application of LDA to extract topics (412):

The LDA method is applied to the bag of words encodings of all

D k the documents of user . The input to the LDA algorithm is

D _k typically the bag of words encodings of and a predefined

T number of topics . The LDA algorithm uses dirichlet priors for the document-topic and word-topic distributions to establish a generative model and fit it to the data. With LDA, the method can

Y extract topics (412) from a document corpus, where each topic is human-interpretable and characterized by the words that they are most strongly associated with. The characterization is done in w _s terms of the conditional distribution for each word to appear in d d Y a document , given that is of topic . For example, using common words instead of those in Figures 2 and 5, a “topic 2” could be characterized by having a higher probability of generating terms such as “oil, gas, drilling, pipes, Keystone, energy,” etc. and not others within the dictionary. In the context of the invention, the topics are the common routines or behaviours characterizing types of days and its relationship with the defined words (200, 500) shown in Figures 2 and 5.

The LDA method uses variational inference to fit the generative model, predefined, which allows to characterize an approximation of the posterior distributions of the model word-topic to the available data in the corpus. d _j d _j

Furthermore, for a new document , the probability of belonging to each topic can be inferred using the characterized d _j conditional probability given the words in , e.g. 5% topic 1, 70% topic 2, 10% topic 3, etc.

Based on the dominant topics, the routines of the user are extracted (413) for each defined interval.

Alternatively, other topic modelling methods could be applied such as LSA (Latent Semantic Analysis), PLSA (Probabilistic Latent Semantic Analysis) and lda2Vec (LDA in Deep Learning).

Alert generation (420)

The alert mechanism takes into consideration the following two aspects: i) Distance, d, is computed (425) as the absolute difference between the routine vector and current level vector, where each level is mapped to the following numbers, for example: High level to number 3, Medium to number 2, and Low to number 1. As such, for instance the distance between Low (in routine) and High (in actual) for the noise level is 2. The current level vector, extracted as a word or keyword, is constructed (424) based on the sensor-based feature levels previously defined for each feature computed (423) using the real-time raw sensor data (421). ii) Weights, w, are computed (422) from the correlation between the sensor-based feature and the target feature (e.g., depression level). Thus, this gives priority to features that are higher correlated to the target feature (e.g., steps sum, or sleep duration compared to battery level for depression).

Through this, the goals of the alarm generation (420) mechanism are:

Diversity: by utilizing a weights vector per target variable, the system is able to adapt for detecting signs of various abnormal conditions through the use of correlations between the features and target (e.g., anxiety level, depression level, well-being score).

Accuracy: to be more accurate as the routine is captured automatically and

Personalization and granularity: to specify at a much more granular level what part of the user’s routine has been affected, in terms of being captured per interval, per individual and per sensor level.

Based on the calculated distance vector and weights vector, the method can determine the features that are most different: steps sum, accelerometer- based energy, time spent at home, etc.

An alert measure a is computed (426) as: å w, * d _t a = where i= 1, 2... n, n representing the number of features built based on the sensors raw values,

Wj represents the weight as the correlation between the feature i and the target feature, and d I represents the distance between the current level and routine level for feature i.

The computed alert measure a is compared with a threshold value (427), such as the average, 75th percentile value etc., and the alerts are raised or triggered (428) when the alert measure a is higher than the threshold value determined

(by the system or by the user). The user can label the alert as true or false, and then the system can learn the threshold as it goes. Moreover, the threshold can also be set by the clinician or end user.

Figure 6 shows an example of the alert mechanism in a possible embodiment. An example of a routine extracted as one keyword (600), along with the weights vector (601), is illustrated for a time interval defined as the current morning. The distance vector (603) is computed as the distance between i) the keyword (600) extracted from the most dominant topic extracted for each time interval, and ii) the actual keyword (602); i.e, the absolute difference between the routine vector and the current levels vector for the current morning, as shown in Figure 6.

Based on these, the alert measure is computed (604) and based on an identified threshold (o), an alert is triggered (605) when this morning’s sensor features levels are too divergent from the extracted routine.

Note that in this text, the term “comprises” and its derivations (such as “comprising”, etc.) should not be understood in an excluding sense, that is, these terms should not be interpreted as excluding the possibility that what is described and defined may include further elements, steps, etc.