Field of Art

Aspects of the present invention relate generally to systems and methods for combining human signals to estimate affective, physiological or behavioural states ("human states").

Background

Human-machine interfacing is an area of increased interest. In particular, situations where the human body generates biometric and emotional data which can be identified and then introduced to computer software and hardware, such that it can dynamically respond, are of increasing, wide-ranging relevance to the technological community and beyond. Selected wider applications include sports and performance, health and safety, automotive and mobility, gaming and entertainment, and well-being, to name a few examples.

Bio-emotional monitoring can be used for enhancing user experiences, personalising engagement and feedback, as well as performance monitoring. This could be for health purposes, emotional understanding, or for scenarios where fatigue, stress or intoxication could be life threatening. For example, in Formula 1 (F1) driving, even marginal gains in performance e.g. due to different levels of tiredness can provide a crucial advantage as well as enhance safety and comfort.

Human state including affective state monitoring is an increasingly important factor in the evaluation of human activities. It is useful not only from an individual perspective-by providing individual feedback on psychological processes related to the experience of events-but also at a collective level in helping experts to support and understand an individual's psychological processes. Other applications which could benefit from human state monitoring include automotive ride-share applications (to assess and improve customer experience), e-sports and affective gaming (that is, gaming which responds to the feelings of a player) involving dynamic play/sensitive content, as well as consumer robotics (e.g. emphatics robots in the household industries).

Human states may include affective, physiological or behavioural states. Affective states are related to detectable human signals, both physiological and expressive-these can be monitored by specific devices. However, these indicators of affect have typically been underestimated or overlooked, often due to their lack of consistency and reliability, even though the importance of psychological conditioning is commonly acknowledged. Even if the evaluation of affective states is possible in lab conditions, it remains a challenge in different contexts. The reliability of the data drawn from these signals is often questioned. Signals can be interrupted by technical limitations, disrupted by added noise from irrelevant effects, or be the subject of interpretation errors. There are also additional problems that come from the environment, with extreme conditions for example regarding temperature, vibration, speed, or G-forces.

In order to evaluate human states "in situ" (i.e. beyond the laboratory in different contexts such as home, in the mobility industries or in outdoors activities), it is desirable to evaluate data from a range of devices and to relate them to contextual proprieties. Smart phones and other devices incorporate sensors, transducers and other components for sensing and monitoring a number of parameters including video, motion, location, acceleration, orientation, temperature, pressure, acoustic and optical conditions etc. In addition to smartphones, there are a growing number of "wearable devices" or "wearable sensors" that can be incorporated into personal clothing or other objects such as jewellery, watches, earbuds/headphones etc. and which may also detect, record and transmit data about the environment or person wearing them.

With the development of such devices it is now possible to assess the dynamic progression of both physiological signals such as heart rate, breathing rate or galvanic skin response and expressive signals such as body movements, voice and facial expression in places and during certain activities that were previously prohibitive in terms of cost and reliability. However, despite the technical advances, the analysis of such signals in a psychological perspective remains a challenge. In particular, the interpretation of changes in a human's signals can sometimes be contradictory. Through the combination of physiological and expressive signals and by interpreting which affect or emotion they are conveying, it is possible to understand a person's reactions in a specific context and to provide accurate individual feedback. It is to these problems, amongst others, that aspects according to the present invention attempt to offer a solution.

Summary of Invention In a broad, independent aspect, a system comprises a plurality of devices for sensing, detecting or measuring physiological and expressive data streams associated with a person's affective experience over time, the system comprising a processor configured to:

synchronise the data streams to obtain synchronised data;

select feature data from the synchronised data, the feature data being descriptive of at least one human state of the person; model the feature data using a locally weighted polynomial regression model; and process the modelled feature data using multivariate autoregressive state-space modelling in order to identify at least one potential trend in the experience over time.

Physiological data, such as heart rate, breathing rate or galvanic skin response, and expressive data, such as text/emoji, body movements, voice and facial expression are relevant information in the assessment of human affective states. Contextual data such as location, speed, acceleration and video recording can also provide important insight into the context of these affective experiences. In addition to audio and video data, infra-red sources such as camera data can also provide contextual characteristics. "Human states" refers in this context to emotional, behavioural, or physiological characteristics (or a combination of these) The at least one human state may represent at least one of valence, arousal and dominance of the affective state of the person, for example. The at least one human state may include at least one physiological or behavioural characteristic such as stress, fatigue, intoxication, distraction, positivity. Intoxication and fatigue are of particular importance in automotive use cases related to driving safety for example.

The processor may represent a series of modules, wherein physiological and expressive data are treated by the specific computer based modules to extract information e.g. the valence, arousal and dominance (here after called VAD) which are universal dimensions used to characterise affective states.

These dimensions may then be compared to contextual data in order to obtain meaningful insight about human appraisals of affect related events. In that perspective, a computer- implemented method for combining physiological and expressive data and to relate the obtained affective states to the context may comprise the steps of: receiving data from multiple devices, analysing their characteristics, combining these multimodal data streams with contextual information. The system also optionally provides solutions to display the states with visual, haptic and sound, "data humanisation" and environmental feedback. By "data humanisation" we mean the conversion of a given data set into an object or action embodying the meaning of that data set in a manner comprehensible to a person, such as an end user. This might take the form of an informational cue or alert or some other feedback to the end user. In the motor-sport example, dips in performance may be fed to a heads-up display, suggesting the driver do something in order to optimise his performance.

Preferably, in addition to VAD dimensions, the models also carry out Human State analysis and Machine Learning (ML) training or prediction. Human State analysis refers to aggregated and generalised emotional states (e.g. stress, comfort, engagement etc.) ML training refers to utilisation of extracted features to train a model for emotional and human state labelling; ML prediction refers to predicting such labels both real-time and post-session.

In a dependent aspect, the locally weighted polynomial regression model is a non-linear mixed-effects model including a nested random component. This enables the provision of an optimal variance structure, ultimately leading to a more precise pattern measurement of states while taking into account the time series data stream measured from the context.

In a dependent aspect, locally weighted polynomial regression models are comprised of a residual pattern and a time dependent smooth pattern. Advantageously, the problem of residual patterns (which is encountered with known linear models) is overcome. The model may include covariates and random effects.

In preferred embodiments, the locally weighted polynomial regression model is a Generalized Additive Mixed Model (GAMM).

In a dependent aspect, the locally weighted polynomial regression model comprises a smooth term for estimating data variability. This further increases accuracy and speed of the system. In a dependent aspect, processing the modelled feature data using multivariate autoregressive state-space modeling analysis comprises determining states over time e.g. by identifying their Arousal, Valence, and Dominance dimensions. This enables accurate evaluation of the variability of the physiological rhythms and therefore better feedback regarding an individual's affective states. In performance-critical environments it is valuable to measure the situational data and affective data in particular, and make them applicable to the individual's performance and training programs.

In a dependent aspect, data sensed, detected or measured by the plurality of devices comprises physiological data and experience context data. By providing both individual and contextual data, the interpretation of measurements is more accurate.

In preferred embodiments, the plurality of signals comprises a plurality of devices. This enables unobtrusive measurements to the user, increasing flexibility and opportunities for taking measurements during individual experiences. Preferably, the system comprises a plurality of data sources, including transient data storage means, wherein the processor is configured to execute said data processing in real-time.

In further preferred embodiments, the processor comprises a machine learning module for analysing the feature data. Advantageously, an artificial intelligence (Al) model may be trained to provide feedback, e.g. by labelling human states and predict such labels.

In a further aspect, a computer-implemented method for human state analysis comprises: receiving physiological and expressive data from different devices, analysing their own characteristics, combining these data streams in relation to contextual information, and providing individual visual, haptic, sound feedback, data humanisation and/or environmental feedback. A computer-implemented method for human state analysis may comprise: contextual data recorded simultaneously with the physiological and expressive data. For example, the contextual data includes location data, speed, acceleration, video recording (e.g. infrared) and descriptions "tags" of the situation provided by individuals. Preferably, the method comprises the analysis of contextual data to provide the individual's appraisal description of the experiences.

Preferably, the method further comprises evaluating the temporal characteristics of each data stream to describe the affective state. For example, the method further comprises the combination of the characteristics according to individual appraisals of the situation to provide individual visual, haptic, sound, data humanisation and/or environmental feedback.

The feedback may be used to provide information relating to performance assessment.

There is also provided a computer program product embodied in a non-transitory computer readable medium for affective state analysis, the computer program product comprising code which causes one or more processors to perform operations of: analysing the VAD characteristics of the data stream, combining these multimodal data streams and relating them with the context to provide individual visual, haptic, sound data humanisation and/or environmental feedback. In another broad, independent aspect, there is provided a method of measuring states associated with a person's experience over time, the method comprising the steps of:

sensing, detecting or measuring, with a plurality of devices, physiological and expressive data streams associated with the person's experience over time;

synchronising the data streams to obtain synchronised data;

selecting feature data from the synchronised data, the feature data being descriptive of at least one of human state of the person;

modelling the feature data using locally a weighted polynomial regression model; and processing the modelled feature data using multivariate autoregressive state- space modeling to identify at least one potential trend in the state over time.

Preferably, the method further comprises the steps of: sensing, detecting or measuring context data associated with the person's experience over time, synchronising signal values received from the plurality of devices with the context data; and displaying the states to individuals according a multi-modal perspective including visual, haptic and sonic feedback.

Various features, aspects, and advantages of various embodiments and comparative examples will become more apparent from the following further description.

Brief Description of Figures

Examples of the present invention will now be described with reference to the accompanying drawings, where:

Fig. 1 is a flow diagram for affective states analysis;

Fig. 2 is a flow diagram describing the "algorithmic pipeline", that is, the process of obtaining valence, arousal and dominance time series coming from every affective signal, as well as preforming human state analysis and machine learning, in real-time ("live") or post-session; and

Fig. 3 is a flow diagram for displaying individual's affective feedback. Detailed Description

As an individual experiences a specific event, the individual's human state can provide valuable insight to their probable future behaviours. The human state may include a physiological, behavioural or affective state. An affective state comprises any mental state that can be described according to valence, arousal and dominance. These are states such as interest (positive valence, low arousal, and medium dominance), excitement (medium valence, high arousal and high dominance) or fear (negative valence, medium arousal and low dominance). Instead of relying on imprecise evaluations based on only one data stream, combining multiple data streams analysed according to their valence, arousal and dominance and providing individually-tailored feedback relative to an individual's affective state can prove valuable for a variety of reasons, including monitoring athletes' performance in specific affective contexts, regulating psychological states of individuals who have physiological or psychological vulnerabilities, or managing an individual's environments (e.g. car, home, robots) for a better adaptation to the context.

While interacting with the environment, individuals can display physiological changes and expressions of their affective state. Data measured about these changes coupled to a computer are evaluated for example according VAD dimensions. For example, wearable devices can capture an individual's heart rate, heart rate variability, breathing rate, skin temperature, electrodermal activity, electromyography, brain activity, photo plethysmography; cameras can capture facial expression, thermic changes, body postures; microphones can capture voice frequency, pitch, intonation, intensity, text editors can capture affect related words and emojis. In parallel, smartphones can capture location, speed, acceleration, temperature, humidity, lighting levels, noise levels; cameras can capture environment-specific events. Many other data recording capabilities are possible. Some examples utilise multiple sensors in order to capture individual state.

Other data related to an individuals' affective state can be determined; the age, gender, height, weight of individuals being monitored for example.

The physiological and expressive data is sent to a specific web-based VAD module, Human State module and ML model to analyse their VAD dimensions and aggregated/generalised emotional states, as well as train a model for labelling and prediction. The contextual data is sent to a specific web-based appraisal module to identify the event-related influence with the physiological and expressive data. Then both VAD dimensions and context appraisals are processed to be combined and treated by a web-based Emotion Al system. The output Tenderer that uses the data that results from the analysis may be an individual visual, haptic and/or sound feedback.

The rendered output can include text, icons, pictures, graphs, binary data, sounds, lights, vibrations, or any other form or output that can be interpreted by a person or another computer for example. Preferably, the rendered output includes a graph showing the evolution of the different VAD dimensions as well as the appraisal provided by the context. For example, the rendered output includes sounds, vibrations, and lights. The result of the affective state analysis can be displayed in a dashboard where it can be displayed or compared with the ongoing activities already included. Fig. 1 is a flow diagram 100 for affective state analysis and display. The flow 100 describes a computer-implemented method for affective state analysis that includes in the same embodiment one recording data streams time series coming from either subjective expression 101, face 102, voice, 103, movement 104 or physiology 105. One or more data stream time series may be recorded simultaneously. The data stream time series can be collected from multiple devices and from multiple sources while the individuals interact with their environment. For example, the multiple data streams include cell phones, cameras, microphones, electrodes, photosensors, thermal sensors among the possible sensors.

The flow 100 also includes the recording of additional data about the context of the emotion felt or expressed 106. The context data are related to the environment such as speed acceleration, location, weather, period of time but also related to the individual such as age, gender, ethnicity, skills and past experiences. In some embodiments, the context data can be only related to the environment whereas in other embodiments the context data also contains information about individuals. The context data about the environment can be recorded with the accelerometer included in the smartphones or smart watches, the location can be recorded by the GPS location of the individual. The location can be determined using any type of identification including, but not limited to, latitude, longitude, altitude, bearing coordinates. Practically, the location information could identify the journey of an individual in a room, in a building, in a city, or on any surface that can be mapped. Additionally, the context may comprise environmental information 106.

Preferably, the context comprises an activity 106 performed by the individual. The activity included at least one of driving a vehicle, having a human-computer interaction, a human robot interaction, performing a physical activity or performing a psychological activity. The context can further include information further identifying the context such as the name and type of vehicle, the name and type of human-computer interaction, the name and type of human-robot interaction, the name and type of physical activity, the name and type of psychological activity. The additional data can include analysis of affective states perceived by a human annotator and forms the physiological and expressive data stream previously mentioned.

The additional data can include information about the identity of the individual. The information about the individual can be in any form, but in some cases the information about the identity of the individuals includes a name of the individual and /or an identity value for the individual. The identity of the individual can be determined by any method, but not limited to, manual entry by the individual, an operator, or by a computer system account login.

The various data and additional data from multiple time series are send by internet and stored in on a physical server. Every data stream includes a timestamp. The analyses of each data stream according to valence, arousal and dominance are related to the original timestamps in order to allow comparison and synchronized to obtain the valence, arousal, and dominance values for every data stream in relation to the time. In other cases for audio and video data streams, these can be synchronized using a tag, which is marker of time series, in order to identify specific and significant timestamps related to environmental events. The tags can be manually imputed by the individuals or automatically designed according to a significant pattern in the time series data stream.

The flow 100 includes sending time series data streams to a web service 108. The web service can be contacted over a network or a network of networks— such as internet, native compiled libraries, cloud (AWS) and 'chips'— and the web service can be hosted on different computers which may or may not be situated near to the individual. 'Chips' refers to the incorporation of all states of a pipeline running on a microprocessor specifically designed for said pipeline.

The flow 100 describes a way to analyse every time series data stream recording from physiological and expressive measures into valence, arousal, and dominance. A specific VAD module, Human State module and Machine Learning (ML) model is designated according the characteristic of the affective signal: subjective expression VAD module, Human State module and Machine Learning (ML) model 109, Face VAD module, Human State module and Machine Learning (ML) model 1 10, Voice VAD module, Human State module and Machine Learning (ML) model 1 11, Movement VAD module, Human State module and Machine Learning (ML) model 1 12, Physiology VAD module, Human State module and Machine Learning (ML) model 1 13 and Appraisal module 1 14 for the data related to the context. These analyses are performed using locally weighted polynomial regression models, to enable accurate modelling of the feature data, for every specific signal and a multivariate autoregressive state-space modeling. The advantages of this combination is two-fold: on one hand, the locally weighted polynomial regression model enables the identification of the signal of individual physiological changes while taking into account the autocorrelation structure and the random effects of the signal; on the other hand the multivariate autoregressive state-space modeling extracts shared variance of the physiological and expressive changes explained by individual affective states. In turn, this leads to a quicker, more efficient, precise and accurate estimation of affective states. The flow 100 further comprises the synchronising system, that records multiple data streams in an effective manner to analyse human states, also supporting interpretation and communication of that data, machine learning and real-time feedback, transforming data into visual and concise information to help individuals to understand their affective, behavioural or physiological states.

The flow 100 describes an affect analysis module in which all the time series are compared in order to analyse the affect felt and/or expressed in a specific situation 1 15. Using machine learning processing with random forest clustering, the patterns are classified relative to their evolution, and are associated with the characteristics of the context signal through the appraisal module 1 14.

The flow 100 describes a method to provide individuals' feedback 116 according the characteristic of the affect analysis 1 15. The individual feedback can be visual 1 17 made of basic information such as light or color or displayed on a screen via a dashboard. The individual feedback can be haptic 1 18 made of different vibrations. Further, the individual feedback can be sounds 1 19 describing the characteristics of the VAD analysis. The individual feedback can also be data humanisation 120 describing the human state characteristics or environmental data 121.

Fig. 2 is a flow diagram 200 of the statistical treatment to process each data stream time series. The data stream time series related to human states can be divided into two categories: the physiological measures (heart rate, breathing rate, electro dermal activity, etc.) and the expressive measure (subjective sentiment, facial expression, voice, body postures and movement). The context measures may be used as predictors of the physiological changes.

Fig. 2 illustrates the main stages of an example method of processing data 200 relating to affects: data collection 201, where BAN means Body Area Network; signal pre-processing, 202; feature extraction, 203; and event classification/pattern interpretation 204. These steps are described in detail below.

At step 201, data is collected from the various data streams. The signal 202 can include subjective expression, face, voice, body movement or physiological changes. At step 203, signal-pre-processing is carried out with suitable filters to remove outliers and clean the data streams 204. For example, a feature extraction module layer is input raw data (e.g. heart rate) and it derives sub-features output data, such as a change in heart rate. At step 205, relevant features are extracted from certain data streams such as the high frequency of heart rate variability from the ECG and the skin conductance level/ response from the GSR measure. At step 204, the analysis of the patterns in the data streams are interpreted according to the context and individual appraisal processes. Although direct observation of data streams provides indications of the meaning of the physiological changes, the application of a statistical model is necessary in order to assess the significance of these physiological changes over the time and according to every participant.

In order to obtain the optimal structure from each feature, the present inventors propose that a local regression smoother (LOESS) analysis should be used.

LOESS is known in the art and usually denotes a method that is also known as locally weighted polynomial regression. At each point in the range of the data set a low-degree polynomial is fitted to a subset of the data, with explanatory variable values near the point whose response is being estimated. The polynomial is fitted using weighted least squares, giving more weight to points near the point whose response is being estimated and less weight to points further away. The value of the regression function for the point is then obtained by evaluating the local polynomial using the explanatory variable values for that data point. The LOESS fit is complete after regression function values have been computed for each of the data points.

Advantageously, GAMM allows for variables and those variables provide for a smoother pattern - they act to smooth the results out. At step 207, the analysis of the patterns in the data streams are interpreted according to valence, arousal and dominance characteristics 208, human state analysis 210 ML analysis (training or prediction) 21 1 and behavioural analysis 212 of the time-series and compared to the context and individual appraisal processes. Although direct observation of data streams provides indications of the meaning of the affective changes, the application of a statistical model is necessary in order to assess the significance of these affective changes over the time period and relative to each participant.

Human State analysis 210 refers to aggregated and generalised emotional states (e.g. stress, comfort, engagement etc.) ML training 211 refers to utilisation of extracted features to train a model for emotional and human state labelling; ML prediction 211 refers to predicting such labels both real time and post-session. Machine learning analysis and training is performed on the output of the feature extraction module 205 to improve the quality of data sets and enhance-decision making based on patterns for example. Behavioural analysis 212 refers to inference of generalised human behaviour for example (but not limited to) habitual behaviours or social / socio - communicative behaviours.

As shown in Figure 2, execution of all stages of the algorithmic pipeline can be carried out in real time 213 or as close to real-time as possible. This can be within a specific time period from current time, for example 10 seconds or less. The method can also be carried out post- session 214, which means after the data recording has finished such that the entire dataset has been collected. Real-time ("live") analysis differs technically from post-session analysis; in live analysis direct data streams are being received from sensors rather than being bundled up into data packets. This may therefore require a transient data storage system. Furthermore, processing of the data may occur in more than one discrete module at once, before the data is combined. That is, the modules process data may concurrently in time.

Preferably, the system comprises a plurality of data collecting nodes. In a use case scenario, a driver carries a mobile phone and data capture is performed by the mobile phone at all times, as the driver carries the mobile phone. In addition, the system may have a site- specific, e.g. car based data capture. Each of these two modules will capture different types of data. The combination of site-based and cloud-based processing has the advantage of allowing for continuous processing or means for creating a data source such as prior contextual data (e.g. has the driver been to a pub before getting into a car?). Biometric data received from wearable devices for example can also be processed on site. A site specific processor usually has more processing power than a mobile device, or different capabilities.

The interpretation of human characteristics, including e.g. valence, arousal and dominance characteristics 208, are performed using multivariante autoregressive state-space modelling. Multivariate autoregressive state-space modeling is a technique known in the art used to detect common patterns in a set of time series and posit relationships between these series and explanatory variables.

In order to extract the trends in physiological changes, the multivariate autoregressive State- Space modeling is used to extract shared variance of the data stream changes explained by individual affective states. The multivariate autoregressive state-space modeling is assumed that the common trends follow AR (1) process.

The model extracts a best fit line in a high dimensional surface combining all data streams. Advantageously, the multivariate autoregressive state-space modeling is performed using the previously extracted GAMMs with a diagonal and unequal correlation structure. For a better factor loading the trends are rotated using the varimax method.

Using multivariate autoregressive state-space, in combination with the GAMM model that uses a smooth term to estimate the variability of the data, enables accurate evaluation of the evolution of the trends during the time of the affective experience.

Using optimised models, namely a combination of GAMM and multivariate autoregressive state-space modeling, it is possible to analyse the signal according VAD or human state patterns in order to provide individual affective feedback 209. The provided data are not restricted by a lab environment but close to the "ground truth" of the affective changes. It enables accurate feedback to individuals about their affective states.

Accordingly, when implemented with embodiments according to the invention, wearable technologies are very useful to provide affective measurements. They are particularly interesting in the evaluation of the variability of the physiological and expressive changes during specific events. As consequent and antecedent physiological and expressive changes are related to contextual events, their variability is a relevant indicator of individual affective experience. In this perspective, providing visual, tactile, and audio feedback can allow individual to modify their behaviours in a more accurate way.

Fig. 3 is a flow diagram for displaying individual's affective feedbacks 300. Individual affective feedbacks can be visual, haptic and/or a sound 301. These feedbacks are displayed to the individuals in real-time according their affective states 302. These feedbacks are displayed on a large variety of devices such as virtual or augmented reality devices, smart phones, smart watches or any possible screen, speakers and haptic devices 303. These feedbacks are provides in a specific context 304 in order to make individuals regulating and adjusting their actual affective state. Various environments and contexts are specified; they can be indoor or outdoor 305 and related to various activities 307. The aim of the feedback information is to make people about to learn from their own affective state in order to improve their relationship 306 with the environment of the context 307.

Applications of the present invention are wide and include consumer car market, as well as F1 in so called 'human reactive cockpits' to enhance safety and comfort. Other applications which could benefit from human state monitoring include ride-share applications (to assess and improve customer experience), e-sports and affective gaming (that is, gaming which responds to the feelings of a player) involving dynamic play/sensitive content, as well as consumer robotics (e.g. emphatics robots in the household industries).

Any of the embodiments described above may include more or fewer steps than those disclosed.

Additionally, it will be appreciated that the term "comprising" and its grammatical variants must be interpreted inclusively, unless the context requires otherwise. That is, "comprising" should be interpreted as meaning "including but not limited to".

Moreover, the invention has been described in terms of various specific embodiments. However, it will be appreciated that these are only examples which are used to illustrate the invention without limitation to those specific embodiments.