The present invention relates to a physiological signal processing, in particular to processing physiological signals sensed from the subject to determine the state of a subject.

The determination of the state of a human subject from physiological signals sensed from the subject arises in many fields including medicine, health, fitness and sport. The state of the subject may include the physical state such as the health or medical condition of the subject, or the mental or emotional state of the subject.

A vast body of work has been directed to determining the state of a subject by interpreting physiological signals measured using sensors, which may be in contact with the subject or contactless, such signals including the typical vital signs of heart rate, breathing rate, blood pressure, body temperature and blood oxygen saturation (Sp02) as well as other physiological signals such as a respiration signal, electrocardiogram (ECG) electroencephalogram (EEG), electro dermal activity (EDA or skin conductance), and many more.

Basic approaches to determining the state of a subject can include placing thresholds on the values of a selected limited number of such physiological signals and determining a subject to be normal if they are within the thresholds and abnormal if they are outside them.

Simple thresholds, however, have the disadvantage that the appropriate thresholds vary from subject to subject, for example, for subjects of different age and fitness, and setting narrow thresholds tends to result in too many false alarms and setting broad thresholds may mis-classify the subject as normal. Further, simple threshold setting does not acknowledge that there may be useful information in the combination of the

physiological signals. For example, a blood pressure which is low, but within normal thresholds, may be unusual if it is associated with a heart rate which is high but within normal thresholds.

Thus, the way that physiological signals vary together is of interest.

Physiological signals are variable and noisy and thus while increases in available computer processing power promise improvements in automated determination of a subject’s state from such physiological signals, delivering systems which function well- enough in practice (i.e. quickly and reliably) is a difficult problem. Automated systems which respond too slowly or which do not reliably give accurate results tend simply to be ignored.

One aspect of the present invention provides a way of processing physiological signals which allows the information within them to be recognised more easily by an automated system.

Another aspect of the invention provides a way of interpreting combinations of physiological signals in a way which brings out information from the way they vary together.

According to the invention there is provided a method of determining the state of a subject from a plurality of different physiological signals comprising the steps of: receiving the plurality of different physiological signals, the physiological signals being

synchronised and including a periodic signal; defining successive time windows, each defined as corresponding to at least one respective periods of the periodic signal;

processing each of the physiological signals in the time window to extract information relating to the state of the subject; determining the state of the subject from said information; and outputting the result of the determination.

Thus in accordance with this aspect of the invention the physiological signals are divided into time windows or segments which correspond to one or more periods of a periodic physiological signal. Thus rather than the time windows being of constant length, the time windows vary as the periodicity of the periodic physiological signal varies.

The step of determining the state of the subject may be based on information relating to the state of the subject from one time window or from a plurality of time windows, and where a plurality are used they may be successive. Combining the results from several time windows can effectively strengthen the signal and improve accuracy.

The physiological signals are synchronous - i.e. synchronised on a common timescale. This may be achieved by synchronising sensors, or time-stamping the signals, or by asking the subject to perform some action which is detected in all sensors and can be used to synchronise the signals. For example, if a subject taps their chest with their hand it gives a characteristic pattern in the respiration sensor signals that can be used for synchronisation.

The periodic physiological signal may be a respiration signal representing inhalation and exhalation. Such a signal may be obtained, for example, by sensing the percentage expansion of the subject’s chest obtained from a chest-worn device (such as an impedance pneumography sensor, e.g. Respiban™) or means of a non-contact chest movement signal.

The time window may be one breath (i.e. one inhalation followed by one exhalation), or more than one successive breath.

In an alternative embodiment the periodic physiological signal may be a heartbeat signal, such as from an electrocardiogram, oxygen saturation monitor, a simple pulse monitor or the like. In this case the time window may be a single heartbeat or a plurality of successive heartbeats.

By basing the windowing or segmenting of the physiological signals for analysis on the periodic physiological signal such as one or more breaths or one or more heartbeats, rather than on an arbitrary constant time window, the extraction of information from other physiological signals is improved. It effectively uses the fact that there can be

relationships between the other physiological signals and the varying periodicity physiological signal (e.g. heart varies with breathing), so extracting information on this natural timescale, which can vary, gives better results.

Where the physiological signals include a respiration signal, the processing step may comprise separating the respiration signal into an inhalation signal and an exhalation signal and processing these as separate signals. Thus information relating to the state of the subject is extracted from the inhalation signal and exhalation signal.

Where the physiological signals comprise an ECG the step of processing the physiological signals may comprise extracting the beat to beat timing of the ECG signal (e.g. R peak to R peak) and the step of extracting information may be conducted on the resulting peak-to-peak timing signal. The complete ECG is complex and contains a lot of information, and so deriving peak-to-peak timing, or the heart rate, reduces and simplifies the amount of information to be processed.

Where the physiological signals comprise an EDA signal the processing may comprise extracting a slowly-varying component of the EDA signal for processing to extract information relating to the state of the subject.

The analysis of the signals to extract information may comprise extracting features of the signals within the time window. There are a variety of well-known methods of feature extraction which may be used, and a typical list of signal features that may be extracted for physiological signals is indicated in Table 1 below. Table 1

In an embodiment of the present invention a different set of features is extracted from the physiological signals. In particular, the path signature of the physiological signals in the time window is calculated. The signals may be used to form a continuous path in a multi-dimensional space, and the path signature is the values of the various iterated integrals of the path in that space (i.e. of signals in the time window). For any set of signals there is an infinite number of iterated integrals and so in accordance with this embodiment the path signature is calculated to a predetermined order, resulting in a finite number of terms in the path signature. Each term is the value of one of the iterated integrals. The values of the terms in the path signature are effectively measures of the shape of the signals over the time window.

In an embodiment transformations may be applied to the paths before computing the signatures. For example, the cumulative sum or lead-lag transformations may be used to form paths from the signals. In one application, for example, applying the lead-lag transformation on a 4-dimensional path (e.g. formed by the ECG, EDA, Inhale, Exhale signals) creates an 8-dimensional path whose signature can be computed.

The predetermined order is optimised as a hyperparameter according to the amount of data available. Successively higher orders result in exponential increases in the number of terms in the path signature and the order is chosen to balance the additional information gained with increasing order against the amount of data available to train on that information. With higher orders computing power may be an issue. In one embodiment disclosed below the order is 3 but the order may be, for example, in the range 2 to 6, more preferably 3 to 5.

The step of determining the state of the subject from the extracted information is conducted by a trained classifier, i.e. a trained machine learning algorithm. In the case of the extracted information comprising extracted features, in particular such as a path signature for example, the use of a trained machine learning algorithm is particularly fast, accurate and efficient.

The invention may be used to determine the state of the subject, in particular the health or emotional state of the subject, such as whether the subject is stressed or unstressed, the cardiac health of the subject or the presence of sleep apnoea.

The present invention also provides a method of training a classifier to determine the state of a subject from a plurality of physiological signals, in accordance with the method of any one of the preceding claims. The principles of such training processes are a well-known aspect of machine learning and use a set of training data comprising physiological signals and expert-assessed states, and the training process adjusts the classifier to match its outputs to the expert-assessed states to a desired degree of accuracy. The physiological signals in the training data are processed for input to the classifier in the same way as above, and thus in this case the training method comprises the steps of:

training the classifier on a training data set prepared from a plurality of physiological signals from plural subjects and associated subject states, the physiological signals being synchronous and including a periodic signal; the training data set being prepared by:

defining a time window corresponding to one or more periods of the periodic signal;

processing each of the physiological signals in the time window to extract features of the physiological signal in the time window.

The physiological signals may include a respiration signal and the processing step comprises separating the respiration signal into an inhalation signal and an exhalation signal and extracting features from the inhalation signal and exhalation signal.

The physiological signals may comprise an electrocardiogram and the step of processing the physiological signals comprises extracting the peak to peak timing of the electrocardiogram signal in the time window and extracting features from the peak to peak timing.

The physiological signals may comprise electro dermal activity and the step of processing the physiological signals comprises extracting a slowly varying component of the electrodermal activity signal and extracting features from the slowly varying component.

The state of the subject may at least one of: stressed, unstressed, a particular emotional state, the state of the subject’s cardiac health, presence of sleep apnoea.

The extraction of features may comprise calculating the path signatures to a predetermined order over the time window.

The classifier may be trained by means of a gradient boost algorithm.

The invention extends to apparatus for determining the state of a subject from a plurality of physiological signals, the apparatus operating according to the methods above. Such apparatus includes an input for receiving physiological signals from a plurality of sensors which may be contact or contactless sensors, a signal processor such as a data processor for processing the physiological signals and determining the state of the subject, and a display for outputting the result of the determination.

According to further aspects of the present invention, there are provided a computer program capable of execution by a computer apparatus and configured, on execution, to cause the computer apparatus to perform a similar method, a computer-readable storage medium storing such a computer program, and a computer apparatus arranged to perform a similar method. The computer program may comprise program code means for executing the steps of the method.

The invention will be further described by way of examples with reference to the accompanying drawings in which:-

Fig. 1 is a flow diagram illustrating an embodiment of the invention;

Fig. 2 schematically illustrates an apparatus in accordance with an embodiment of the invention.

Figs. 3A-3C illustrate three physiological signals usable in an embodiment of the invention;

Figs. 4A-4C illustrate the pre-processing of an ECG signal in accordance with an embodiment of the invention;

Figs. 5A-5C illustrate pre-processing of an EDA signal in accordance with an embodiment of the invention;

Figs. 6A-6C illustrate pre-processing of a respiration signal in accordance with an embodiment of the invention;

Figs. 7A-7C illustrates the pre-processing of a single breath signal to separate inhalation and exhalation in accordance with an embodiment of the invention;

Fig. 8 illustrates paths formed from the pre-processed physiological signals of Figs. 3-7 in accordance with one embodiment of the invention;

Fig. 9 is a flow diagram illustrating the processing of physiological signals in one embodiment of the invention;

Fig. 10 is a flow diagram illustrating feature extraction in one embodiment of the invention;

Fig. 11 illustrates paths formed from the physiological signals of Figs. 3-7 over several breaths for use in one embodiment of the invention; and

Fig. 12 is a flow diagram illustrating the training of a classifier for use in an embodiment of the invention;

Some embodiments of the invention will now be described in detail.

Fig. 1 schematically illustrates a processing method in accordance with an embodiment of the invention. The method may be performed using a computer apparatus. To achieve this, a computer program capable of execution by the computer apparatus may be provided. The computer program is configured so that, on execution, it causes the computer apparatus to perform the relevant steps of the method.

The computer apparatus, where used, may be any type of computer system but is typically of conventional construction. The computer program may be written in any suitable programming language. The computer program may be stored on a computer- readable storage medium, which may be of any type, for example: a recording medium which is insertable into a drive of the computing system and which may store information magnetically, optically or opto-magnetically; a fixed recording medium of the computer system such as a hard drive; or a computer memory. Fig. 2 illustrates a computer apparatus 1 suitable for executing the method. As illustrated, the computer apparatus 1 comprises an input 5 for receiving physiological signals from sensors 2, 3 and 4 and supplying the physiological signals to a signal processor 6. The input may be any suitable interface.

The the computer apparatus 1 comprises a signal processor 6 that may be a software-controlled data processor such as a conventional computer system. The signal processor 6 outputs the result of its determination to a display 7. Although three physiological signal sensors 2, 3 and 4 are illustrated, fewer or more may be provided.

A first embodiment of the invention will be described in which the physiological signals input are an ECG signal, an EDA signal and a respiration signal. Thus the sensors 2, 3 and 4 are respectively an ECG sensor 2, an electro dermal activity sensor 3 and a respiration sensor 4 such as a chest-worn device for measuring the expansion of a chest or an impedance pneumography sensor. Figs. 3A, 3B and 3C illustrate example respiration, EDA and ECG signals respectively from such sensors.

In accordance with this embodiment of the invention in step 101 the physiological signals such as the three signals illustrated in Figs. 3A-3C are received by the input 5 and passed to the data processor 6 for processing. In a first step 102 the physiological signals may be individually pre-processed. In the case of the three signals above, the pre processing may be as follows.

Figs. 4A-4C illustrate the pre-processing of the ECG signal. The raw ECG signal as illustrated in Fig. 3C and Fig. 4A is complex and contains a lot of information. For the purposes of this embodiment the information in the ECG signal is reduced by

automatically identifying the R peaks in the signal, illustrated by crosses in Fig. 4B. The peak-to-peak time is then measured and the timing of each peak is plotted against beat number as illustrated in Fig. 4C to form a peak-to-peak timing signal. It should be noted that the peak-to-peak time is not constant, but varies, so the line plotted in Fig. 4C is not straight.

Figs. 5A-5C illustrate the pre-processing of the EDA signal. Fig. 5A illustrates the raw EDA signal and in the pre-processing step 102 this is filtered to separate a slowly varying component SCL as illustrated in Fig. 5B and a rapidly varying component SCR as illustrated in Fig. 5C. This filtering may be performed by any suitable low-pass filtering technique such as applying a low pass filter with a cut-off at, for example, 5 HZ, or selected between 3Hz and 10Hz, to the raw EDA signal. The SCL and SCR components may be extracted using the method disclosed in“ Development and evaluation of an ambulatory stress monitor based on wearable sensors” by J. Choi, B. Ahmed, and R. Gutierrez-Osuna, IEEE Transactions on Information Technology in Biomedicine 16, 2 (2012).

Figs. 6A-6C illustrate the pre-processing of the respiratory signal. Fig. 3 A and Fig. 6A illustrate the raw respiratory signal which is processed to isolate the inhalation and exhalation of individual breaths. This is achieved by detection of peaks and minima, for example using a freely-available peak finding algorithm such as Python Scipy’s find peaks. Fig. 6B illustrates the detected start and end points of inhalation and exhalation and Fig. 6C illustrates the division of the respiration signal into individual breaths, each breath starting from an inhalation start point and extending to the next inhalation start point. In this embodiment the respiration signal is used not only as a physiological signal contributing to the classification of the subject’s sate, but also to define a time window for segmentation of the other physiological signals.

In step 103 the individual breath waveforms are taken, one is illustrated in Fig. 7A, and as illustrated in Figs. 7B and 7C the inhalation and exhalation components are separated to form separate inhalation and exhalation signals. The inhalation signal consists of the inhalation part of the reparation waveform which is then set to a constant value after exhalation starts, and correspondingly the exhalation signal consists of a constant zero value until exhalation starts followed by the inversion of the exhalation part of the breath waveform.

In step 104 the other physiological signals are segmented using the breath start and end times from the respiration signal. Then in step 105 the physiological signals, including the inhalation and exhalation signals, are analysed for each time window or segment to extract information relating to the state of the subject. In this embodiment the analysis is conducted on a segment corresponding to a single breath. However in an alternative embodiment the time window may be formed by a plurality of successive breaths.

Fig. 8 illustrates by way of example the physiological signals from Figs. 3-7 for a single breath. These consist of: an inhalation signal and an exhalation signal (the two signals illustrated in Fig. 7C), an EDA signal (which is the slowly varying component illustrated in Fig. 5B for this time period) and a peak-to-peak ECG timing signal (which is the section of Fig. 4C for this time period). For illustrative purposes in Fig. 8 these signals are all normalised to lie between zero and one on an arbitrary scale (the y-axis in Fig. 8) and on a common time scale (the x-axis) but normalisation may not be essential for further analysis. The analysis of the signals for the selected time window will be described in more detail below, but the result of the analysis is to output information relating to the state of the subject. This information may be, for example, that the subject is stressed or unstressed, the emotional state of the subject, the cardiac health of the subject or whether the subject is suffering sleep apnoea in this time period. The information for this time period may either be output immediately as in step 106, or may be combined with information from preceding time periods in step 107 for output of a combined result in step 108. Therefore the output may depend only on the state as determined from the current time window (step 106), or may depend on the states reported over several time windows (steps 107 and 108). Different ways of combining such state information for several time windows are well known such as a rolling average, Kalman filter, voting, use of a state machine etc..

Having output the results from the analysis of this time window, in step 109 the process returns to step 105 to analyse the next time window. It should be noted that in the case of a time window extending across several segments or several cycles of the periodic signal, the time window may move by its own length or may move by only one or more periods of the physiological signal in which case successive time windows overlap.

Fig. 9 illustrates one embodiment of the analysis of the segmented physiological signals that may be performed in step 105, as follows.

In step 201 the time window under consideration is taken and in step 202 a feature extraction process is used to extract particular features of the segments of physiological signals. Many signal feature extraction algorithms are known. Table 1 above illustrates a number of typical features of physiological signals that may be extracted. Then in step 203 the features are used to determine the state of the subject. This may involve testing the value of features against predetermined thresholds, comparing them with normal values for those features, or inputting them into a trained classifier as explained in more detail in the embodiment below. The determined state of the subject is then passed for output or combination with previous determinations in step 106 or 107.

In the case of the determination of the state of the subject being performed by a trained classifier, a particularly effective embodiment of the invention lies in deriving the features of the pre-processed and segmented physiological signals using the signature method, which gives a set of numerical values from the signals, the set being known as the “path signature”. The signature method is a way of analysing the evolution of multiple signals and, in particular, efficiently encoding the shape of the signals (i.e. how they vary, both in themselves and against each other). The path signature is obtained by calculating iterated integrals (to a predetermined order) of the signals over a particular interval (in this case the time window under consideration). The signature method is described, for example, in“ Rough paths, Signatures and the modelling of function on streams’ by T. Lyons, 2014, or in a more basic way in:“ A primer on the signature method in machine learning’ by Chevyrev and Kormilitzin. Open-source path signature calculation software is available such as esig: (https://github.com/kormilitzin/the-signature-method-in-mach ine- leaming) or iisignature:

(https://www.google.com/search?q=iisignature+bottler&rlz =lC5CHFA enGB785GB785

&oq=iisignature+bottler&aqs=chrome..69i57i0i69i60 12i012.2399i0i4&sourceid=chrome&i e=UTF-81

Remembering that in simple terms integrating a function gives the value of an area, e.g. the area under a curve, referring to Fig. 8 one order of the path signature is the integral of each of the illustrated signals over the illustrated interval, i.e. the area underneath each of the lines. It is also possible however, to integrate one signal with respect to another, creating another set of values representing areas. Further, higher order integrals may be calculated representing volumes in a multidimensional space with each dimension corresponding to one of the illustrated signals. The signals may be integrated multiple times to any arbitrary order so that in general terms there is an infinite number of signature values (values of integrals) which may be calculated. The lowest order is simply the value of the change in the signal from start to finish of the interval. The next order corresponds to areas underneath the signal, and so on. The set of values resulting from these integrations characterise the mutual shapes of the signals and form the path signature of the signals.

Formally, the set of signals is regarded as forming a path in a multidimensional space, one dimension corresponding to each signal. By a path X we refer to a continuous mapping over an interval from a to b in a space of dimension d where d is the number of different signals:

of bounded variation.

We denote coordinate paths by where each is a real

valued path in the space The terms of the signature are the collection of all iterated integrals of X:

By convention the zeroth order term is 1.

The first order terms of the signature are the values of:

So for the four paths shown in Fig. 8 this would just be the difference between the starting and finishing values of the inhalation signal, the exhalation signal, the slowly varying component of the EDA signal and the ECG peak-to-peak timing signal.

The second order terms of the signature are the values of:

The third order terms of the signature are, correspondingly, the set of values of triple integrals, and so on for higher orders.

As more orders of the signature are considered, the finer the level of detail is gained about the path. However, this comes at the cost of increased numbers of features for the learning algorithms. Thus the order truncated to is chosen as a problem-dependent hyperparameter. For the application and signals discussed here it was found that increasing above about order 3 starts to see a decrease in accuracy that would have to be resolved with more data. Thus in the case of the signals such as illustrated in Fig. 8 an adequate characterisation of the path signature may be achieved by calculating integrals up to the third order. However a higher or lower value of the order may be chosen.

Fig. 10 is a flow diagram illustrating feature extraction in one embodiment of the invention that may be performed in step 202. In this embodiment a path is formed in step 301 from the pre-processed physiological signals. In this embodiment the lead-lag transformation is applied to the signals in step 301 before calculation of the path signature to the desired order in step 302, so the path whose signature is calculated is formed by the lead-lag transformed signals which has twice the number of dimensions as there are signals ( a lead signal and a lag signal for each original signal).

The output of step 302 is a set of numerical values representing the way in which the signals vary in the interval under consideration. These signature values constitute values which may then be input in step 303 to a trained classifier. The signature values may be used directly with a linear classifier or the log-signature values may be calculated and used with a nonlinear classifier. The classifier is a trained machine learning algorithm which determined the state of the subject on the basis of the signature values. The training process will be described below. In step 304 the classified state of the subject is output for display in step 106 or for combination with earlier values in step 107. The combination with earlier values to output an improved determination may also be achieved using machine learning, e.g. by using a second trained classifier. Thus to combine the results of n breaths, for each breath, the first classifier gives a probability of that individual breath being stressed, which gives a weak but significant signal. For each breath the previous n probabilities can be used as features in a new classification algorithm (e.g. a Linear Discriminant Analysis classifier) to combine the weak signal into a strong one. This then gives an overall output of stress of the current breath given the n previous breaths.

The embodiment above segments the physiological signals into segments of length one breath as illustrated in Fig. 8. However in an alternative embodiment the signals may be segmented into longer time windows of several breaths. Fig. 11 illustrates the signals from Figs. 3 to 7 for six breaths. Again, in the illustration the signals are normalised to lie between zero and one on an arbitrary scale, but normalisation is not necessary in the calculation of path signatures.

Fig. 12 illustrates the training process to provide the trained classifier used in step 303. This embodiment is designed to classify the state of a subject as stressed or unstressed on the basis of the ECG, EDA and respiration signals.

In step 401, therefore, the ECG, EDA and respiration signals are received together with information on the state of the subject. This therefore constitutes a training data set for the classifier. A suitable training data set is, for example, the publicly available WESAD (wearable stress and effect detection) data set discussed in ICMI' 18, October 16- 20, 2018, Bolder, Co, USA, which has ECG, EDA and respiration signals for 15 subjects together with information on their affective states (neutral, stress and amusement) over the time period of monitoring. For the purpose of training the classifier used in the embodiment above the neutral and amused states were merged as being“unstressed”.

In step 402 each of the physiological signals is pre-processed in the same way as discussed above with reference to Figs. 3 to 5 and in step 403 the start and end times of the individual breaths are identified as discussed above with reference to Fig. 6. The physiological signals are then segmented using the breath start and end times in step 404 and in step 405 the path is created for each of the pre-processed physiological signals for that segment. If desired a transformation is applied, such as the lead-lag transformation, and in step 406 the features of the signals are extracted, such as by calculating the path signatures to a predetermined order and in step 407 these features are used together with the associated subject states to train a classifier, for example using one of the well-known gradient boosting algorithms (e.g. xgboost, lightgbm, skleam’s

GradientBoostingClassifier). This involves a machine learning algorithm going through an iterative adaptation process to so that the output it derives from the input features (e.g. path signature) agrees to a desired degree of accuracy. Suitable examples of a trainable classifiers are Gradient boosting, ADABoost, Random Forests, Neural Networks etc..

In the case of using information from multiple successive segments as in steps 107 and 108, the training process correspondingly trains a second classifier in step 408 based on the training data set information across multiple segments.

In the embodiment described above the log-signature is used which is a condensed form of the information that generally works well provided one uses a nonlinear classifier. As an alternative one can use the terms of the signature and a linear classifier in both the training and use aspects of the invention.

Embodiments of the invention as described above using calculation of path signatures over a single breath were tested on the WESAD data set by repeatedly training it on 14 out of the 15 subjects and then testing it on the 15th (all metrics were obtained using a leave one person out prediction method). In the case of path signatures being calculated over a single breath the method of the embodiment achieved an accuracy of identifying the stressed state of 90.58% with an Fl-Score of 82.91 (which is better measure of accuracy due to the imbalanced classes (fewer cases of stress)).

The accuracy increased to 97.8% by combining the results of several breaths (as in steps 107 and 108 of Fig. 1), e.g. 16 successive breaths corresponding in this data set to one minute of data, and the Fl-Score was increased to 95.81.

The embodiments above are based on segmenting physiological signals using a respiration signal as the periodic signal. However in alternative embodiments the physiological signals may be segmented using alternative periodic signals. For example, in some applications it may be useful to segment signals according to the heartbeat signal, analysing on the basis of time windows corresponding to each heartbeat, or a plurality of heartbeats. Any other periodic physiological signals, with a varying periodicity, may be used to segment the physiological signals to be analysed.