Cross-Reference to Related Patent Applications

This patent application claims priorities of European patent application No. 22197073.4 filed on 22.09.2022 and of Italian patent application No. 102023000019473 filed on 21.09.2023, the entire contents of which are incorporated herein by reference.

Technical Field of the Invention

The present invention generally relates to real-time monitor and/or predict behavioural states, namely among Awake (W), Drowsiness (D) and Sleep (S) phases, and/or transitions thereof to determine a sleepiness or drowsiness level of a subject.

State of the Art

As is known, poor sleep is one of the world’s most common health problems, costing up to USD 700 billion across five OECD (Organisation for Economic Co-operation and Development) countries.

In particular, sleepiness at the wheel is among the most common risk factor for car accidents; for instance, in 2018, the World Health Organization, WHO, reported that up to 20% of all traffic accidents and up to 50% of deaths and serious injuries related to motor vehicles are thought to be associated with sleepiness at the wheel.

In recent years, the Council of the European Union voted to adopt regulations proposed by the European Parliament in March 2019 to mandate the presence of advanced safety systems in automobiles, including driver drowsiness and attention warning systems, in the coming years, hereinafter referred to as GSR, General Safety Rules.

Established, well-known solutions are based on driving style/vehicle trajectory evaluation and/or on eye tracking and facial expression through camera system, which can be easily installed in the vehicle and can perform additional tasks (e.g. child detection, in-cabin monitor and similar). Currently, major automotive suppliers provide solutions mainly based on such a “behavioural” assessment of the driver and, more generally, different measures have been implemented to detect drowsiness:

- Image based measures: some drowsiness signs are visible and can be recorded by cameras or visual sensors, in particular by recording the driver’s facial expressions and movements, especially the head movements. Generally, such systems are non-intrusive, non- invasive and cost-effective, as they require only a camera to collect the needed data, and they are able to provide details on the subject's state of sleepiness at a very advanced stage, when his cognitive state is no longer able to carry out its activity; however, the system’s performance is significantly affected in cases where it is difficult to track facial data due to obstacles and they cannot cover the large population of people who fall asleep with their eyes open, especially people with Obstructive Sleep Apnea Syndrome, OS AS;

- Biological-Based Measures: many biological signals, such as brain activity, heart rate, breathing rate, pulse rate and body temperature signals, have been used to detect the driver’s drowsiness. These biological signals, also known as physiological measures, are proven to be more accurate and reliable for detecting drowsiness; in particular, the accuracy is due to their ability to capture early biological changes that may appear, in the case of drowsiness, thus alerting the driver before any physical drowsiness signs appear. Several activities are aimed at developing cost effective and the least intrusive, possibly contactless, sensors able to provide accurate measurement of the required biometric parameters;

- Vehicle-Based Measures: this method depends on tracing and analysing driving patterns, the latter forming a unique driving pattern. Thus, the driving patterns of a drowsy driver can be distinguished from those of an alert driver. However, since it is an indirect way of detecting drowsiness, such solution is neither accurate nor fast enough to regain the consciousness level of the driver; and

- Hybrid-Based Measures: a hybrid drowsiness detection system exploits a combination of image-, biological, and vehicle-based measures to extract drowsiness features, with the aim of producing a more robust, accurate and reliable drowsiness detection system.

A further known system for detecting and predicting transitions between awake, drowsiness and sleep phases is disclosed in the international patent application WO 2020/043855 Al, wherein full photopletismography (PPG) signal is used.

Currently, image-based measures and vehicle-based measures are the most commonly used due to the inherent simplicity and low cost.

Ob ject and Summary of the Invention

The Applicant notes that the adoption of the new GSR in Europe about driver drowsiness monitor systems for vehicles is aimed at reducing fatalities on road but introduces new challenges; therefore, the Applicant believes that the current known solutions are subject to improvements.

In particular, with respect to image-based measures, the system’s performance is severely affected in cases where it is difficult to track facial data due to, e.g., obstacles; additionally, image-based measures provide details on the subject's state of sleepiness at a very advanced stage, i.e. when his cognitive state is no longer able to carry out its activity. Above all, they cannot cover the large population of people who fall asleep with their eyes open, especially people with OS AS.

With respect to vehicle-based measures, since they are an indirect way of detecting drowsiness, such measurements are neither accurate nor fast enough to regain the consciousness level of the driver; consequently, a proactive approach aimed at limiting dramatic effect on the driver who lost the control of the vehicle should take into account functional safety requirements and ask for a dedicated design of the electronic control system.

The Applicant furthermore notices that safety critical systems require for fail-safe or even fail-operational solutions, implying that a robust system must rely on hardware and/or software diversity in order to cope with both random and systematic failures.

The object of the present invention is to provide an electronic system designed to monitor and/or predict behavioural states and/or transitions thereof to determine a sleepiness or drowsiness level of a subject that solves at least in part the problems of the known solutions.

According to the present invention, an electronic system designed to monitor and/or predict behavioural states and/or transitions thereof to determine a sleepiness or drowsiness level of a subject is provided, as claimed in the appended claims.

Brief Description of the Drawings

Figure 1 schematically shows an electronic system according to an embodiment of the present invention.

Figure 2 schematically shows an electronic system according to a further embodiment of the present invention.

Figure 3 schematically shows an electronic system according to another embodiment of the present invention.

Figure 4 schematically shows an electronic system according to a yet further embodiment of the present invention.

Detailed Description of Preferred Embodiments of the Invention

The present invention will now be described in detail with reference to the accompanying drawings in order to allow a skilled person to implement it and use it. Various modifications to the described embodiments will be readily apparent to those of skill in the art and the general principles described may be applied to other embodiments and applications without however departing from the protective scope of the present invention as defined in the appended claims. Therefore, the present invention should not be regarded as limited to the embodiments described and illustrated herein but should be allowed the broadest protection scope consistent with the features described and claimed herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning commonly understood by one of ordinary skill in the art to which the invention belongs. In case of conflict, the present specification, including the definitions provided, will control. Furthermore, the examples are provided for illustrative purposes only and as such should not be considered limiting.

In particular, the block diagrams included in the attached figures and described below are not to be understood as a representation of the structural features, i.e. constructional limitations, but must be understood as a representation of functional features, i.e. intrinsic properties of the devices defined by the effects obtained, that is to say functional restrictions, which can be implemented in different ways, so as to protect the functionalities thereof (operational capability).

In order to facilitate the understanding of the embodiments described herein, reference will be made to some specific embodiments and a specific language will be used to describe the same. The terminology used herein is used for the purpose of describing particular embodiments only and is not intended to limit the scope of the present invention.

As it will be apparent from the disclosure set in the following paragraphs, the present electronic system brings together multiple sensors acquiring different and complementary information, specifically both biometric and images, that intrinsically provide a thorough view to an end user of a subject’s cognitive level. In particular, the sleep onset prediction capability is based on the seamless interaction among the different sensing and processing units of the electronic system itself; as a result, a broader coverage of comer cases may be obtained, providing a robust solution with respect to the ones currently available.

As better described below, the use of multiple sensors (and, in general, sensing technologies) as well as computational units in the present electronic system aims at improving:

- sensitivity (namely, percentage of correct diagnoses, given that the subject is actually drowsy);

- specificity (namely, percentage of correct diagnoses, given that the subject is actually awake);

- accuracy (namely, percentage of correct diagnoses, considering both drowsiness and awakeness diagnoses); and

- robustness (namely, in terms of fail-operational/safe performances of the system).

Figure 1 shows an electronic system 1 designed to monitor and/or predict behavioural states and/or transitions thereof to determine a sleepiness level of a subject and comprising:

- a sensory system 2 configured to output at least one vision signal and a biometric signal indicative of respective physiological quantities related to the subject; and

- an electronic processing unit 3 in communication with the sensory system 2 and being configured to receive the corresponding vision and biometric signals and process them separately to output corresponding data.

The electronic system 1 is designed to process the data outputted by electronic processing unit 3 to determine a behavioural state and/or transitions thereof to determine the sleepiness level of the subject and generate the feedback to provide to an end user, here the subject.

According to an aspect of the present invention, the sensory system 2 comprises:

- a vision sensor 4 configured to detect a visual physiological quantity (e.g. eyes movement, pupil area analysis, etcetera) and generate the vision signal; and

- a contactless sensor 5 (e.g. radar, imaging PPG and similar) configured to detect a biometric physiological quantity (e.g. heart rate, heart rate variability, breathing rate, percentage of oxygen in the subject’s blood, SpCh, etcetera) and generate the biometric signal.

According to an aspect of the present invention, the electronic system 1 further comprising a control logic 6 in communication with electronic control unit 3 and configured to receive the data outputted by electronic processing unit 3 and process them to determine a behavioural state and/or transitions thereof to determine the sleepiness level of the subject and make the electronic system 1 to provide for a corresponding feedback to be provided to the end user.

As shown in the embodiment of Figure 1, the electronic processing unit 3 is powered by a respective power supply unit 7.

As furthermore shown in the embodiment of Figure 1, the electronic system 1 further comprises a further electronic processing unit 8 in communication with the electronic processing unit 3 and the control logic 6; therefore, in the embodiment of Figure 1, the electronic processing unit 3 and the further electronic processing unit 8 are configured to receive the vision signal and the biometric signal respectively and process them separately to output corresponding vision and biometric data relative to the behavioural state of the subject. Furthermore, the further electronic processing unit 8 is powered by a respective power supply unit 9 in communication with the further electronic processing unit 8 thereof. Therefore, according to the embodiment of Figure 1, the electronic system 1 is based on two different processing units (also referred to as PUs) which are respectively tailored to vision algorithm processing task and the contactless sensor task to process the vision signal and the biometric signal respectively.

Furthermore, according to an aspect of the present invention, the control logic 6 synthesises the information coming from the processing units 3, 8 with the aim of performing a more precise diagnosis about the sleepiness level of the driver, in terms of sensitivity, specificity and accuracy. Moreover, the control logic 6 is configured to ensure an improved level of functional safety by assessing in real-time the hardware/software functional integrity of the sensory system 2.

The electronic system 1 further comprises a communication unit 10 in communication with the electronic processing unit 3, as well as the further electronic processing unit 8 in the embodiment of Figure 1, and the sensory system 2 and configured to:

- receive the visual and biometric signals from the sensory system 2; and

- transmit the feedback to the end user and to the sensory system 2.

According to an aspect of the present invention, the communication unit 10 is also configured to manage the synchronisation between the processing units 3, 8. Furthermore, in order to transmit the feedback to the end user and to the sensory system 2, the communication unit 10 is configured to send a feedback to the end user through external units or devices, such as mobile devices, i.e. send a notification (e.g. vocal messages, haptic, etcetera) to the end user.

The electronic system 1 further comprises an input/output, I/O, unit 11 in communication with the electronic processing unit 3, as well as the further electronic processing unit 8 in the embodiment of Figure 1, and configured to manage the transmission of analogue/digital signals between the components of the electronic system 1.

Thus, the present electronic system 1 has a redundant from a hardware/software point of view thanks to:

- the presence of independent power supply units 7, 9;

- the presence of different and independent electronic processing units 3, 8;

- the presence of different software running on the electronic processing units 3, 8 and based on different input signals, namely the vision and biometric signals; and

- the presence of different types of sensors.

Figure 2 shows an alternative embodiment of the electronic system 1 of Figure 1; in particular, elements and components that are common between Figures 1 and 2 are referred to in Figure 2 with the same reference number and will not described further hereinafter.

In particular, in Figure 2, the control logic 6 and the electronic processing unit 3 are integrated in a safe microcontroller 12, in particular communication with the further electronic processing unit 8; furthermore, the power supply unit 7 and the further supply unit 9 are integrated in a safe power supply unit 13 configured to power the safe microcontroller 12 and, according to the embodiment of in Figure 2, the further electronic processing unit 8.

The electronic system 1 of Figure 2 is thus a cost-effective solution wherein integrated architectural options were considered, namely: - collapsing the power supply units 7, 9 in a single safe power supply unit 13, still featuring redundancy concepts and integrated watch-dog; and

- collapsing the control logic 6 and the electronic processing unit 3 in the safe microcontroller 12, which is based on multi-core architecture and specifically designed for safety critical automotive applications.

According to a further aspect of the present invention, not described herein, the control logic 6 can also be integrated in the safe microcontroller 12 with the further electronic processing unit 8.

Figure 3 shows an alternative embodiment of the electronic system 1 of Figure 1 and Figure 2; in particular, elements and components that are common between Figures 1, 2 and 3 are referred to in Figure 3 with the same reference number and will not described further hereinafter.

In particular, considering the even more widespread use of wearable devices for assessing the health status of the subject, the Applicant noted that the electronic system 1 could benefit from the potential interaction with smart wearables; Figure 3 in particular shows the electronic system 1 and a wearable device 14, in communication with each other and wherein the latter is configured to transmit and receive information relating to the status of a subject.

The Applicant notes that the use of the wearable device 14 is beneficial on different aspects, namely:

- it extends the health monitor covering periods outside the vehicle mission (e.g. while sleeping); and

- it addresses the functional safety and in particular it improves the availability of the electronic system 1 in case of a hardware failure.

Considering the embodiment of Figure 3, the electronic system 1 implements tripleredundancy scheme; in this way, a voting mechanism could be implemented through a truth table based on the independent signal measurement, from different sensors (/'.<?. the sensory system 2 and eventually the wearable device 14), and computing, from different PUs (i.e. the electronic processing units 3, 8). Consequently, a graceful degradation mode can be applied, still running the sleep-prediction algorithm based on a single sensor and then informing the driver that the electronic system 1 is affected by hardware problems and maintenance operation is needed.

On this regard, the Applicant notes that, in the solution of Figure 3, firstly, combining the use of a wearable device 14 with the electronic system 1 seamlessly allow to extend the health monitoring of the subject over the day and not limited on the time spent in the vehicle; in this way, the electronic system 1 is able to retain details about health status of the subject prior entering in the vehicle and, in the same manner, the post-driving phase, thereby retaining details from the previous activity in the vehicle. The same holds for the activities performed before entering in the vehicle, including sleep quality. Such information are used to set the daily baseline, often referred to “body battery", thus better defining the “fit-to-drive" capabilities of the subject; in this way, the present electronic system 1 provides a new way of monitoring critical subjects on long term period, in a fully non-intrusive manner, thus identifying possible health trends and supporting prognostic measures.

Furthermore, the redundancy of the electronic system 1 improves also the quality of the performance; as a matter of fact, some vital parameters (e.g. HR and RR) are acquired with different type of sensors. In particular, the RR measurement is more precise when measured by a RADAR, while the HR is more precise when measured by a wearable device such as the wearable device 14; thus, it is possible to select the most accurate sensing technology.

Furthermore, the smart combination of the information acquired from vision sensor 4 and the contactless and wearable sensors 5, 14 is important to mitigate artifacts from movements. In fact, both contactless and wearable sensors 5, 14 are sensitive to the motion of the subject. While the vision sensor 4 easily allows to assess if the subject is fully awake and, if so, the measurement from contactless and wearable sensors 5, 14 could run in a sort of “active mode” and more easily discarding wrong values affected by motion. As soon the subject enters a more relaxed status, preceding a potential drowsy state, the measurement from contactless and wearable sensors 5, 14 could run in a sort of “drowsy mode”, thus intensifying the processing as more valid data, less affected by motion artefacts, are available.

In addition, the electronic system 1 according to the present invention can be adapted for after-market and low-cost applications by combining the functionalities of the vision sensing and the biometric sensing from the wearable device 14 thus avoiding the use of the RADAR sensing (e.g. both processing and sensing), which is an expensive device.

Figure 4 shows an alternative embodiment of the electronic system 1 of Figure 3; in particular, elements and components that are common between Figures 3 and 4 are referred to in Figure 4 with the same reference number and will not described further hereinafter. Here the sensory system 2 only comprises the vision sensor 4.

The present invention has several advantages.

In particular, the present invention allows to achieve:

- an improved overall accuracy of the measurement by combining different sensors with different capabilities; and

- an improved system availability.

In further detail, the present electronic system 1 is provided with an architecture that foresees also the possibility to seamlessly integrate information from nomadic devices (e.g. smartwatch, smart wearable sensors and similar devices) in order to extend the monitoring activity beyond a vehicle mission.

Furthermore, the present electronic system 1 complies with the existing GSR and already addresses future normative needs while providing much improved desired diagnostic performances as well as functional safety and SOTIF (Safety Of the Intended Functionality).