Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
TRIGGERING AUTONOMOUS CONTROL BASED ON DRIVER COGNITIVE LOAD
Document Type and Number:
WIPO Patent Application WO/2020/126020
Kind Code:
A1
Abstract:
A method for triggering an onset of a vehicle control function by a vehicle control unit (120)in a vehicle (100).The method comprises; obtaining sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant (230), estimating a present level of occupant cognitive load based on the data, comparing the present level of occupant cognitive load to pre- determined trigger criteria; and if the present level of occupant cognitive load meets the pre- determined trigger criteria, initiating a vehicle control function by the vehicle control unit.

Inventors:
ANDERSSON STEFAN (SE)
BRUNNEGARD OLIVER (SE)
LARSSON ANNIKA (SE)
THOR JOHAN (SE)
LANG JOHN (SE)
Application Number:
PCT/EP2018/086443
Publication Date:
June 25, 2020
Filing Date:
December 20, 2018
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
VEONEER SWEDEN AB (SE)
International Classes:
B60W40/08; B60K28/06; B60W50/00; B60W50/08
Foreign References:
US20180009442A12018-01-11
US20180120837A12018-05-03
US20180281808A12018-10-04
US20180009442A12018-01-11
DE102016215302A12018-02-22
JP2017174058A2017-09-28
Attorney, Agent or Firm:
SANDSTEDT, Jonas (SE)
Download PDF:
Claims:
CLAIMS

1. A method for triggering an onset or cessation of a vehicle control function by a vehicle control unit (120), the method comprising obtaining (SI) sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant (230), estimating (S4) a present level of occupant cognitive load based on the data, comparing (S5) the present level of occupant cognitive load to pre-determined trigger criteria; and if the present level of occupant cognitive load meets the pre determined trigger criteria, initiating (S6) a vehicle control function by the vehicle control unit .

2. The method according to claim 1, wherein the at least one vehicle occupant comprises a driver (230) of the vehicle (100) .

3. The method according to any previous claim, wherein the at least one vehicle occupant comprises one or more passengers of the vehicle (100) .

4. The method according to any previous claim, comprising obtaining (S2) vehicle position data related to a vehicle environment context, wherein the pre-determined trigger criteria are at least partly based on the vehicle environment context.

5. The method according to any previous claim, comprising obtaining (S3) data related to a vehicle driving scenario context, wherein the pre-determined trigger criteria are at least partly based on the vehicle driving scenario context.

6. The method according to any previous claim, wherein the sensor data comprises gaze direction data in relation to a road on which the vehicle travels, and/or gaze concentration data indicating whether the occupant gaze is biased towards the road.

7. The method according to any previous claim, wherein the sensor data comprises velocity data, and/or steering micro movement data and/or steering wheel reversal data, and/or lane positioning data indicating a position or track of the vehicle (100) in relation to a lane on the road on which the vehicle travels, or in relation to a road boundary of the road on which the vehicle travels.

8. The method according to any previous claim, wherein the pre-determined trigger criteria comprise one or more thresholds against which the present level of occupant cognitive load is compared .

9. The method according to any previous claim, wherein the pre-determined trigger criteria comprise one or more time- factors arranged to reduce influence of transient sensor data.

10. The method according to any previous claim, wherein the pre-determined trigger criteria are configured in dependence of a present vehicle occupant identity.

11. The method according to any previous claim, wherein the vehicle control function comprises an autonomous vehicle drive function or a function for requesting vehicle control by the control unit or a function for re-configuring a steering system of the vehicle to be stiffer compared to a default steering system reconfiguration or a vehicle control function configured to reduce vehicle speed or to reduce a vehicle acceleration performance .

12. The method according to any previous claim, wherein the vehicle control function comprises uploading the sensor data, and/or the estimated present level of cognitive load to a remote server (140) .

13. The method according to any previous claim, comprising estimating a future occupant cognitive load based on previous data obtained as sensor data or cognitive load data from the vehicle and/or from a remote server (140) .

14. A vehicle control unit (120) arranged to trigger an onset or cessation of a vehicle control function, the control unit comprising;

an obtaining module (Sxl) configured to obtain sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant (230),

an estimating module (Sx4) configured to estimate a present level of occupant cognitive load based on the data,

a comparing module (Sx5) configured to compare the present level of occupant cognitive load to pre-determined trigger criteria; and

an initiating module (Sx6) configured to, if the present level of occupant cognitive load meets the pre-determined trigger criteria, initiate a vehicle control function by the vehicle control unit.

15. A vehicle (100) comprising the vehicle control unit

(120) according to claim 14.

Description:
TTTTV

TRIGGERING AUTONOMOUS CONTROL BASED ON DRIVER COGNITIVE LOAD

DESCRIPTION OF THE DISCLOSURE

The present disclosure relates to vehicle sensors and control units for autonomous or semi-autonomous drive. There are disclosed improved methods for triggering an onset or cessation of vehicle autonomous control by a vehicle control unit based on a present cognitive load of one or more vehicle occupants, such as a vehicle driver. There present disclosure also relates to remote servers for processing cognitive load data obtained from one or more vehicles.

A vehicle may comprise a plurality of external sensors, such as cameras, infra-red detectors, radio detection and ranging (radar) sensors and light detection and ranging (lidar) sensors. These external sensors monitor the environment surrounding the vehicle to, e.g., detect objects in a vicinity of the vehicle.

Vehicles may also comprise internal sensors configured to, e.g., monitor the driver and any other occupants in the vehicle. These sensors can be used to determine a state of the driver, such as a present cognitive load, which can be used as input to various vehicle functions;

US20180009442 discloses a cognitive load driving assistant which performs actions to decrease driver cognitive load when high driver cognitive load levels are detected. Examples of such actions comprise reducing volume of an audio entertainment system and modifying a driving route to one associated with a reduced cognitive load.

DE102016215302 uses head movement of a vehicle driver to estimate driver cognitive load. It is proposed to adjust driver assistance systems such as emergency brake assistance systems and lane departure warning systems based on the estimated cognitive load of the driver.

JP2017174058 discusses methods for attracting the attention of a driver who is under an inappropriate cognitive load. Cognitive load is here determined based on a sensed steering angle.

It is an object of the present disclosure to provide methods and systems for triggering onset of vehicle autonomous control, which further exploit estimated cognitive loads of vehicle occupants to increase safety and acceptance of autonomous systems .

The object is obtained by a method for triggering an onset or cessation of a vehicle control function by a vehicle control unit. The method comprises obtaining sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant and estimating a present level of occupant cognitive load based on the data. The plurality of sources may, for example, comprise any of gaze data, steering angle data, pedal action data, foot position, driver pose, vehicle lane positioning, and vehicle speed over time. The method also comprises comparing the present level of occupant cognitive load to pre-determined trigger criteria, and, if the present level of occupant cognitive load meets the pre-determined trigger criteria, initiating a vehicle control function by the vehicle control unit.

Safety is improved in that vehicle occupants under too high and/or too low cognitive loads are identified and countermeasures taken to compensate accordingly. The driving experience is also improved in that annoyance and stress is reduced .

Advantageously, the method may act on both too high and too low cognitive load levels. The disclosed methods are robust, in that sensor data related to occupant cognitive load are obtained from a plurality of sources.

Further advantages are obtained by the features set out in the dependent claims. There are also disclosed herein arrangements, systems, electronic control units (ECU) , vehicles and computer program products associated with the above-mentioned advantages.

The present disclosure also relates to remote server systems arranged to obtain cognitive load data from one or more vehicles, and to process the cognitive load data in order to improve, e.g., vehicle route selection and to provide early warning of upcoming scenarios related to an inappropriate level of cognitive load.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will now be described in detail with reference to the appended drawings, where:

Figure 1 illustrates a vehicle with sensors and a control unit;

Figure 2 schematically shows a vehicle interior space;

Figure 3 is a flow chart illustrating methods;

Figure 4 schematically shows a control unit;

Figure 5 illustrates an example computer program product;

Figure 6 shows an example remote server with connected vehicles; and

Figure 7 schematically illustrates a cognitive load map.

DETAILED DESCRIPTION

Aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings. The different arrangements, devices, systems, computer programs and methods disclosed herein can, however, be realized in many different forms and should not be construed as being limited to the aspects set forth herein. Like numbers in the drawings refer to like elements throughout.

The terminology used herein is for describing aspects of the disclosure only and is not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

The terms cognitive distraction and cognitive load are often used synonymously to refer to a reduced ability to perform a task requiring a level of concentration or attention, such as driving a vehicle. Cognitive distraction is often used to refer to a more general concept related to the diversion of attention away from a given activity, e.g., driving, and toward some competing activity, like interacting with an in-vehicle entertainment system or cellular phone. By contrast, cognitive load typically refers to the "amount" of cognitive resources demanded from, e.g., the driver by a competing activity or from a driving context. For instance, a driving scenario with high traffic density in a complicated urban environment is usually associated with a higher level of cognitive load compared to a scenario involving driving on a country-side road without much traffic. However, a driver driving on that same country-side road may experience high cognitive load in case of reduced visibility or slippery road condition.

Cognitive distraction may sometimes occur in situations with high as well as low cognitive load, for example, in the case of mind wandering.

Aspects of the present disclosure relate to both cognitive load and cognitive distraction. Both visual distraction and mental distraction are considered. The Yerkes-Dodson law is an empirical relationship between arousal and performance. The law dictates that performance increases with physiological or mental arousal, but only up to a point. When levels of arousal become too high, performance decreases. The process is often illustrated graphically as a bell-shaped curve which increases and then decreases with higher levels of arousal. Arousal is related to cognitive load. Consequently, there is a desired level of cognitive load under which a driver of a vehicle performs best. Too low cognitive loads may lead to mind wandering and prolonged reaction times, which are undesirable. Too high cognitive load may also lead to a reduced driving performance, such as missing important road signs and failing to notice objects close to the vehicle.

The techniques disclosed herein involve the vehicle estimating a present cognitive load level of a driver or vehicle occupant using a plurality of sensor sources. In case the cognitive load level reaches too high levels, or too low levels, the vehicle will act and perform an autonomous vehicle function. For instance, the vehicle may request control or simply assume control from the driver. The vehicle may also change some control characteristics of the vehicle, like reducing speed or acceleration. The vehicle may also re-configure certain key vehicle parameters in case of a driver experiencing high levels of cognitive load. For instance, the steering system can be made stiffer to dampen excessive steering maneuvers.

Cognitive load can be detected in many ways. Some key aspects of the present disclosure comprise using several different information sources to estimate a present level of cognitive load in a vehicle occupant. This is an advantage since the cognitive load estimation becomes more robust. Also, personalization can be improved by using different sensor sources, since individuals give indications of cognitive load level in different ways. Thus, some types of sensor data become more relevant than other types depending on the vehicle occupant. A level of driver cognitive load can be estimated by, e.g.;

Determining if the gaze of the driver is on the road, or is directed elsewhere;

Determining if a gaze direction of the driver is pointing in a constant direction (often forward) , or if the drivers gaze is wandering around possibly in an erratic manner;

Determining if the driver is executing consecutive steering micromovements, which means that the driver is applying consecutive and rapid adjustments of steering wheel angle or is associated with a high steering wheel reversal rate;

Determining if the driver is maintaining a constant vehicle speed, or of the driver is performing a series of smaller speed changes .

Determining if the driver drives in some key patterns associated with confusion, such as taking more than one turn in a round about .

Steering wheel reversal rate is generally defined as the number, per minute, of steering wheel reversals larger than a certain minimum angular value, often referred to as the gap size. Roughly, given a steering wheel angle signal A(t), a steering wheel reversal is taken to be a portion [ 11 ; 12 ] of the signal such that A is stationary at both tl and t2 (i.e. dA(tl)/dt = 0 and dA(t2)/dt = 0), and such that |A(tl) - A(t2) | > Amin, where Amin is the gap size.

It is appreciated that the cognitive load of a vehicle driver is important in the types of systems discussed herein. However, other vehicle occupants may also experience cognitive loads that may be of importance to a vehicle control system. For instance, a high level of cognitive load in a vehicle passenger may be undesirable. For instance, a taxi service may desire to provide a certain cognitive load experience for its customers. Thus, it may be appropriate in some scenarios to activate a vehicle control function which leads to a reduced cognitive load of the passenger as well as the driver. Autonomous drive taxi services may want to monitor cognitive load levels of the passengers, since no driver cognitive load level can be monitored.

Herein, a vehicle position is a location of the vehicle in some reference system, like the World Geodetic System 1984 (WGS 84), which can be obtained from, e.g., an on-board global positioning system (GPS) .

A vehicle is associated with different contexts which are relevant for determining what effects can be expected from a given level of cognitive load. A vehicle environment context indicates what type of environment the vehicle is in, like a suburban or urban environment, a narrow city center road or a wide countryside road. A vehicle driving scenario context indicates what type of driving scenario that is experienced in the environment context. For instance, night-time driving or low visibility driving on a country-side road is likely to affect a driver differently compared to a clear sky day-time driving scenario context. Heavy traffic also affects most drivers differently compared to sparse traffic when it comes to driver reaction and tolerance to various levels of cognitive load.

The present disclosure relates in part to initiating one or more vehicle control functions. It is appreciated that the term vehicle control functions are given a broad interpretation herein. Consequently, a vehicle control function may be associated with the vehicle assuming control over drive functions, but also simply requesting control, or adjusting some vehicle parameters like steering wheel stiffness.

Figure 1 schematically illustrates a vehicle 100 travelling on a road 130. The vehicle is moving in a forward direction F. The road has two lanes 131, 132. The vehicle comprises one or more on-board sensors 110, configured to communicate with a vehicle control unit 120. The one or more on-board sensors may, e.g., comprise a camera, an infra-red detector, a radar sensor, and/or a lidar sensor. The vehicle control unit 120 will be discussed in more detail below in connection to Figure 4.

Communication and collaboration between driver and artificial intelligence (AI) control is likely to determine consumer adoption of new technologies. With new automated driving technologies slated to soon become available to consumers, most notably enabling vehicles to function with limited human involvement in a greater number of specific conditions, the need will emerge to be able to not just use but trust automated assistance in quickly emerging and complex situations.

Drivers (and passengers) must trust that automated systems will make the right decisions. Said systems must have the capacity to discern and respond to different driver skill levels and emotions. This human machine interaction must not only sometimes perform within seconds, but the give-and-take must result in learning, with collaboration getting better over time.

Consumer adoption of autonomous technology is likely to depend in large part on how the autonomous operations are experienced. If handled badly by automation, they'll result in a large decline in driver trust, which may take a long time to recover from, as some drivers may not use the automation feature for a long period (thereby negating the safety benefit of the feature) .

For the foreseeable future, the experience of driving will remain a dynamic system that presents an almost infinite combination of externalities, like weather, road conditions, and connectivity, thereby rendering automation unable to be perfectly reliable in every circumstance. Simply put, drivers will still need to be able to drive, when necessary, making driver understanding of the system within which she or he is embedded, and the system' s understanding of the driver, essential for achieving safety.

Lack of trust is already a major roadblock for adoption for currently available vehicle technologies. Many drivers disable usage of vehicle automation, such as lane keep assistance or adaptive cruise control, citing their belief that the functions are unreliable, provide feedback at the wrong times, or are simply annoying.

There's no reason to believe that the introduction of more sensors or services alone will automatically address driver understanding and expectations. The ultimate remit for autonomous driving technology is to innovate trust. This means innovating not only when but how notifications are shared but going beyond warnings to encompass the design of system actions as well as system availability. If the automated and human actors in a vehicle can make reasonable assumptions of the others' performance and skill, collaboration could be implemented using existing functions in novel ways, such as adaptive cruise control and lane keep alerts. Better communication about capabilities and understanding of driver and vehicle roles are tools; trust is the outcome.

Trust means continuing to use a system or service even though it may fail sometimes, as failure is a learning opportunity for intelligent systems much as it is for an intelligent creature.

In the future, automated vehicles will adjust to driver preferences and learn as they drive, rather than only providing the binary option of turning on or off. Vehicles will even take initiative as an active system rather than passively awaiting commands .

Currently, driver engagement in automated driving systems amounts to little more than the ability to switch the system on and off, and sometimes set parameters such as cruise control speed, or initiating lane changes. In response, the system shows an icon if it is active, sees a lead vehicle, or recognizes lane markings. This communication between driver and system is basically a visual or haptic monologue during a short time window, not a dialogue within a shared context of collaboration and experience.

From the consumer and user perspective, this makes it very difficult to communicate system capabilities. Drivers use trial- and-error for figuring out what the system is able to do, then make their assumptions of whether they can perform non-driving tasks safely. Instead of reducing risk, such behaviors increase it for a period of time as drivers figure out how and if they are "driving" or "riding".

This means the vehicle system needs to be capable to cope with all events that might affect driver and occupant safety, as well as driver and occupant trust in the system. Not only does the driver need to trust automation to actually use it, but the automation needs to trust the driver to handle situations and communicate if it needs to interfere in normal situations. If not, drivers may feel annoyed. These circumstances can be complex, and emerge quickly, and a vehicle's actions and responses will have a large impact on how the driver will trust the vehicle in the future.

Consider a situation in which a child in the back seat drops a toy and erupts in a fit, drawing the driver's attention away from the road, or a passenger begins to feel sick because of the precision by which the vehicle can automatically speed up or slow down.

When driving, this means the vehicle system will need to keep the driver engaged in the driving task by collaborating. That way, the driver will not be made to suddenly have automation interfere in a potentially surprising situation but will have learned during normal driving that they and the vehicle need to cooperate. Such driver-vehicle collaboration can constitute establishing a safeguard and perimeter of protection where automation, rather than human, does the monitoring. Automation, after all, can monitor tirelessly for extended periods of time. Such safeguarding will need to affect vehicle dynamics by reducing risk and making sure that if the driver' s attention is directed to a non-driving related task, the vehicle can still keep its occupants and vulnerable road users safe by reducing speed and driving more conservatively. The collaboration between driver and vehicle will be a partnership, based on trust, that both parties will have a hand in building. And, even though a vehicle might perform within operational tolerances, the movement could well cause car sickness for one or more passengers, so it means adjusting vehicle dynamics and possibly driving related visual cues to a point where the occupants feel well .

The vehicle needs to get to know not only its driver, but also its occupants to handle situations successfully. The resulting two-way trust is the building block of collaboration and makes good on the safety promise of any system.

For adoption of advanced vehicle automation, altering the user experience dynamic to one of more collaboration will require significant innovation in system functionality, communication, and system self-monitoring capabilities. The design of system logic and interfaces inside and outside the vehicle will be as important as those functional improvements, too, since the human-machine interface (including the driving task) needs to provide a real-time understanding of what the other can and cannot do (and perceive) ; otherwise, some situations are likely to cause a breakdown in two-way trust between drivers and their vehicles, and make subsequent uses of said systems less likely.

People drive differently in different situations, whether determined by internal characteristics (tired, agitated, distracted) , or external circumstances (some people are just uncomfortable driving in rain or snow) . It is vital for future human vehicles and systems to learn about individual drivers and specific tolerances, if only to avoid false positives and alerts that might cause them to disengage safety or comfort systems.

This does not mean that we will need to sacrifice understandability or predictability, but rather explore ways that drivers can instruct their cars on what to do in a more nuanced fashion that suits their personalities and proclivities. This also goes for the vehicles themselves, so learning how to be clear about things as simple as software or map updates should key into driver expectations and preparation.

Vehicles will need to be more flexible in their functionality depending on external situations, much as humans are. Further, drivers will need to learn how to trust their vehicles as well as understand their own roles. Successfully addressing these factors could well be the key to determining consumer adoption of autonomous driving tools, not to mention delivering on the safety benefit promised from their use.

There is disclosed herein examples comprising an artificial intelligence-equipped car that can understand and respond to context, using external and internal sensing combined with algorithmic AI to create a unified assessment of what is going on with the occupants, vehicle, and driving situation, and then acting based on this evaluation.

According to some aspects, deep learning algorithms senses driver gaze, emotion, cognitive load, drowsiness, hand position, posture, and then fuses this information with data on the external environment to yield driving experiences that are not only safer, but feel that way, too.

For instance, some vehicle systems discussed herein is able to recognize when a driver appears preoccupied or distracted, and automatically increase the vehicle's following distance - and if the driver appears confused, the vehicle can explain its decision, helping the driver and passengers learn about the vehicle functionality. When traveling with this type of vehicle, drivers won't get annoyed by too-frequent warnings and shut systems off, but will only perceive the systems when relevant, increasing both functional and perceived safety benefits.

Figure 2 schematically illustrates a vehicle interior with a vehicle driver 230. One or more interior sensors 210 are arranged to monitor various aspects of the driver 230 and the vehicle interior. The internal sensors 210 may comprise, e.g., one or more cameras to obtain visual data of the driver, a steering wheel sensor giving information about present and past steering wheel angles, a pedal sensor configured to detect on which pedal or pedals, if any, the driver's feet are located.

The control unit 120 is arranged to estimate a present level of occupant cognitive load based on the data obtained from the plurality of sensors. Depending on the outcome of the estimation one or more vehicle control functions may be initiated, such as a request by the vehicle for autonomous control or a take-over by the vehicle of various aspects of vehicle control.

Figure 3 is a flow chart illustrating methods. In particular, there is illustrated a method for triggering an onset or cessation of a vehicle control function by a vehicle control unit 120. Examples of the vehicle control function will be given below .

It is appreciated that a vehicle control function, herein, is not just a function for autonomously driving the vehicle. Other control functions are also considered, such as requests by the control unit, and re-configuration of vehicle parameters. It is appreciated that the disclosed methods can be used both for triggering an onset of some function, and for stopping the same function, or for stopping some other function.

The method comprises obtaining SI sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant 230. Examples of how this obtaining can be implemented will be discussed below. Many different data sources can be utilized in estimating cognitive load. For instance, drivers and other vehicle occupants exhibiting different levels of cognitive load are likely to have different gaze patterns, eye movements, foot positions, and poses. Drivers and other vehicle occupants are also likely to control the vehicle differently with respect to speed control and lane positioning.

The disclosed methods mainly relate to estimating cognitive load of the vehicle driver, however, the methods are also applicable to estimating cognitive load of other vehicle occupants, such as vehicle passengers. It is appreciated that future vehicles may not necessarily be associated with a specific driver. Rather, any vehicle occupant may assume control of the vehicle in some way or may be relieved of certain control functions due to having too low or too high level of cognitive load.

According to some aspects, the method comprises obtaining S2 vehicle position data related to a vehicle environment context.

According to some other aspects, the method comprises obtaining S3 data related to a vehicle driving scenario context.

Data such as the above may for instance be in the form of vehicle position estimates, and/or camera data indicating what the exterior environment looks like. Radar and lidar data may also be used to determine if there are many moving objects in a vicinity of the vehicle 100, or if the driving environment context does not comprise many moving objects in the vehicle vicinity .

According to some aspects, the sensor data comprises gaze direction data in relation to a road on which the vehicle travels, and/or gaze concentration data indicating whether the occupant gaze is biased towards the road or in some other direction. An increased level of 'eyes on the road' behavior is indicative of increased cognitive load. Also, gaze concentration, which means that the vehicle occupant gaze is focused in some direction, often a forward direction, is indicative of an increased level of cognitive load.

According to an example, a driver which regularly, with a high frequency, shifts gaze direction is estimated to have an increased cognitive load. In other words, a driver which does not rest his/her eyes on the road, but has a more erratic gaze direction, is estimated to be under an increased level of cognitive load.

According to some aspects, facial expression and pose is used to estimate cognitive load. For instance, a driver having a surprised or chocked facial expression, or pose, is estimated to be under an increased cognitive load.

According to some aspects, the sensor data comprises velocity data, and/or steering micro movement data and/or steering wheel reversal data, and/or lane positioning data indicating a position or track of the vehicle 100 in relation to a lane on the road on which the vehicle travels, or in relation to a road boundary of the road on which the vehicle travels. A driver exhibiting a higher degree of velocity adjustment is likely to have a higher level of cognitive load. The same is true for a driver which positions the vehicle more accurately in relation to the lane than normal. In other words, the more accurate driving the driver exhibits, the more cognitively loaded the driver may be. An increased level of steering micromovements, such as increased steering wheel reversal behavior, is also indicative of an increased level of cognitive load. A driver exhibiting a reduced level of velocity adjustment is likely to be under a lower level of cognitive load. The same is true for a driver which positions the vehicle less accurately in relation to the lane. In other words, the less accurate driving the driver exhibits, the less cognitively loaded the driver may be. A reduced degree of steering micromovements, such as steering wheel reversal behavior, is also indicative of a reduced level of cognitive load.

According to some aspects, certain driving behavioral patterns are assumed associated with an increased level of cognitive load. For instance, a driver taking two successive turns in a round about is estimated to be under an increased level of cognitive load .

The different sources of information related to vehicle occupant load may be treated separately or merged into a single statistic. The merging may then result in a value on some pre-determined scale. A suitable implementation of cognitive load scales and merging may be obtained by experimentation or by computer simulation .

For example, a scale of 0-100 may indicate estimated driver cognitive load based on driver gaze, and another scale of 0-100 may indicate estimated driver cognitive load based on velocity data. Experimentation may then connect certain driver gaze behavior in a number of reference scenarios with the scale. Experimentation may also connect certain velocity data in a number of reference scenarios with the scale. The two scales may then be merged based on some weights, which weights may also be determined based on experimentation. Consequently, a driver having erratic gaze and uneven vehicle velocity will be estimated as being under a relatively large cognitive load, perhaps around 80 on the scale from 0-100. A driver having a less erratic gaze behavior and a more constant vehicle velocity will be estimated as being under a relatively normal cognitive load, perhaps around 50 on the scale from 0-100.

Weighting two or more sensor data sources may be performed by multiplying one source estimate on a scale by a first weight and multiplying another source estimate on a scale by a second weight, followed by summing and normalization using the sum of the two weights. I.e., C= (Cl *W1+C2 *W2 ) / (W1+W2 ) , where Cl is the first cognitive load estimate, C2 is the second cognitive load estimate, and W1 and W2 are the two weights.

Referring again to Figure 3, the method comprises estimating S4 a present level of occupant cognitive load based on the obtained data and comparing S5 the present level of occupant cognitive load to pre-determined trigger criteria.

According to some aspects, the pre-determined trigger criteria comprise one or more thresholds against which the present level of occupant cognitive load is compared.

This allows for a comparison routine which is of relatively low complexity. The different types of sensor data are processed to generate a single measure of cognitive load, say on a scale from 0 to 100. The estimated level of cognitive load is then compared to the threshold, and if the threshold is breached then the vehicle control function is triggered. Connecting back to the example above, the value C may be compared against a threshold T, wherein a result C<T results in some action and another result OT results in some other action.

It is appreciated that there may optionally be both high and/or low thresholds implemented. This way a too low cognitive load in a, e.g., a driver, may result in triggering a first vehicle control function, while a too high cognitive load triggers a second vehicle control function.

According to other aspects, the pre-determined trigger criteria comprise one or more time-factors arranged to reduce influence of transient sensor data. The time factors serve to reduce impact from transient data. For instance, the threshold comparison routines discussed above may be associated with a time factor in that it is required that the estimated level of cognitive load stays above or below the threshold for a certain amount of time before the vehicle control function is triggered. In other words, the time factors function like a low-pass filtering, or averaging, or the sensor data, such that anomalies and intermittent erroneous sensor data readings are suppressed.

According to an example, the period may be set at 10 seconds, or 30 seconds. Thus, a period of, say, 6 seconds of high cognitive load will not trigger the control function, while a period of 30 seconds of high cognitive load will trigger the control function.

According to further aspects, the pre-determined trigger criteria are configured in dependence of a present vehicle occupant identity. It is appreciated that persons react differently to different cognitive load scenarios. It may therefore be advantageous to personalize the pre-determined trigger criteria, such that one person may trigger the control function at some cognitive load level while some other person triggers the control function at some other cognitive load function. As noted above, some drivers may not like driving in snow, while other drivers may not have any issues with driving in snow.

It is also appreciated that the cognitive load data may be dependent on the person who generated it, necessitating some degree of calibration to obtain a high degree of performance. For instance, some persons may be associated with pronounced facial expressions, while some other persons are associated with less pronounced facial expressions. Also, some people may be more prone to changing vehicle drive characteristics under high cognitive load than others.

Advantageously, the personalization is iteratively adjusted in response to driver reaction, or in response to vehicle occupant input. For example, the system may comprise a feedback system, like buttons or a voice-controlled feedback system, where a vehicle occupant may inform the system when the vehicle occupant feels the system did something annoying or triggered the vehicle control, function too often, or too seldom. Some further key aspects of the present disclosure relate to the pre-determined trigger criteria which are used in the comparison .

In case vehicle environment context data has been obtained, the method may, advantageously, base the pre-determined trigger criteria at least partly on the vehicle environment context. Thus, cognitive load levels can be evaluated against different trigger criteria in different vehicle environment contexts. For instance, a lower cognitive load may be acceptable when driving on a country-side road compared to when driving in a busy city center, since the two environment contexts may not require the same reaction times. Also, a somewhat increased cognitive load can be expected from most drivers in busy city center driving contexts, making an adjustment of trigger criteria warrantable.

With reference again to Figure 3, if the present level of occupant cognitive load meets the pre-determined trigger criteria, the proposed method initiates S6 a vehicle control function by the vehicle control unit.

According to aspects, the vehicle control function comprises an autonomous vehicle drive function or a function for requesting vehicle control by the control unit. Thus, advantageously, a vehicle occupant such as a vehicle driver, is relieved of vehicle control when the occupant exhibits either too high and/or too low cognitive load levels. Consequently, vehicle drivers that are not fit to control the vehicle in some driving contexts or scenarios are not permitted to do so, or are suggested to hand over control, improving vehicle safety for both the vehicle occupants as well as for other road users.

However, as mentioned above, the vehicle control function may also for example comprise a function for re-configuring a steering system of the vehicle to be stiffer compared to a default steering system reconfiguration. This is not a take-over by the vehicle of the overall drive function, but just a re- configuration to achieve a higher or lower level of cognitive load by the vehicle occupants. Increasing steering wheel stiffness, for instance, is likely to lead to an increased level of driver cognitive load. Thus, is a too low cognitive load is estimated for a given driver, the system may trigger a stiffening of the steering wheel function, which is likely to lead to increase cognitive load, placing the driver closer to the peak of the Yerkes-Dodson curve discussed above.

Also, according to another example the vehicle control function comprises a vehicle control function configured to reduce vehicle speed or to reduce a vehicle acceleration performance. This aspect is mainly related to safety, i.e., reducing speed if the driver is estimated to have a cognitive load which places the driver far from optimum on the Yerkes-Dodson curve.

Figures 1 and 3 also shows a remote server 140. The sensor data, and/or the estimated present cognitive loads discussed above may be uploaded to the remote server via a wireless communication link. The server consequently obtains cognitive load data from the vehicle. The cognitive load data can be associated with a vehicle position. This way, the remote server 140 obtains a spatial distribution of cognitive load data.

In other words, according to some aspects, the vehicle control function comprises uploading sensor data, and/or an estimated present level of cognitive load to the remote server 140.

The spatial distribution of cognitive load data can be used to, e.g., predict a future level of vehicle occupant cognitive load based on previously estimated cognitive loads. For instance, suppose a certain driver normally experiences an increased level of cognitive load at a given location. The system may then predict a future level of cognitive load based on the data uploaded to the server and on a present location of the vehicle, based on that the vehicle is approaching an area where certain cognitive load levels have been estimated in the past. The data uploaded to the remote server 140 may also be associated with a time stamp. Then, if the vehicle approaches a location where the driver normally experiences a high level of cognitive load in a certain time interval (like rush hour) , the vehicle may request control or perform some other support function. This function is of course also applicable when a future driver cognitive load is predicted to be too low (such as when driving at night) .

Thus, according to aspects, the method comprises estimating S41 a future occupant cognitive load based on previous data obtained as sensor data or cognitive load data from the vehicle and/or from a remote server 140.

The server 140 may also collect sensor data and cognitive load data from other vehicles. In case the uploaded cognitive load data is associated with respective positions, a cognitive load map can be created by the server. This map then shows cognitive loads estimated for vehicle occupants at different locations in a traffic infrastructure. The cognitive load map may show, e.g., average cognitive load on some scale, or some other statistic of cognitive load.

Figure 6 illustrates a scenario 600 where a plurality of vehicles 100 communicate cognitive load data Cl, C2, C3, C4 with a remote server 140. The communication also comprises position data PI, P2 , P3 , and P4.

It is appreciated that the term 'map' is herein taken to mean an association or mapping between cognitive load data and position data. Thus, a cognitive load map is interpreted different from a mere presentation of cognitive load data.

The cognitive load map may also be associated with a time of day, such that estimated cognitive load levels are associated with both a time of day and a location. The optional time stamp data is shown in Figure 6 as Tl, T2, T3, and T4. The cognitive load map can be seen as a type of 'weather map' , showing different areas associated with certain types of driver experiences over time, similar to areas with rain and areas with sunshine over time.

Figure 7 shows an example cognitive load map 700. The map shows three areas 710, 720, 730. Area 710 is associated with a low cognitive load, as reported by vehicles travelling through area 710. Areas 720 and 730 are associated with high levels of cognitive load, as reported by vehicles travelling through these areas presently or previously. As noted above, the cognitive load map 700 may also be associated with a certain time interval. For instance, a rush hour time interval cognitive load map is likely different from a nigh-time cognitive load map. Perhaps the rush-hour cognitive load map shows high driver cognitive loads around a city center, while the night-time cognitive load map shows increased levels of cognitive load at a non-lit street.

A plurality of functions may be implemented based on this cognitive load map;

A first cognitive load map function allows a vehicle occupant to download map data associated with vehicle occupant cognitive load. The vehicle occupant is thus informed about areas associated with high or low levels of cognitive loads. This allows the vehicle occupant to, e.g., select a route associated with a reduced level of cognitive load in case he or she is tired, and to select a route associated with an increased level of cognitive load in case he or she is more rested.

A second cognitive load map function allows a meeting scheduler function to use the cognitive load map data to suggest moving, e.g., a scheduled phone meeting in case the vehicle navigation system shows that a vehicle route will pass an area of high cognitive load during the meeting time. The scheduler may also propose to cancel the meeting in case the cognitive load conditions are not favorable. For example, suppose a vehicle driver intends to have a phone meeting while driving home from work. In case there is an accident or other event resulting in increased levels of cognitive loads uploaded by vehicles passing through some area, the meeting scheduler function may suggest that the meeting is not held while driving through the area associated with increased levels of cognitive load.

A third cognitive load map function allows a navigation system to select a vehicle route associated with a certain level of cognitive load. A user can then select a navigation option which navigates along a route associated with a reduced level of cognitive load compared to other routes. This navigation choice may be selected, e.g., if a driver is already tired and not up for a high cognitive load drive. The example map 700 shows two alternative routes between location A and location B. One route 740a is somewhat shorter but passes through an area associated with an increased level of cognitive load. An alternative route 740b between A and B is associated with a more normal level of cognitive load, since it passes between areas 720 and 730 associated with increased levels of cognitive load.

It is appreciated that the functions related to the cognitive load map can be implemented independently of the other features of cognitive load estimation and vehicle control functions discussed herein. Consequently, there is disclosed herein a method for processing vehicle occupant cognitive load data, the method comprising obtaining sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant (230), estimating a present level of occupant cognitive load based on the data, and uploading the sensor data and/or the estimated present level of occupant cognitive load to a remote server 140. There is also disclosed herein a remote server 140 arranged to receive cognitive load data from one or more vehicles, and to process the cognitive load data as herein discussed.

According to aspects, the remote server is arranged to provide the first, second and third cognitive load map functions discussed above.

It is appreciated that the server 140 may also be implemented in a vehicle, or may be a distributed server, such as a block chain technology data server.

The server 140 may also be used to collect vehicle occupant data comprising monitored reactions, such as surprise, fright, drowsiness, and the like. This sensor data can be uploaded to the server 140 together with location data. The server 140 may then correlate the sensor data against position, to detect location where a plurality of vehicle occupants shows similar reactions. The server 140 may then issue information such as a warning to vehicles approaching the given location or area. For instance, support vehicle drivers tend to show surprise or fright at a given location, such as an intersection or at some location along a countryside road. A warning or information signal may then be issued to other vehicles approaching the location. This way the vehicle occupants receives an early warning, informing them to be on the look-out for the event which caused the surprise or fright reaction.

Figure 4 schematically illustrates, in terms of a number of functional units, the components of a control unit 120 according to an embodiment of the discussions herein. The control unit may be implemented as part of a vehicle electronic control unit (ECU) .

Processing circuitry 410 is provided using any combination of one or more of a suitable central processing unit CPU, multiprocessor, microcontroller, digital signal processor DSP, etc., capable of executing software instructions stored in a computer program product, e.g. in the form of a storage medium 420. The processing circuitry 410 may further be provided as at least one application specific integrated circuit ASIC, or field programmable gate array FPGA. The processing circuitry thus comprises a plurality of digital logic components.

Particularly, the processing circuitry 410 is configured to cause the control unit 120 to perform a set of operations, or steps. For example, the storage medium 420 may store the set of operations, and the processing circuitry 410 may be configured to retrieve the set of operations from the storage medium 420 to cause the control unit 120 to perform the set of operations. The set of operations may be provided as a set of executable instructions .

The processing circuitry 410 comprises several modules configured to perform different parts of the methods discussed in connection to Figure 3 above.

Figure 4 thus schematically shows a vehicle control unit 120 arranged to trigger an onset of a vehicle control function. The control unit comprises

an obtaining module Sxl configured to obtain sensor data from a plurality of sources related to a cognitive load of at least one vehicle occupant 230, an estimating module Sx4 configured to estimate a present level of occupant cognitive load based on the data, a comparing module Sx5 configured to compare the present level of occupant cognitive load to pre-determined trigger criteria; and an initiating module Sx6 configured to, if the present level of occupant cognitive load meets the pre-determined trigger criteria, initiate a vehicle control function by the vehicle control unit. According to some aspects, the control unit 120 also comprises a second obtaining module Sx2 configured to obtain vehicle position data related to a vehicle environment context, wherein the pre-determined trigger criteria are at least partly based on the vehicle environment context.

According to some other aspects, the control unit 120 comprises a third obtaining module Sx3 configured to obtain data related to a vehicle driving scenario context, wherein the pre determined trigger criteria are at least partly based on the vehicle driving scenario context.

The storage medium 420 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

The control unit 120 further comprises an interface 430 for communications with at least one external device, such as the external and internal vehicle sensors 110, 210 discussed above. As such, the interface 430 may comprise one or more transmitters and receivers, comprising analogue and digital components and a suitable number of ports for wireline communication.

The processing circuitry 410 controls the general operation of the control unit, e.g. by sending data and control signals to the interface 430 and the storage medium 420, by receiving data and reports from the interface 430, and by retrieving data and instructions from the storage medium 420. Other components, as well as the related functionality, of the control node are omitted in order not to obscure the concepts presented herein.

Figure 5 shows a computer program product 500 comprising computer executable instructions 510 to execute any of the methods disclosed herein.