CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is related to commonly-assigned US Patent Application No. 63/296,448 filed January 4, 2022, incorporated herein by reference.

FIELD

[0002] This technology relates to vehicle sensing and control systems, and more particularly to active door actuation for automotive and other vehicle applications. The technology herein has particular advantageous applicability to systems and methods for tracking a driver of a vehicle in order to automatically open a door for the driver.

BACKGROUND & SUMMARY

[0003] Modern vehicles are often designed with a keyless ignition system. The keyless ignition system includes a fob that the driver can keep tucked away in their pocket, purse, briefcase, or anywhere in proximity to the wireless receiver inside the vehicle. The fob transmits a unique wireless signal to the car's computer system, which validates that the correct signal has been sent and allows the driver to push a button on the dashboard or console to start the engine. Shutting off the motor is just as hassle-free: Simply press the start/stop button. In addition to keyless ignition, most systems also include keyless entry, allowing the driver to enter the car without inserting a key or hitting a button on the fob.

[0004] Among other advantages, these systems are designed to help deter break-ins and vehicle theft. Since the car's computer will only recognize a signal from its own fob, thieves will have a harder time getting in and starting the engine. There's no metal key that can be copied and no mechanical steering-column lock that can be forced or otherwise defeated. See e.g., Gareffa, “What You Need to Know About Keyless Ignition Systems”, Edmunds (2018).

[0005] Another of the automated functions in demand on modern vehicles is what is sometimes called a “butler function” of automatically opening and closing the doors of a vehicle for driver and passenger ingress/egress, placing a baby in the car seat in the back seat of the vehicle, depositing groceries in the back seat, or the like. Active Door Activation (ADA) for automotive applications are likewise often either cued by someone approaching from the outside or by a user in the car. The most common type of automotive ADA is opening the rear hatch of a minivan or SUV. However, of particular interest for many automotive manufacturers is providing ADA for at least the front doors - the driver side and a passenger side. [0006] Much work has been done on ADA and associated sensing and related technology in the past. See for example:

[0007] US20200081 127 describes a TOF sensor placed such that a first field of view of the TOF sensor perceives a region surrounding a passenger door, in order to detect a blockage of the passenger door.

[0008] US20200087973 describes a door moving device comprising a force providing device for providing a force for controlling a movement of the door, a drive device that is configured to generate a drive torque for powering the force providing device, and an electrically actuated brake device that is configured to generate a braking torque for braking the force providing device when not supplied with electricity.

[0009] USP 10,964, 133 describes a position sensing component configured to provide position data in response to detecting the presence or position of vehicle structure relative to the sensor.

[0010] US 2017/0306684 is based on single-pixel IR TOF sensors with ranges of typically 1 meter. This publication cites the Osram SFH 4550 as an example TOF sensor. It mentions integrating a visual camera for “additional information,” but does not disclose what this information is or how it supports an overall system approach. Furthermore, it requires distinct and separate short and long range sensor modes, which may slow down processing.

[0011] Schmitz, S . and Roser, C., "A New State-of-the-Art Keyless Entry System," SAE Technical Paper 980381 , 1998, h t t s : // d o i . o r g describes a keyless entry system that uses a fob so that the vehicle door or trunk opens on successful verification as if there were no locks. [0012] Further improvements are possible and desirable. In particular, in addition to improving accuracy and reliability, it would be beneficial to provide a system that detects that the driver of a vehicle is approaching, recognizes the driver and automatically opens the door of the vehicle in response to the recognition.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example ADA system.

[0014] Figure 2 is a flowchart of example functionality of the Figure 1 system.

[0015] Figure 3 shows an example ADA Perception process.

[0016] Figure 4 is a more detailed block diagram of an example ADA system.

[0017] Figure 5 shows a more detailed flowchart for the comfort entry detection.

[0018] Figures 6A, 6B show example driver approaches.

DETAILED DESCRIPTION OF NON-LIMITING EMBODIMENTS

[0019] The example technology herein provides a vehicle sensor package and placement for advantageous use in a variety of different contexts and use cases including but not limited to active door activation (ADA) in response to recognition that the driver of the vehicle has approached the vehicle.

[0020] Example ADA System With Driver Recognition

[0021] Figure 1 is a block diagram of an example non-limiting ADA system 100 including driver recognition. In the embodiment shown, system 100 may be an overall onboard vehicle control and monitoring system that includes an active door actuator 102, which in turn includes a door actuation system 104 and an environmental sensor system 150. The door actuation system 104 includes one or a plurality of door locks/actuators 106 and a door actuator control unit 108. The environmental sensing system 150 includes one or a plurality of sensors 152 and a sensor control unit 154.

[0022] In example embodiments, the sensors 152 sense the environment surrounding a vehicle and in response, the environmental sensing unit 150 permits or controls the door actuator control unit 108 to actively open and close the vehicle doors or permits or control how far the door is opened to avoid striking a detected object. For example, in one embodiment, the sensors 152 can detect the driver is approaching and also provide precise position information for objects such as curbs so the system 100 can determine whether opening a door will cause the door to strike the curb and if not, may open the door automatically for the driver. The door actuators 106 can use electromechanical linear actuators, hydraulic brakes, electromechanical locking/unlocking mechanisms, and other devices to automatically open and close the vehicle doors as (and to the extent) authorized or cleared by the environmental sensing unit 150. The vehicle doors can provide ingress support by holding an intermediate position between fully opened and fully closed to allow the driver or passenger to use a partially-opened door as an assist to get in or out of the vehicle.

[0023] Figure 2 shows a high level operating schematic diagram of the Figure 1 system. In this example, inputs 52 are provided including door status/angle; keyfob data; and other sensor data. System 100 performs object detection 54 by denoising point cloud data; removing points belonging to the ground; and dividing the remaining elevated points into objects using a segmentation process. System 100 further performs object tracking 56 by determining whether an object is static or dynamic, and for dynamic objects, estimating the object motion trajectory. If an object is static (“yes” exit to decision block 58), system 100 performs door clearance estimation 60 to determine if there is clearance to open and close the door. If an object is dynamic (“No” exit to decision block 58), system 100 provides road user protection 62 by determining if there is a cyclist or vehicle that will likely be impeded by motion of the door. System 100 further includes a driver comfort entry 64 that tracks the driver of the car approaching the car for hands-free operation of the door.

[0024] Figure 3 is a further block diagram showing ADA perception of system 100. In the example shown, the sensor package 152 provide digital sensor outputs such as position (xyz) information in three dimensional space, amplitude, distance, ambient information and in some embodiments, point clouds of detected objects. Also inputted: door angle information for each of plural doors, and key fob data including quadrant and range (distance). Key fob data may include a unique or semi-unique value, string, digital signature or other information that is authenticated as matching the vehicle. The key fob quadrant and range may be detected using conventional wireless technology including a radio receiver providing an RSSI indicator to indicate range and by controlling a beam-forming patch antenna to estimate the direction of the key fob with respect to a wireless radio receiving antenna within the vehicle. Authentication tokens other than a key fob may be used.

[0025] As Figure 3 further shows, system 100 performs static object detection 180 and dynamic object detection 182, and combines those outputs to provide door clearance estimation 190 used to recommend or control the system to open a particular door to a particular maximum door open angle. In the example shown, the dynamic object detection is also input to a dynamic object tracker 184 that tracks dynamic objects as the move toward or away from the vehicle. A door open check 186 is used if necessary to stop opening a door in motion in order to reduce the probability of a collision 188. In some embodiment, system 100 may make a decision to reverse the door opening process to attempt to close the door if a moving object is rapidly approaching the open or partially open vehicle door. The ADA perception block of Figure 3 can also or alternatively provide alerts to alert the driver of ego vehicle, the passenger of ego vehicle, an approaching bicyclist, pedestrian, runner or motorist, or other relevant personnel of impending collision. [0026] Figure 4 is a more detailed block diagram of a non-limiting ADA system 100. In this embodiment, sensors 152(0), 152(1 ), ... 152(n) produce digital messages at high speed. These digital messages provide precise information as to the position and size of detected objects in one embodiment. These messages are filtered by a message filter 156 and provided to an ADA Detector 158 that performs process such as shown in Figure 3. The ADA Detector 158 may comprise a high speed processor that executes instructions stored in non-volatile storage to detect objects, bicycles/pedestrians, and dynamic retroreflectors. A fusion processor or processing block 160 uses the independent outputs all detectors to produce a fused result indicating, with a high degree of reliability, whether a static or dynamic object is present and the position including distance of the dynamic object.

[0027] A door clearance detection processor 164 receives door dimension data 166 from a door transformation processor or engine 168 that transforms the door dimensions using for example angle encoder outputs or any other sensor that can detect current door open angle. For example, if a sensor 152 is mounted on a door which is currently being opened manually or automatically, the controller is configured to detect the position (current pose) of the sensor based on the detected current opening angle of the door. The controller may then transform the sensor position data into a different coordinate system and use it to detect whether a static or dynamic object is within an Area of Regard in respect of the sweep of the same or different door. Thus, in one embodiment, a moving sensor mounted on one door is used to detect and track the position of another moving object (e.g., a bicyclist or runner, another door of the vehicle, etc.) even though the vehicle on which the moving sensor is mounted is stationary as its door is opening.

[0028] The door clearance detection processor 164 uses the transformed data and the object detector output (which specifies a position of a detected object in the vehicle coordinate space) to detect whether a soon-to-be-swept area of an opening vehicle door is going to collide with a detected object. Meanwhile, a multi-object tracking processor 162 tracks the positions of objects reported by the object detection over time at a high sampling rate. Such tracking takes into account movement and/or position of the door if the sensor is mounted on the door.

[0029] System 100 can also track objects that were not in the Area of Regard but are moving into the Area of Regard (e.g., a bicyclist). A dynamic object collision predictor 172 predicts when a collision between the opening door and a tracked object is likely to occur and issues a stop control to the affected door via a Controller Area Network (CAN) bus such as defined by ISO 1 1898- 1 for data link layer and ISO 11898-2 for physical layer, or other vehicle control/communications network/bus 174. Such a stop control can stop the door from further opening and lock the door in place in a partially opened position to avoid a collision while still enabling ingress/egress (e.g., such as when fully opening the door will cause the door to strike a vehicle in an adjacent parking space). In addition or instead of positively controlling door movement, the system could provide an audible, visual and/or tactile/haptic warning to warn a driver or passenger that the door is going to collide with an object. As mentioned above, a comfort entry detector 170 may also command door opening based on tracked objects 162 (e.g., a driver approaching the vehicle) and door clearance detection 164.

[0030] The above-described control system can be implemented by any combination of hardware and software. For example, one example implementation may use one or more general purpose processors executing program instructions stored in non-transitory memory. Other implementations may use hardware to perform fast processing e.g., of sensor outputs. Any of the above functions may but need not be performed by artificial intelligence/machine learning using one or more neural networks.

[0031] Effective or Optimal Sensor Placement

[0032] The technology herein relates to among other things optimal, proper or effective placement of sensors 152 on the vehicle. Figures 5A-5D catalog such possible placements. Figure 5A shows mounting a sensor on one more of the pillars (A, B, C) that suspend the roof of the vehicle over the rest of the chassis. Figure 5B shows a sensor placed within the interior “door jamb” surface of a vehicle door. Figure 5C shows different mounting positions for a sensor within a door-mounted side view mirror nodule. Figure 5D shows another possible mounting position of a sensor within the mirror of a door-mounted side view mirror nodule. There are other possibilities such as on a rear bumper, in the front grill of the vehicle, etc.

[0033] Candidate sensor configurations are listed in the Table below with potential coverage areas:

[0034] In this context, the vertical roof support structure located between the front and rear doors on a typical vehicle corresponds with the A pillar located on either side of the windshield at the front of the car and the C pillar which sits on either side of the rear window on a typical sedan. The B pillar is a vertical structure between the front door and the rear door. On SUVs, minivans and wagons with extended cargo areas, the C Pillar is the vertical structure behind the rear door and the vertical supports at the rear of the vehicle are called D pillars.

[0035] A survey of available technologies finds a few that have sufficient resolution and sampling speed at a cost compatible with automotive applications. Since the area of regard is relatively short, the sensors most appropriate to cover it are considered “near-field sensors.” While the near-field is not strictly defined, it is generally thought to be sensors with a maximum range of 25 m or less, and in some cases, much less. Four different sensor modalities may be considered:

• Ultrasound - Emits high-frequency sound waves and determines distance to objects based on return signal time of flight.

• Short range radar - Generally lower power, lower resolution and lower cost than longer range radars designed for ADAS functions such as brake assist.

• Short range Lidar - Usually solid state Lidar based on MIMO technologies such as VCSEL emitters and SPAD detector arrays.

• Multi-pixel array based (2D) Time of Flight Camera. • Multi-pixel array based (2D0 optical or infrared camera with position determining capabilities.

[0036] The technology herein is not limited to any particular sensor or sensor design. Rather, any sensor using any technology such as including but not limited to the designs described above may be suitable. Different applications and use cases may dictate different sensor choices. For example, some current ultrasonic and short range radar sensors exhibit lower resolution, but multiple sensors used and “fused” together may provide acceptable performance. Alternatively, lower cost use cases may provide qualified results and warn/expect the driver or passenger to use their own observation powers to eliminate edge cases such as guy wires, especially fast moving objects and/or insufficient ambient light/visibility conditions. Thus, the technology herein can use any sensor and/or any combination of sensors as described above and/or any new or additional sensor(s) invented hereafter that may satisfy the general design criteria described above.

[0037] Example Comfort Entry Feature

[0038] A goal of the ADA driver tracking algorithm is to provide the comfort entry feature of Figure 4 block 170. In one embodiment, an entry demand is defined by the driver motion: the driver with the key fob walks towards one of the door and then stops at the door. See Figure 6A.

[0039] The ADA driver tracking algorithm detects the entry demand driver motion and issues a door opening signal. In one embodiment, assumptions may or may not include:

[0040] In one embodiment, only one person is in the sensor detection range and the system assumes that person has the key fob or other authentication token (the presence of the key fob or other authentication token is detected rather than assumed; the assumption is that the detected person has the detected key fob or authentication token). In one embodiment, the system uses contactless wireless communication to detect proximity of a fob that is matched to the vehicle.

[0041] In one embodiment, no classification task is performed yet, i.e., any dynamic objects are considered human. [0042] In one embodiment, the driver can only approach the door from a particular direction such as the back of the car (this constraint is relaxed if more sensors are used).

[0043] Inputs

[0044] detected object bounding boxes indicating position of a dynamic object that is assumed to be the driver when the system also detects that the key fob or other authentication token is at the same general distance and in the same quadrant or other general area as the dynamic object is detected.

[0045] Outputs:

[0046] the door with the entry demand: it can be driver-front, driver-rear, passenger-front, passenger-rear, or none

[0047] distance clearances for all doors

[0048] angle clearances for all doors.

[0049] An example ADA driver tracking algorithm takes detected object position and size (bounding boxes) as input. First, a multi-object tracking task (Figure 5 block 162) is performed to establish tracks of different dynamic (moving) objects. In one embodiment, each track includes the following information:

[0050] mean and covariance estimate of the position, velocity and acceleration of the object;

[0051] dynamic status: whether the tracked object ever moves. To estimate the probability that the object is dynamic, an error functions may be used:

[0053] If the computed dynamic probability is greater than a specified threshold, the dynamic status of the object is assigned as True.

[0054] After the moving object tracks are established, in one embodiment each of the tracks will be checked to see if the object track meets all of the following conditions (Figure 5 block 502):

[0055] Valid door condition: whether the tracked object is located within the designated area belonging to a door (502b). [0056] Observed condition: whether the track is detected (a observed object is associated with the track) (502c)

[0057] Dynamic condition: whether the track is a dynamic track (its dynamic probability is once greater than the threshold) (502f)

[0058] Stopped condition: whether the track’ s current dynamic probability is below the dynamic threshold (so it is static at the current time instant) (502e) [0059] Has_clearance condition: whether the door associated to the track has opening clearance (i.e. the door clearance is greater than a predefined clearance threshold). (502f).

[0060] If a track meets the five conditions mentioned above, it is a valid candidate driver track.

[0061] The last step is to check whether the valid candidate driver track is unique (Figure 5, decision block 504). If not, that means it is possible that there is another person(s) located in the door entry region with the actual driver. In this case, no door opening signal will be issued (block 508) (this e.g., may prevent a carjacking or other undesirable situation). If the valid candidate driver track is unique and the driver has not manually opened the door, the algorithm will output a door entry demand for the associated door (block 506).

[0062] In practice, to trigger the door unlocking and opening, the following conditions are met in one embodiment:

[0063] 1. driver carries the key fob or other authentication token such as a smart phone with an authentication app running on it.

[0064] 2. driver is recognized as a dynamic object

[0065] 3. driver stops in an area near the car defined as an “entry zone”

[0066] 4. the door is free to open without being blocked by other objects

[0067] 5. The driver previously exited the vehicle.

[0068] To perform a comfort entry gesture:

[0069] the driver walks toward the entry zone of the target door (see Figure 6A). During walking, the driver should be recognized by the sensor as a dynamic object.

[0070] the driver stays in the entry zone, then the targeted door should open. [0071] if the driver is not recognized as a dynamic object by the algorithm for some reason, a one-step operation may be performed to trigger the door opening: [0072] driver can move one step backwards (Figure 6B) and then return to the original position and stay. The algorithm will recognize the driver as a dynamic object and trigger the door opening.

[0073] Once the driver is inside the vehicle, the driver can in one embodiment depress the brake pedal. The system will detect this brake pedal depression as a signal to close the door, and so long as there are no obstructions in the area the door will sweep, issue a command for the door the close automatically.

[0074] In one embodiment, the one step can be perform to the left hand side or right hand side. Various visual and/or audio cues may be implemented to guide the driver to perform the comfort entry gesture. Turn around scenarios may be used to ensure the driver tracking algorithm operates properly when the driver changes her intention to enter the car. In one embodiment, if the system also detects a second large moving object approaching on the passenger side in addition to a first large moving object with a fob approaching on the driver’ s side, the system can be programmed to open both the driver’ s side door and the passenger side door to allow ingress of both the driver and a passenger.

[0075] In the above-described embodiments, authentication or recognition of the driver is performed by polling or detecting the presence of a fob, smart phone or other authentication device/token in the neighborhood of a detected large moving object. However, other contactless authentication or recognition techniques including but not limited to facial recognition, gesture recognition, passcode recognition, etc. could be used instead or in combination.

[0076] All patents, patent applications and publications cited herein are incorporated by reference for all purposes as if expressly set forth.

[0077] While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not to be limited to the disclosed embodiment, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.