ACOUSTIC DETECTION SYSTEM FOR A VEHICLE AND METHOD FOR ITS OPERATION

The present invention relates to an acoustic detection system for a vehicle, to a method of acoustic detection of an event for a vehicle, and in particular to an environmental event and vehicle state detection of an autonomous vehicle using acoustic sensors.

Vehicles, and in particular autonomous vehicles, comprising radar, lidar, camera, and ultrasonic devices as sensors for recognizing aspects of an environment of the vehicle are well-known in the state of the art. Sensors which detect electromagnetic waves may operate in a passive way by merely sensing electromagnetic waves originating from other sources, or in an active way by emitting a signal and sensing its reflection.

Ultrasonic sensors are employed in the state of the art for range detection, operating by actively emitting ultrasound waves and sensing their reflection in order to determine a distance to a reflecting surface.

At least some of these sensors are employed to recognize particular events in the environment of the vehicle. Such an event may for example consist in a collision of the vehicle with another object. A system for detecting a collision may e.g. be found in WO201 8144041 A1 , which discloses, for an autonomous vehicle, a control system, tasked with analyzing the sensor data for a collision and generating a claim directed at an insurance provider. A further system, detecting not the collision but rather a risk of it, may be found in CN103569112A, with the system estimating a collision threat on a basis of matched data from a sensor and a camera. Another system disclosing a method to identify a collision threat may be found in US2012306663A1 .

Due to the possibility of sensor errors, autonomous vehicles make a high redundancy of sensors for each individual task necessary, or at least desirable. Furthermore, if a system exploits preexisting sensors originally introduced on an autonomous vehicle for a different purpose, the particular design or blind spots of these sensors may lead to an incomplete performance. Both in order to achieve a redundancy as well as for an execution of new tasks, sensors operating by mechanisms complementary to those mentioned above are of advantageous benefit. This can be achieved by mechanisms aimed at a passive detection of audible acoustic signals. Therefore there is a demand for a system configured to use passive acoustic sensors on a vehicle in order to detect, in an environment of the vehicle, specific events or specific states of the vehicle.

A contribution towards meeting this demand is provided by an acoustic detection system according to claim 1 , a vehicle, in particular an autonomous vehicle, of claim 10, a method of acoustic detection of an event of claim 11 , and a computer product of claim 12. The dependent claims refer to further advantageous realizations of the subject matter of the independent claims.

The present invention relates to an acoustic detection system for a vehicle. The vehicle comprises one or more acoustic sensors, configured to sense perceptible acoustic waves in an environment of the vehicle, and to produce sensor data thereof. Sensor data from the acoustic sensors will be referred to as acoustic sensor data. The system is characterized by a reception module, configured to receive the acoustic sensor data, and by a detection module, configured to detect, in the acoustic sensor data, an event on or around the vehicle.

The vehicle may in particular be an autonomous vehicle with or without a driver. More specifically, the vehicle may be an autonomous commercial vehicle. If a driver is present on the vehicle, advantageously the acoustic sensors are not specifically adapted to sense utterances of the driver, but rather to detect events outside and/or in a cargo area of the vehicle.

The reception module is configured to collect the acoustic sensor data from the acoustic sensors, and to provide the acoustic sensor data to the detection module. If the vehicle is an autonomous vehicle, the reception module may be further configured to receive information from components of an automatic driving system.

The detection module is adapted to detect events, or respectively information contained in acoustic emissions thereof, on or around the vehicle. This detection may involve recognizing patterns in the acoustic sensor data, differentiating or isolating these patterns, and classifying them according to the corresponding events. Detecting an event in the acoustic sensor data may be based on an analysis of an amplitude sequence, a spectrum, and/or a cepstrum of the acoustic sensor data, and may be achieved by means of a trained neural network. Conventional signal processing techniques may also be employed, either alternatively or additionally to a neural network algorithm, in order to achieve the detection. By the analysis of the acoustic sensor data, various environmental events which are hard to detect by other sensors can be identified or verified, such as, for example, a collision with another vehicle, a collision with a human, or a detection of gusts of wind hitting the vehicle. The analysis of the acoustic sensor data can provide significant information regarding the state of the vehicle, and identify events around the vehicle (like collisions of the vehicle) and/or events on the vehicle (like a flapping cover of cargo, a humming of a defect wheel bearing, or an exploding tire). If the detection module has detected an event on or around the vehicle, it may be further configured to send information about the event to another component of the vehicle or to a driver. In particular, if the vehicle is an autonomous vehicle, information obtained by the detection module may be employed as input for a behavior and path planner of an autonomous driving system. The detection module may also be configured to broadcast the information about the event to a receiver remote from the vehicle, e.g. by employing a system for wireless communication on the vehicle. Depending on the embodiment, the information may comprise a warning issued upon detection of the event, merely the fact that the recognized event has occurred, a time and a relative position where and when the event occurred, and/or more data about the recognized event.

Depending on the embodiment, the reception module and the detection module may be realized separately, or within one electronic control unit. They may also be partially or fully integrated into each other, or into other components of the vehicle.

Optionally the vehicle comprises at least one further sensor and/or camera for producing further sensor data indicative of the event, and the reception module is further configured to receive the further sensor data and to include the further sensor data in detecting or in confirming the event. The inclusion of the further sensor data may e.g. be implemented by sensor data fusion. The inclusion may in particular comprise a temporal and/or spatial correlation of the acoustic sensor data and the further sensor data. Including the data of further sensors on the vehicle is in particular advantageous for corroborating or enhancing information about the event obtained from other components and/or sensors of the vehicle, and/or for an identification of a cause, direction, intensity, or other modality of the event. The acoustic detection of the event may furthermore complement the further sensor data. The further sensor may e.g. be a camera adapted to supply an autonomous driving system with an image or video from a certain part of the surroundings of the vehicle, but with a blind spot immediately adjacent to the vehicle. If this is the case, the camera can be used to detect an object that has a risk to collide with the vehicle, but the camera may not be able to actually detect the collision itself. This gap may be closed by the acoustic detection system.

Optionally the at least one further sensor and/or camera is configured to sense one or more of the following: an acceleration, a torque, a yaw moment, a wheel speed, an inclination, a collision, a wind, a loading or an unloading, an opening of a loading area, an image of surroundings of the vehicle.

The acceleration may e.g. be a downward acceleration of a cargo area (as e.g. on a trailer of a commercial vehicle), corresponding to a weight deposited in the cargo area. The detection module may be configured to correlate the downward acceleration with a detected sound generated by goods dropping on a floor of the trailer, or generated by a person entering the cargo area. In the latter case, the detection module may be configured to correlate this event with further information about a state of the cargo area, and e.g. issue a warning of a potential burglary if the cargo area is in a locked state.

The acceleration may also be in a longitudinal or lateral direction of the vehicle, and the detection module may be adapted to correlate the acceleration with a sound of a collision with an object or a human, or with a gust of wind hitting the vehicle. The sound of the collision may also be correlated with a torque or a yaw moment of the vehicle. The inclination of the vehicle may be correlated with a noise at the vehicle, e.g. in order to identify an impact of wind on the vehicle.

Especially on an autonomous vehicle there may exist further sensors which are adapted to detect a collision of the vehicle, as e.g. a camera and/or an acceleration sensor. The detection module may be configured to support the collision detection by correlating this data with an event detected in the acoustic sensor data.

Optionally, the reception module is further configured to retrieve, e.g. from an optional data storage of the detection system or elsewhere in the vehicle or via a means for communication of the vehicle from a remote sender, acoustic characteristics indicative of the event; and the detection module is further configured to detect the event by comparing the acoustic sensor data with the retrieved acoustic characteristics.

Here, the term acoustic characteristics refers to data which allows to identify particular events by characteristic properties e.g. of a spectrum, cepstrum or other frequency data, or of an amplitude sequence. The event may not be identified individually, but rather classified, i.e. identified as belonging to a particular class of events. The detection or identification of the event may be achieved by pattern matching. It may result in a likelihood or probability, with a value depending on the outcome of the pattern matching process. Instead of the reception module to retrieve the acoustic characteristics, the detection module may be configured to store a library of acoustic characteristics.

Optionally, the detection module is further configured to issue a warning to a remote server and/or to a person in the vehicle.

Optionally, the detection module is in particular configured to detect, as the event, one or more of the following: a collision of the vehicle with a human, a collision of the vehicle with an animal, a collision of the vehicle with another vehicle, a collision of the vehicle with another object, and/or another vehicle backing up (reversing) towards the vehicle.

Collisions may be detected by particular features in the acoustic sensor data, such as amplitude and/or frequency peaks, particular forms of time sequences or spectra, and/or by identifying particular distinctive sounds like a shattering glass, an opening airbag, or a crushing car body part. In a detection of a collision, the detection module may also be configured to detect a signal of a horn in the acoustic sensor data. Commercial vehicles or buses which are reversing can generate a special kind of sound, which the detection module may be configured to detect. If the vehicle is autonomous, this information may in particular be further utilized in a behavior planning or a trajectory planning.

Optionally the detection module is configured to detect, as the event, a side wind effect on the vehicle. Side wind is particularly relevant for a commercial vehicle, and especially for a truck-trailer vehicle combination. High-speed driving during strong side wind can require a significant counter-steering action from a driver or a lane-keeping autonomous driving task in order to keep the vehicle in the middle of the lane. A compensation of side wind effects requires a reliable detection. For sensors like e.g. acceleration sensors, typically present on the vehicle and employed for this task in known systems, side wind can be hard to detect and to distinguish from other causes, like e.g. a lateral slope of the road. Acoustic sensors present a powerful tool for detecting noise generated by the side wind. The detection module may be configured to detect gusts of wind, or a continuous wind situation. The detection module may be configured to apply an algorithm to subtract from the acoustic sensor data an effect of a headwind expected from a velocity of the vehicle. Information about wind or side wind effects may be corroborated by, or used in a corroboration of, a side wind detection from another system present on the vehicle.

Optionally, the detection module is further configured to estimate, from the side wind effect, a speed of the side wind.

Optionally the vehicle comprises a cargo area for transportation of cargo, at least one acoustic sensor of the one or more acoustic sensors is configured to sense audible acoustic waves in the cargo area, and the detection module is further configured to detect one or more events corresponding to the following: a cargo coverage failure, like e.g. a flapping of a covering, a cargo disintegration, a movement of cargo, a burglary (forceful intrusion) of the cargo area, a change in the cargo area that generates noise.

Embodiments further relate to a vehicle, in particular an autonomous vehicle, comprising one or more acoustic sensors configured to sense acoustic waves in an environment of the vehicle and to produce acoustic sensor data thereof, characterized by the acoustic detection system of any of the previous claims.

The present invention further relates to a method of acoustic detection of an event for a vehicle, the vehicle comprising one or more acoustic sensors configured to sense acoustic waves in an environment of the vehicle and to produce acoustic sensor data. The method is characterized by receiving the acoustic sensor data; and detecting, in the acoustic sensor data, the event on or around the vehicle.

This method may also be implemented in software or a computer program product. Embodiments of the present invention can, in particular, implemented by software or a software module in an electronic control unit. Therefore, embodiments relate also to a computer program having a program code for performing the method, when the computer program is executed on a processor.

Some examples of the systems and/or methods will be described in the following by way of examples only, and with respect to the accompanying figures, in which:

Fig. 1 depicts a schema of an acoustic detection system for a vehicle according to an embodiment of the present invention;

Fig. 2 illustrates a collision detection by means of the acoustic detection system;

Fig. 3 illustrates a detection of a reversing vehicle in a surrounding of the vehicle by the acoustic detection system;

Fig. 4 illustrates a side wind detection by means of the acoustic detection system;

Fig. 5 depicts a method of acoustic detection of an event for a vehicle according to the present invention. Fig. 1 shows a schematic representation of an acoustic detection system 100 for a vehicle 200 according to an embodiment of the present invention. The vehicle comprises one or more acoustic sensors 210, configured to sense perceptible acoustic waves 10 in an environment of the vehicle and to produce sensor data (i.e. , acoustic sensor data) thereof. The acoustic system 100 is characterized by a reception module 110, configured to receive the acoustic sensor data, and by a detection module 120, configured to detect, in the acoustic sensor data, an event on or around the vehicle 200. The vehicle 200 may in particular be an autonomous vehicle, and especially an autonomous commercial vehicle. The reception module 110 may be configured to receive data of further sensors on the vehicle, e.g. of an acceleration sensor measuring an acceleration of the vehicle 200, or of a camera producing an image of surroundings of the vehicle 200. If the vehicle 200 is an autonomous vehicle with an autonomous driving system, the reception module 110 and/or the detection module 120 may be further configured to send and receive data from other components of the autonomous driving system, in particular from a planning unit or from a means for communication.

The detection module 120 may e.g. be configured to detect, in the acoustic sensor data, a collision with another road user or some other object and/or a sound of a gust of wind or of continuous wind affecting the vehicle (with an appropriate subtraction for a headwind).. If the vehicle 200 is a commercial vehicle comprising a cargo area, the detection module 120 may be configured to detect events in the cargo area, like e.g. a flapping of a covering, falling cargo, an opening of a door and/or a loading/unloading of cargo into/from the cargo area, or an intrusion into the cargo area. The detection module 120 is configured to discriminate between different effects. The detection module 120 may detect the event by means of a machine learning algorithm, or by comparing the data with acoustic characteristics for an amplitude time series, or a frequency spectrum and/or cepstrum. The detection system 120 may be configured to store a library of acoustic characteristics, and/or the reception module may 110 be configured to retrieve these characteristics from a remote station, e.g. via a means for wireless communication of the vehicle 200. Fig. 2 illustrates a collision detection by means of the acoustic detection system 100 on a commercial vehicle 200, especially an autonomous commercial vehicle. The commercial vehicle comprises an acoustic sensor 212 (whose position may be different than depicted), configured to detect sound in the surroundings of the vehicle 200. The upper part of the figure represents a situation at a time t1 , the lower part of the figure represents the situation at a later time t2. In the upper part of the figure, the commercial vehicle 200 drives on a road 20, with other vehicles 32, 33 in its vicinity. An arrow on the vehicle 32 driving in front of the commercial vehicle 200 indicates an acceleration of the vehicle 32 due to a braking of the vehicle 32. The commercial vehicle 200 does not have time to slow down sufficiently in order to avoid collision, and in the lower part of the figure, the commercial vehicle 200 collides with the vehicle 32. The collision produces a sound 10 which is captured by the acoustic sensor 212. The detection module 120 of the acoustic detection system 100 (not depicted) is configured to, after receiving the acoustic sensor data, detect the collision of the commercial vehicle 200 with the vehicle 32. The detection module 120 may be further configured to issue a warning or a report to a driver of the commercial vehicle 200, or send such a report to a remote station by means of a communication unit of the commercial vehicle 200, e.g. by wireless transmission. Furthermore, the reception module 110 is configured to receive, besides acoustic sensor data from the acoustic sensor 212, data from an acceleration sensor of the commercial vehicle 200, which measures a backward acceleration of the commercial vehicle 200 at the time of the collision. Rapid (unintended) change in an acceleration vector recognized by the acceleration sensor and a specific noise corresponding to the collision of the autonomous vehicle 200 with the other vehicle 32 can be correlated in the detection module 120. The detection module 120 may e.g. be configured to detect the collision, or further information about the collision, by correlating a time of the peak of the backward acceleration with a time, e.g. of a peak of an amplitude, in the acoustic sensor data. The detection means 120 may be configured to detect further information about the collision, such as a direction of the collision relative to the commercial vehicle 200 and/or an intensity of the collision, from including magnitude and direction of the acceleration in the detection algorithm.

Acoustic sensors, like the exemplary acoustic sensor 212 shown in the figure, can be placed at different positions on the commercial vehicle 200. In particular, one or more acoustic sensors may be installed on a trailer of the commercial vehicle 200 (e.g. at the rear end of the trailer). The detection module 120 may be configured to detect a collision by a specific noise, as e.g. a typical sound in case of docking at a yard. The detection module 120 may be further configured to receive, via the reception module 110, an information from another component of the commercial vehicle 200 (as e.g. an autonomous driving system) that the commercial vehicle 200 is reversing (backing up), and utilize this information for the detection of a collision of the commercial vehicle 200 while backing up against a docking yard.

Fig. 3 illustrates a detection of a reversing commercial vehicle 35 in a surrounding of the commercial vehicle 200 with the acoustic detection system 100 (not depicted) by the acoustic detection system 100. The commercial vehicle 200, here at standstill, comprises a sensor 214 located at the rear part of a trailer of the commercial vehicle 200. The reversing commercial vehicle 35 moves in a general direction of the commercial vehicle 200. While doing so, the reversing commercial vehicle 35 emits a sound 12, warning bystanders about its action. The acoustic sensor 214 is configured to pick up the sound 12, and the detection module 120 is configured to detect, as the event, the backing up of the reversing commercial vehicle 35. The detection module 120 may be configured to identify, based on the acoustic data of the sound 12, the reversing other commercial vehicle 35, as well as determine a direction and distance of the reversing commercial vehicle 35 relative to the commercial vehicle 200.

Fig. 4 illustrates detection of a side wind 60 by means of the acoustic detection system 100 (not depicted) on the commercial vehicle 200. The commercial vehicle 200, which may in particular be an autonomous vehicle, comprises an acoustic sensor 216 (whose position may be different than depicted), configured to detect a sound of the side wind 60, and/or particularly of gusts of the side wind 60. Without compensation, the side wind 60 can have an effect of disturbing the commercial vehicle 200, such that the commercial vehicle 200 would follow the trajectory 70, and thus get off the lane or road 20, unless counter-steering 230 is applied in order to keep the commercial vehicle 200 on the road 20. The commercial vehicle 200 may in particular be equipped with another device, or an autonomous driving task, to identify and compensate the effect of the side wind 60 on the commercial vehicle 200, e.g. by means of a lateral acceleration. The effect of the side wind 60, however, can be hard to identify unambiguously, as acceleration may also occur due to bending or a slope of the road 20. The detection of the side wind 60 by means of the acoustic sensor 216 and the acoustic detection system 100 (not depicted) may serve as an additional input for the device or autonomous driving task to identify and compensate the effect of the side wind 60 on the commercial vehicle 200.

Fig. 5 shows steps of a method of acoustic detection of an event for a vehicle 200 according to the present invention. The vehicle 200 comprises one or more acoustic sensors 210, 212, 214, 216, configured to sense acoustic waves in an environment of the vehicle 200, and to produce acoustic sensor data thereof. One step of the method comprises receiving S110 the acoustic sensor data. A further step of the method comprises detecting S120, in the acoustic sensor data, the event on or around the vehicle 200.

This method may also be a computer-implemented method. A person of skill in the art will readily recognize that steps of the above-described method may be performed by programmed computers. Embodiments are also intended to cover program storage devices, e.g., digital data storage media, which are machine or computer readable and encode machine-executable or computer-executable programs of instructions, wherein the instructions perform some or all of the acts of the above-described method, when executed on the computer or processor.

The description and drawings merely illustrate the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

Furthermore, while each embodiment may stand on its own as a separate example, it is to be noted that in other embodiments the defined features can be combined differently, i.e. a particular feature described in one embodiment may also be realized in other embodiments. Such combinations are covered by the disclosure herein, unless it is stated that a specific combination is not intended. In particular, while reference signs for acoustic sensors can differ in the above description, they may refer to the same acoustic sensor. On the other hand, like labels of acoustic sensors in different figures do not necessarily imply the same acoustic sensor. While each figure displays only one acoustic sensor, it is understood that the vehicle 200 may contain several acoustic sensors for the same task, at various positions on the vehicle. In all figures, the symbol for an acoustic sensor is symbolic, and positions of acoustic sensors are only schematic, to the effect that they may be different from those depicted.

LIST OF REFERENCE SIGNS

10, 12 acoustic waves

20 road 22 other driving surface

32, 33, 35 other vehicles

60 side wind

70 trajectory

100 acoustic detection system 110 reception module

120 detection module

200 vehicle

210, 212, 214, 216 acoustic sensors

230 counter-steering t1 , t2 points in time

S110, S120 steps of a method