TJÄRNSTRÖM, Fredrik (Stigbergsgatan 16, Linköping, S-58245, SE)
| Claims : 1. A driver assisting system (10) for a motor vehicle, comprising an imaging means (11) adapted to record images of a region surrounding the motor vehicle, an object detec¬ tion means (15) for detecting objects in the recorded im¬ ages by image processing, an object processing means (16, 17) for applying further processing to objects detected in the recorded images, and a driver assisting means (18) controlled depending on the result of said object detec¬ tion and object processing, characterized in that said driver assisting system (10) comprises a detection performance estimating means (20) adapted to provide an esti¬ mation of the detection performance of said object detec¬ tion means (15) using a result from said object processing means (16, 17 ) . 2. The driver assisting system as claimed in claim 1, wherein said detection performance is estimated using a result from a tracking means (17) in said object processing means (16, 17) . 3. The driver assisting system as claimed in claim 2, wherein said detection performance is estimated depending on the frequency of missed detections in a track of a detected object over time. 4. The driver assisting system as claimed in claim 2 or 3, wherein said detection performance is estimated depending on the distance at which a detected object is detected first . 5. The driver assisting system as claimed in any one of claims 2 to 4, wherein said detection performance is esti¬ mated depending on the height of a detected object in the image when this object is detected first. 6. The driver assisting system as claimed in any one of claims 2 to 5, wherein said detection performance is esti¬ mated depending on the track length of a detected object in the image. 7. The driver assisting system as claimed in any one of the preceding claims, wherein said detection performance is estimated using a result from a classification means (16) in said object processing means (16, 17) . 8. The driver assisting system as claimed in claim 7, wherein said detection performance is estimated depending on a classification confidence measure provided by said classi¬ fication means (16) . 9. The driver assisting system as claimed in any one of the preceding claims, wherein said detection performance is estimated using a combination of results for a plurality of detected objects. 10. The driver assisting system as claimed in any one of the preceding claims, wherein an indication is made to the driver via said driver assisting means (18) if the detec¬ tion performance fulfils a predetermined condition. 11. The driver assisting system as claimed in claim 10, wherein a low confidence indication is made to the driver if the estimated detection performance significantly falls below a predetermined normal detection performance. A driver assisting method for a motor vehicle, comprising the steps of recording images of a region surrounding the motor vehicle, detecting objects in the recorded images by image processing, applying further processing to objects detected in the recorded images, and controlling a driver assisting means depending on the result of said object de¬ tection and object processing, characterized by providing an estimation of the detection performance of said object detection step using a result from said object processing step . |
A driver assisting system and method for a motor vehicle
The invention relates to a driver assisting system for a motor vehicle, comprising an imaging means adapted to record images of a region surrounding the motor vehicle, an object detection means for detecting objects in the recorded images by image processing, an object processing means for applying further processing to detected objects, and a driver assisting means controlled depending on the result of said object detection and object processing. The invention furthermore relates to a corresponding driver assisting method.
Such systems are generally known, for example from US 7 263 209 B2. Since the object detection performance of such systems depends on different external conditions, the reliability of the driver assisting system may vary strongly. For a thermal imaging system, for example, the object detection performance decreases when the temperature of the scene gets close to 37 °C because of low thermal contrast, or when it is raining or very humid outside because of lower transmission and more homogenous temperature distribution of objects to be detected. Generally, in the case of unfavourable external conditions the reliability of the driver assisting system may become low, possibly causing misleading of the driver and/or suboptimal control of the driver assisting means. Since, furthermore, the driver does not know the current object detection performance of the system, and may have difficulties to estimate it, prob ¬ lems may arise in situations where the driver reliance on the driver assistant system is not in conformity with its current object detection performance. This is particularly problematic when the driver relies on the driver assistant system under conditions where the current object detection performance is low .
The object of the invention is to provide a driver assisting system and method avoiding the above mentioned problems.
The invention solves this object with the features of the in ¬ dependent claims. The estimation of the current detection per ¬ formance of the object detection means by the detection per ¬ formance estimating means allows further action to be taken by the driver assisting system depending on the estimated detection performance, in order to avoid misleading of the driver or suboptimal control of the driver assisting means.
The estimation of the current detection performance on the ba ¬ sis of a result provided by the object processing means ac ¬ cording to the invention is advantageous over an estimation based on image quality, i.e. signal-to-noise ratio in the de ¬ tected images, as a measure of the object detection perform ¬ ance. First of all, the image quality may be difficult to es ¬ timate reliably. Moreover, by using the image quality as a measure of the object detection performance it is not possible to detect a decreased object detection performance resulting from a property of an object itself. For example in the pedes ¬ trian detection by a thermal imaging means, it is much more difficult to detect a pedestrian with a lot of clothes on since he becomes thermally insulated, leading to a decreased thermal contrast against the background and a correspondingly decreased object detection probability.
Preferably the object detection performance is estimated using a result from a tracking means in the object processing means, because the tracking information very distinctly depends on the object detection reliability. A particularly useful quan ¬ tity for estimating the object detection performance is the frequency of missed detections in a track of a detected object over time, i.e. the number of missed detections related to a suited reference quantity, for example per unit length or di ¬ vided by the number of frames. There is a distinct and unambi ¬ guous relationship between the frequency of missed detections and the object detection performance. In particular, the fre ¬ quency of missed detections increases with decreasing object detection performance.
Another useful quantity for estimating the object detection performance is the distance at which an object is detected first, because an object can be detected more early, or al ¬ ready at a larger distance, if the object detection perform ¬ ance is good. Any other quantity which is directly related to the distance of detection may be equally suited. For example, the height of an object in the image when this object is de ¬ tected first may be used as a measure for the detection dis ¬ tance, because objects which are detected early, i.e. at a large distance, have a small height when being first detected in a track. Another such quantity is the overall track length of a detected object.
The invention is not limited to using results from a tracking means for estimating the object detection performance. In combination or alternatively, the detection performance may for example be estimated using a result from a classification means in said object processing means. In particular, a clas ¬ sification confidence measure provided by a classification means may be related to the object detection reliability and, therefore, be suitable as a measure for the object detection performance .
In general, the term object processing means comprises any functional unit which applies specific processing not to the whole image, or parts of the image not related to a detected object, but to a restricted image area corresponding to de ¬ tected objects, in particular detected objects to be displayed to the driver. Object processing comprises object tracking, object verification and/or object classification without being limited thereon.
In order to obtain a reliable result, the detection performance preferably is estimated using a combination of results for a plurality of objects, not just for a single object that may be for example difficult to detect and therefore gives a wrong result which is not representative.
A preferred course of action to be taken by the driver assist ¬ ing system depending on the estimated detection performance is to make an indication to the driver via the driver assisting means if the estimated detection performance fulfils a prede ¬ termined condition. In particular, a low confidence indication may preferably be made to the driver if the estimated detec ¬ tion performance significantly falls below a predetermined normal detection performance. Such information may be very valuable for the driver, telling him that the drivers' vigi ¬ lance should be increased and/or that the driver assisting system should be ignored or in any case not exclusively relied upon. The indication may be optical, acoustical and/or hapti- cal in any suited manner. In a preferred embodiment, an indi ¬ cation is made on a display device used for displaying information relating to detected objects. For example, the image displayed on the display device may be displayed in a differ- ent colour if the estimated detection performance signifi ¬ cantly falls below a predetermined normal detection perform ¬ ance .
The action to be taken by the driver assisting system depending on the estimated detection performance is not limited to making an indication to the driver. For example, the driver assisting system may be temporarily turned off if the estimated detection performance significantly falls below a prede ¬ termined normal detection performance. In that case, the driver assisting system may advantageously be turned on again if the estimated detection performance raises again above a predetermined detection performance.
In the following the invention shall be illustrated on the ba ¬ sis of preferred embodiments with reference to the accompany ¬ ing drawings, wherein:
Fig. 1 shows a schematic illustration of a vehicle mounted
driver assisting system; and
Fig. 2 shows a schematic illustration of a track of object de ¬ tections under normal object detection performance; and
Fig. 3 shows a schematic illustration of a track of object de ¬ tections under reduced object detection performance.
The driver assisting system 10 is mounted in a motor vehicle and comprises an imaging means 11 for recording images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. Preferably the imaging means 11 comprises one or more optical imaging devices 12, in particu ¬ lar infrared cameras, where infrared covers near IR with wave- lengths below 5 microns and/or far IR with wavelengths beyond 5 microns. Cameras in the visual range are also possible. An ¬ other possibility is combination of IR cameras and cameras in the visible range. Preferably the imaging means 11 comprises a plurality imaging devices 12 in particular forming a stereo imaging means 11; alternatively only one imaging device form ¬ ing a mono imaging means can be used.
The imaging means 11 is preferably coupled to an image pre ¬ processor 13 adapted to control the capture of images by the imaging means 11, receive the electrical signal containing the image information from the imaging means 11, warp pairs of left/ right images into alignment and/or create disparity im ¬ ages, which per se is known in the art. The image preproces ¬ sor 13 may be realized by a dedicated hardware circuit. Alter ¬ natively the pre-processor 13, or part of its functions, can be realized in an electronic processing means 14.
The pre-processed image data is then provided to the elec ¬ tronic processing means 14 where further image and data proc ¬ essing is carried out by corresponding software. Expediently, the electronic processing means 14 has access to an electronic memory means 25. In particular, the image and data processing in the processing means 14 comprises an object detection means 15 for detecting objects in the surrounding of the motor vehicle, an object classification means 16 for classifying the detected objects into at least one object category, such as pe ¬ destrians, other vehicles, bicyclists or large animals, and an object tracking means 17 for tracking over time the position of detected object candidates in the recorded images. Depend ¬ ing on the result of the object detection, classification and tracking processing, driver assisting means 18 are activated or controlled by the processing means 14. The driver assisting means 18 may in particular comprise a display means 19 for displaying information relating to an identified object and/or to provide an optical collision warning to the driver. The driver assisting means 18 may further comprise a warning means adapted to provide a collision warning to the driver by suit ¬ able optical, acoustical and/or haptical warning signals; one or more safety systems, in particular restraint systems such as occupant airbags or safety belt tensioners, pedestrian air- bags, hood lifters and the like; and/or dynamic vehicle con ¬ trol systems such as brakes.
The electronic processing means 14 is preferably programmed or programmable and may comprise a microprocessor or micro ¬ controller, such as DSP, FPGA, ASICS, etc. The image pre ¬ processor 13, the electronic processing means 14 and the mem ¬ ory means 25 are preferably realized in an on-board electronic control unit (ECU) and may be connected to the imaging means 11 via a separate cable or alternatively via a vehicle data bus. In another embodiment the ECU and an imaging device 12 can be integrated into a single unit. All steps from imaging, image pre-processing, image processing to activation or control of driver assisting means 18 are performed automatically and continuously during driving in real time.
The electronic processing means 14 furthermore comprises a de ¬ tection performance estimating means 20 adapted to provide an estimation of the detection performance of the object detec ¬ tion means 15 using a result from the object processing means 16, 17. This will be explained on the basis of Figs. 2 and 3. In the preferred embodiment underlying Figs. 2 and 3, the de ¬ tection performance estimating means 20 estimates the detec ¬ tion performance of the object detection means 15 using a re ¬ sult from the object tracking means 17, only. However, the de- tection performance estimating means 20 may additionally or alternatively also use a result from the object classification means 16 for estimating the detection performance of the ob ¬ ject detection means 15. This is indicated in Fig. 1 by a dashed arrow.
Fig. 2 shows a track 21 of one particular object detected by the object detection means 15 under normal object detection performance over a consecutive series of for example twenty images, where every image corresponds to a defined time frame. This track 21 is provided by the tracking means 17 which is adapted to detect relations between detected objects in con ¬ secutive images, showing that they belong to one and the same object in the surrounding of the motor vehicle.
The number given in Fig. 2 for every frame or image indicates the height of the particular object in the corresponding image. For example, when the object of the track 21 was detected first it had a height of twelve in arbitrary units, as indi ¬ cated by the leftmost number of track 21. In the next image the same object had still a height of twelve. In the third im ¬ age the object could not be detected. In the fourth image the object had still a height of twelve, while in the fifth image the object had a height of thirteen because the vehicle ap ¬ proached the object. In the last image, before the object dropped out of view, the object had a height of twenty-seven as indicated by the rightmost number of track 21.
The track 21 contains altogether five missed detections (each indicated by a zero) . This shows that, for example, a fre ¬ quency of about 20% of missed object detections in an object track may be expected under normal conditions. Fig. 3 shows an object track 22 recorded under less favourable external conditions which lead to a reduced performance of the object detection means 15. The reduced object detection per ¬ formance has several consequences. First of all, the object is detected later, at a shorter distance to the vehicle. There ¬ fore, the object has a larger height when it is detected first (for example height fourteen in Fig. 3), and the overall track length from the first detection until dropping out of view is shorter (for example fourteen frames in Fig. 3) . Furthermore, the number of missed detections significantly increases to for example seven in Fig. 3 or 50% relative to the number of frames .
The detection performance estimating means 20 determines from the track length, the object height at first detection, the number or frequency of missed detections, a combination of such factors, and/or any other suitable quantity, that the ob ¬ ject detection performance of the object detection means 15 is significantly reduced in the example of Fig. 3. For example, the detection performance estimating means 20 may determine a reduced object detection performance if the number of missed detections in relation to the number of frames exceeds 30%.
Preferably the detection performance estimating means 20 does not rely on a single track 22, but uses a suited combination of results for a plurality of detected objects or tracks, preferably at least five tracks, more preferably in the range of eight to twenty tracks. For example, the frequency of missed detections averaged over several tracks may be consid ¬ ered. Alternatively, a reduced object detection performance may be assumed if a reduced object detection performance is detected for a pre-defined number of tracks, or for a pre ¬ defined percentage of a predetermined number of tracks. In view of the above, the detection performance estimating means 20 is expediently adapted to store results for a plurality of objects or tracks in the memory means 25.
In case the detection performance estimating means 20 determines a reduced object detection performance of the object de ¬ tections means 15, it expediently causes suitable action in order to reduce or avoid negative consequences. In particular, the detection performance estimating means 20 may control the display means 19 to indicate to the driver a reduced perform ¬ ance of the driver assisting means 10. This may for example be done by changing the colour of the display 19.
Next Patent: MEDICAMENT DELIVERY DEVICE