FIELD

[0001] The disclosed embodiments relate generally to a driver assist system and method thereof, and more particularly, to a system for rearward lane marker identification and tracking.

BACKGROUND

[0002] Because there is typically very little illumination of the road surface behind a vehicle, it may be difficult to distinguish lane lines and the location of other vehicles with respect to the lane lines, when using rearview mirrors.

SUMMARY

[0003] The disclosed embodiments are directed to a rearward lane display system including an image sensor configured to capture one or more images including lane markers forward of a vehicle, one or more display mirrors, and a processor in communication with the image sensor and the display mirrors, where the processor is configured to process the one or more images to predict where the lane markers will appear rearward of the vehicle, and display representations of the predicted lane markers in the one or more display mirrors.

[0004] The image sensor may include a camera with an infrared light source.

[0005] The one or more display mirrors may include an interior rearview mirror assembly.

[0006] The one or more display mirrors may include a side exterior rearview mirror assembly.

[0007] The one or more display mirrors may include a partially reflective and partially transmissive electro-optic mirror element, and a display configured to provide an overlay of the predicted lane markers.

[0008] The processor may be configured to process the one or more images captured from the image sensor by receiving a present image from the image sensor, determining coordinates of lane marker edge points in the present image, modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image, and fitting lines to the modified coordinates for display in the one or more display mirrors.

[0009] Determining coordinates of lane marker edge points in the present image may include utilizing a lane marker detector to identify lane marker positions.

[0010] Modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image may include establishing a coordinate system of the vehicle where an x-axis extends to the left and right of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to a front and rear of the vehicle, rotating the coordinates about the x, y, and z axes, and translating the coordinates along the z axis.

[0011] Modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image may include removing lane marker edge points that exceed a predetermined distance from the vehicle.

[0012] Fitting lines to the modified coordinates for display in the one or more display mirrors may include using a model including one or more of a pair of polynomials which compute an X and Y value for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

[0013] The disclosed embodiments are further directed to a method for displaying rearward lane markers, the method including capturing one or more images including lane markers forward of a vehicle, processing the one or more images to predict where the lane markers will appear rearward of the vehicle, and displaying representations of the predicted lane markers in one or more display mirrors.

[0014] The method may include using a camera with an infrared light source to capture the one or more images.

[0015] The method may include providing an overlay of the predicted lane markers on a partially reflective and partially transmissive electro-optic mirror element.

[0016] The method may include displaying representations of the predicted lane markers using an interior rearview mirror assembly.

[0017] The method may include displaying representations of the predicted lane markers using a side exterior rearview mirror assembly.

[0018] Processing the one or more images captured from the image sensor may include receiving a present image from the image sensor, determining coordinates of lane marker edge points in the present image, modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image, and fitting lines to the modified coordinates for display in the one or more display mirrors.

[0019] Determining coordinates of lane marker edge points in the present image may include utilizing a lane marker detector to identify lane marker positions.

[0020] Modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image may include establishing a coordinate system of the vehicle where an x-axis extends to the left and right of the vehicle, a y-axis extends below and above the vehicle, and a z-axis extends to a front and rear of the vehicle, rotating the coordinates about the x, y, and z axes, and translating the coordinates along the z axis.

[0021] Modifying the coordinates to reflect a change in vehicle orientation between receiving the present image and a previous image may include removing lane marker edge points that exceed a predetermined distance from the vehicle.

[0022] Fitting lines to the modified coordinates for display in the one or more display mirrors may include using a model including one or more of a pair of polynomials which compute an x and y value for a given z value, a least squares regression, a linear regression, a Theil-Sen estimator.

[0023] These and other aspects, objects, and features of the present disclosure will be understood and appreciated by those skilled in the art upon studying the following specification, claims, and appended drawings. It will also be understood that features of each example disclosed herein may be used in conjunction with, or as a replacement for, features of the other examples.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The foregoing aspects and other features of the embodiments are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0025] Figure 1 shows a block diagram of an exemplary vehicle control system in accordance with the disclosed embodiments;

[0026] Figure 2 depicts a diagram of a vehicle incorporating the disclosed embodiments;

[0027] Figure 3 depicts various components of an example of an exterior rearview mirror assembly; [0028] Figure 4 illustrates an example of an interior rearview mirror assembly;

[0029] Figure 5 shows a procedure for predicting and displaying rearward lane marker appearance;

[0030] Figure 6 shows an exemplary forward display of system identified left and right lane markers;

[0031] Figure 7 illustrates an exemplary view of modeled lane lines overlaid on an interior rearview display;

[0032] Figure 8 depicts a left side modeled lane line overlaid on a left side exterior rearview display; and

[0033] Figure 9 depicts a right side modeled lane line overlaid on a right side exterior rearview display.

DETAILED DESCRIPTION

[0034] Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of method steps and apparatus components related to a vehicular system for determining roadway lane markers. Accordingly, the apparatus components and method steps have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0035] In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises ... a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. [0036] It will be appreciated that embodiments of the invention described herein may be comprised of one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of a vehicular system and method for determining roadway lane markers as described herein. The non-processor circuits may include, but are not limited to signal drivers, clock circuits, power source circuits, and user input devices. As such, these functions may be interpreted as steps of a method to determine roadway lane markers.

[0037] Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs) or field programmable gate array (FPGA), in which each function or some combinations of certain of the functions are implemented as custom logic. It should be understood that a combination of any number of approaches may be used. Thus, methods and means for these functions have been described herein. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein, will be readily capable of generating such software instructions, programs and integrated circuits with minimal experimentation.

[0038] Figure 1 shows a block diagram of an exemplary vehicle control system 100 in accordance with the disclosed embodiments. The control system 100 may include a forward facing image sensor 105A, a rearward facing image sensor 105B, an image controller 1 10, a processor 1 15, a lane marker detector 120, a lane departure detector 125, rearward lane projection circuitry 130, and a vehicle interface 140. The lane marker detector 120 includes a left buffer 135 for storing left lane marker edge points and a right buffer 140 for storing right lane marker edge points.

[0039] The forward facing image sensor 105A and the rearward facing image sensor

105B may be cameras incorporating charge coupled devices or CMOS devices. The forward facing image sensor 105A and the rearward facing image sensor 105B may also include one or more light sources, for example, near infrared light sources, and may generally operate to provide a series of image frames. Exemplary image sensors may include those described in U.S. Patent No. 7,432,967, entitled "Control Circuit for Image Array Sensors," U.S. Patent No. 6,008,486, entitled "Wide Dynamic Range Optical Sensor," U.S. Patent No. 7,567,291 , entitled "Vehicle Vision System," U.S. Patent No. 7,683,326, entitled "Vehicle Vision System with High Dynamic Range," U.S. Patent No. 7,423,821 , entitled "Improved Vision System," and U.S. Patent Nos. 8,289,430 and 8,305,471 , both entitled "Improved Imaging Device," all of which are incorporated by reference herein in their entirety.

[0040] The image controller 1 10 may control and receive data from the forward facing image sensor 105A and the rearward facing image sensor 105B, and may also serve to perform pre-processing functions such as image auto-exposure, dynamic range compression, filtering, and color computation.

[0041] The processor 1 15 may be a microprocessor configured discreetly or instantiated using, for example, one or more components such as FPGAs, or a combination of discrete components and FPGAs or the like. The processor 1 15 may request images from the image controller 106 and may also be enabled to directly control and receive images from the forward facing image sensor 105A and the rearward facing image sensor 105B. The lane marker detector 120 and lane departure detector 125 receive the image frames produced by the forward facing image sensor 105A, where the lane marker detector 120 may operate to identify a lane marker position on a roadway, and the lane departure detector 125 may operate to detect a departure from a driving lane. The rearward lane projection circuitry 130 may operate on data from one or more of the lane marker detector 120 and the lane departure detector 125 to provide an image of roadway lanes behind a vehicle to be superimposed on images captured by the rearward facing image sensor 105B.

[0042] In some embodiments, the lane marker detector 120, the departure detector 125, and the rearward lane projection circuitry 130 may be implemented in computer code embodied on a medium readable by the processor 1 15, and the processor 1 15 may be configured to perform lane marker and lane departure detection, and to provide a rearward projection of roadway lanes. In other embodiments, the lane marker detector 120, lane departure detector 125, and rearward lane projection circuitry 130 may be embodied as discrete circuitry including one or more processors, memories, state machines, ASICs, FPGAs, or other components that may operate to perform the functions described herein.

[0043] Typical systems for lane marker detection and lane departure detection may include those described in U.S. Patent No. 7,881 ,839, entitled "Image Acquisition and Processing Systems for Vehicle Equipment Control,” U.S. Patent No. 8,543,254, entitled "Vehicular Imaging System and Method for Determining Roadway Width,” U.S. Patent No. 9,098, 751 , entitled "System and Method for Periodic Lane Marker Identification and Tracking,” and U.S. Patent No. 9,230,183, entitled "Automatic Vehicle Equipment Monitoring, Warning, and Control System,” all of which are incorporated by reference herein in their entirety. Information from the lane marker detector 120 may be projected for viewing, for example, using a heads up display.

[0044] Figure 6 shows an exemplary display of identified left and right lane markers 605,

610 that might be viewed through a windshield or displayed on a windshield of the vehicle.

[0045] The vehicle interface 140 may operate to receive various vehicle parameters from other vehicle systems, including parameters such as speed, pitch, roll, yaw, steering wheel position, and vehicle direction, and provide those parameters to the lane marker detector 120, lane departure detector 125, and rearward lane projection circuitry 130.

[0046] Figure 2 depicts a diagram of a vehicle 200 incorporating the disclosed embodiments. The vehicle 200 may include a left side exterior rearview mirror assembly 205, a right side exterior rearview mirror assembly 210, a center high mounted stop light 215, A-pillars 220a, 220b, B-pillars 225a, 225b, C-pillars 230a, 230b, and an interior rearview mirror assembly 235. It should be understood that the exterior rearview mirror assemblies 205, 210, and the interior review mirror assembly 235 may be automatic dimming electro-optic mirrors, and may incorporate one or more displays as will be explained below. It should also be understood that while the forward facing image sensor 105A, image controller 1 10, processor 1 15, lane marker detector 120, and lane departure detector 125 may be positioned in any suitable location, in some embodiments, the forward facing image sensor 105A, image controller 1 10, processor 1 15, lane marker detector 120, and lane departure detector 125 may be located near, on, or within any of the left side exterior rearview mirror assembly 205, right side exterior rearview mirror assembly 210, center high mounted stop light 215, A-pillars 220a, 220b, B-pillars 225a, 225b, C-pillars 230a, 230b, and the interior rearview mirror assembly 235.

[0047] In some embodiments, the forward facing image sensor 105A may be located on a mount of the interior rearview mirror assembly 235, and in one or more embodiments the image controller 1 10 and the processor 1 15 may be co-located with the forward facing image sensor 105A. In one or more embodiments, the rearward facing image sensor 105B may be located in the center high mounted stop light 215, or located in a mount 245 on the rear of the vehicle 200.

[0048] Figure 3 depicts various components of an example of the left side exterior rearview mirror assembly 205 according to the disclosed embodiments. The left side exterior rearview mirror assembly 205 may include an electro-optic mirror element 305 and an exterior rearview display 310 mounted within a housing 315. The left side exterior rearview mirror assembly 205 may also include one or more filters 320, electrical connections, mounts, and other components for mounting the exterior rearview mirror assembly 300 to the vehicle 200 and for providing a side rearview of the vehicle 200. The electro-optic mirror element 305 may be partially reflective and partially transmissive such that information displayed by the exterior rearview display 310 is visible. The exterior rearview display 310 may utilize LCD, LED, OLED, plasma, DLP, or any suitable display technology and at least operates to provide an overlay of information supplemental to a reflected image provided by the electro-optic mirror element 305. In some embodiments, the exterior rearview display 310 may provide a partial side rearview that may be provided by the rearward facing image sensor 105B and may include predicted lane marker indications, while in other embodiments, the exterior rearview display 310 may provide an entire side rearview including the predicted lane marker indications. It should be understood that the right side exterior rearview mirror assembly 210 may include the same components and may provide the same partial or entire view.

[0049] Figure 4 illustrates an example of the interior rearview mirror assembly 235 according to the disclosed embodiments. The interior rearview mirror assembly 235 may include an electro-optical element 405, a front shield 410 and a rear shield 415 enclosing an interior rearview display 420, a housing 425, and a mounting member 430. The electro-optic mirror element 405 of the interior rearview mirror assembly 235 may be partially reflective and partially transmissive in order for information displayed by the interior rearview display 410 to be visible to a user. The interior rearview display 410 may utilize LCD, LED, OLED, plasma, DLP, or any suitable display technology and at least operates to provide an overlay of information supplemental to a reflected image provided by the electro-optic mirror element 405. In some embodiments, the interior rearview display 410 may occupy only a portion of a surface of the electro-optic mirror element 405 and display a partial rearview provided by the rearward facing image sensor 105B that includes predicted lane marker indications, while in other embodiments, the interior rearview display 410 may occupy a substantial portion or approximately all of the surface of the electro-optic mirror element 405 and may provide an entire rearview provided by the rearward facing image sensor 105B that includes the predicted lane marker indications. Exemplary interior rearview mirror assemblies may include those described in U.S. Patent No. 9,598,018, entitled“Display Mirror Assembly,” and U.S. Patent Application No. 2017/00880055, entitled“Full Display Mirror,” all of which are incorporated by reference herein in their entirety.

[0050] As mentioned above, the rearward lane projection circuitry 130 may operate on data from one or more of the lane marker detector 120 and the lane departure detector 125 to provide a display of roadway lanes that are behind the vehicle 200. For purposes of the disclosed embodiments, a coordinate system relative to the vehicle 200 is defined in terms of a three dimensional coordinate system with an origin at a point on the ground, centered between two rear tires of the vehicle 200. Each point in the coordinate system is defined in terms of (x,y,z) relative to the origin, where an x-axis extends to the left and right of the vehicle, where x is a distance to the right of the vehicle center and -x is a distance to the left of the vehicle center, a y-axis extends below and above the vehicle, where y is a distance below ground level and -y is a distance above ground level, and a z-axis extends to the front and rear of the vehicle, where z is a distance toward the front of the vehicle 200 and -z is a distance toward the rear of the vehicle 200. The rearward facing image sensor 105B position and orientation may be calibrated relative to the vehicle’s coordinate system. In addition, the focal length, optical center, and lens distortion coefficients of the rearward facing image sensor 105B may also be calibrated.

[0051] Figure 5 shows a procedure 500 for predicting where the lane markers will appear rearward of the vehicle 200 and displaying the lane markers for a user.

[0052] The lane marker detector 120 receives image frames from the forward facing image sensor 105A as shown in block 505. As each new frame is received, coordinates of points on the left lane marker, referred to as left edge points, in the image frame are determined within the coordinate system relative to the vehicle 200 and are stored in the left buffer 135, as shown in block 510. Coordinates of points on the right lane marker, referred to as right edge points, in the image frame are determined within the coordinate system relative to the vehicle 200 and are stored in the right buffer 140, as shown in block 515. As each frame is received, the rearward lane projection circuitry 130 calculates an amount of time between receiving the present frame and the previous frame, as shown in block 520.

[0053] In some embodiments, the forward facing image sensor 105, the image controller

1 10, or the processor 1 15 may time stamp frames as they are generated or retrieved, and the lane marker detector 120 may use the time stamps to calculate the amount of time between successive frames. In other embodiments, the lane marker detector 120 may reference a clock signal, or perform some other operation to calculate the amount of time between receiving frames.

[0054] As shown in block 525, the lane marker detector 120 performs the following operations on the coordinates of each of the edge points stored in the left and right buffers 135, 140 to reflect changes in vehicle orientation and position between receiving a present fame and a previous frame:

[0055] the coordinates for each edge point are rotated about the y axis (yaw) using the following matrix

cos(r _y ) 0 sin(r _y ) -x-

0 1 0 y

-sin(r _y) 0 cos(r _y) z

[0056] the coordinates for each edge point are rotated about the x axis (pitch) using the following matrix

[0057] each edge point is rotated about the z axis (roll) using the following matrix; and

[0058] each edge point is translated along the z axis (speed) using the following matrix

[0059] Alternately, the operations on the coordinates of each of the edge points stored in the left and right buffers 135, 140 may be compiled into a single matrix multiplication:

[0060] Points in the left and right buffers 135, 140 that exceed a particular distance from the vehicle as a result of vehicle travel may then be removed from the buffers as shown in block 530. As a non-limiting example, points that are more than 50 meters behind the vehicle 200 may be removed

[0061] A model may then be used to fit a line to the modified coordinates in each buffer independently as shown in blocks 535 and 540. For example, in some embodiments, a pair of polynomials which compute an X and Y value for a given z value may be used. For example, f _x (z) = a ₃ z ³ + a ₂ z ² + a z + a ₀ and f _y (z) = b ₃ z ³ + b ₂ z ² + b _t z + b ₀ may be used. In other embodiments, a least squares regression, a linear regression, a Theil-Sen estimator, or any suitable modeling technique may be utilized to fit a line to the modified coordinates.

[0062] As shown in block 545, the modeled fitted lines may then be plotted over the view shown on the interior rearview display 410. In some embodiments, a projection matrix may be used to determine pixels of the review display 410 corresponding to points of the modeled fitted lines. Figure 7 illustrates an exemplary view shown on the interior rearview display 410 including the modeled fitted lines 705, 710.

[0063] Optionally, the line fitted to the modified coordinates in the left buffer may be overlaid in the view provided by the left side exterior rearview mirror assembly 205, as shown in Figure 8, and the line fitted to the modified coordinates in the right buffer may be overlaid in the view provided by the right side exterior rearview mirror assembly 210, as shown in Figure 9. Other image operations, such as enhancing contrast around the modeled fitted lines, or increasing brightness may also be performed.

[0064] It is noted that the embodiments described herein can be used individually or in any combination thereof. It should be understood that the foregoing description is only illustrative of the embodiments. Various alternatives and modifications can be devised by those skilled in the art without departing from the embodiments. Accordingly, the present embodiments are intended to embrace all such alternatives, modifications and variances that fall within the scope of the appended claims.

[0065] Various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings. Flowever, all such and similar modifications of the teachings of the disclosed embodiments will still fall within the scope of the disclosed embodiments.

[0066] Various features of the different embodiments described herein are interchangeable, one with the other. The various described features, as well as any known equivalents can be mixed and matched to construct additional embodiments and techniques in accordance with the principles of this disclosure.

[0067] Furthermore, some of the features of the exemplary embodiments could be used to advantage without the corresponding use of other features. As such, the foregoing description should be considered as merely illustrative of the principles of the disclosed embodiments and not in limitation thereof.