Extended Side Views

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. provisional patent application 63/198,859, filed November 18, 2020, and titled “Panoramic Visualization Through Trailer with Rear and Extended Side Views,” the content of which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

[0001] This disclosure relates to a trailer imaging system that provides a panoramic view of a rearward environment of a trailer attached to a tow vehicle.

BACKGROUND

[0002] Trailers are usually unpowered vehicles that are pulled by a powered tow vehicle. A trailer may be a utility trailer, a popup camper, a travel trailer, livestock trailer, flatbed trailer, enclosed car hauler, and boat trailer, among others. The tow vehicle may be a car, a crossover, a truck, a van, a sports-utility -vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to the trailer and pull the trailer. The trailer may be attached to a powered vehicle using a trailer hitch. A receiver hitch mounts on the tow vehicle and connects to the trailer hitch to form a connection. The trailer hitch may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack. Other attachment mechanisms may also be used. In addition to the mechanical connection between the trailer and the powered vehicle, in some examples, the trailer is electrically connected to the tow vehicle.

[0003] Some of the challenges that face tow vehicle drivers is maneuvering the vehicle-trailer system, specifically in the rearward direction, while not being able to see behind the trailer. This creates a sizeable blind spot for the driver of the tow vehicle as the driver cannot see what is behind the trailer or behind the line of sight blocked by trailer during normal driving, parking, reversing, etc. This also introduces safety concerns for those in the tow vehicle, in another vehicle on the same roadway or on foot in proximity to the tow vehicle and trailer.

[0004] Solutions exist, including a providing an image in which the trailer appears transparent, and providing an image in which a rear view of the trailer occupies the space in the image corresponding to the trailer. However, these solutions require stitching or other processing intensive and/or computationally demanding algorithms to provide a panoramic rear view of the trailer which cannot be effectively utilized in a vehicle-trailer camera system having limited processing capabilities. Therefore, it is desirable to have a system that provides the driver with a panoramic view of the environment of the trailer that is not computationally demanding.

SUMMARY

[0005] One aspect of the disclosure provides an example arrangement of operations for a method of providing an image having a panoramic view of an environment associated with a trailer of a vehicle-trailer system. The method includes receiving, at data processing hardware, a first image from a rear trailer camera positioned on a rear portion of the trailer; receiving, at the data processing hardware, a second image from a rear vehicle camera mounted to the tow vehicle; receiving, at the data processing hardware, a third image from a side vehicle camera mounted to the tow vehicle; and receiving, at the data processing hardware, a fourth image from a second side vehicle camera mounted to the tow vehicle. The second image, the third image and the fourth image are combined to create a combined image from the cameras mounted to the tow vehicle. The method further includes determining, by the data processing hardware, a region of the combined image from the cameras mounted to the tow vehicle corresponding to a representation of the trailer. The method further includes creating, by the data processing hardware, an image having a rear view from the trailer based upon the first image and upon the determined region. An image having a panoramic view of an environment associated with the trailer is created by combining the combined image from the cameras mounted to the tow vehicle with the image having a rear view from the trailer. The data processing hardware sends to a display screen, the image having the panoramic view of the environment associated with the trailer along with instructions for displaying the image having the panoramic view.

[0006] Creating the image having the panoramic view of the environment associated with the trailer includes overlaying the image having the rear view from the trailer onto the combined image from the cameras mounted to the tow vehicle without performing an image stitching operation. Further, creating the image having the panoramic view of the environment associated with the trailer includes replacing pixels in the region of the combined image from the cameras mounted to the tow vehicle with pixels from the image having the rear view of the trailer. Creating the image having the panoramic view of the environment associated with the trailer is performed without stitching together the combined image from the cameras mounted to the tow vehicle with the image having a rear view from the trailer.

[0007] Combining the second image, the third image and the fourth image to create the combined image from the cameras mounted to the tow vehicle is performed without stitching together the second image, the third image and the fourth image.

[0008] The method may further include determining an angle between the tow vehicle and the trailer, wherein determining the region of the combined image from the cameras mounted to the tow vehicle corresponding to a representation of the trailer is based upon the determined angle.

[0009] The method may further include displaying, by a display device the image having the panoramic view of the environment associated with the trailer.

[0010] In one implementation, the method includes receiving, at the data processing hardware, a fifth image from a second camera mounted to the trailer, and combining, by the data processing hardware, the first image and the fifth image to create a combined image from the cameras mounted to the trailer. In this implementation, creating the image having the rear view from the trailer is based upon the combined image from the cameras mounted to the trailer and upon the determined region. Moreover, combining the first image and the fifth image to create a combined image from the cameras mounted to the trailer is performed without performing image stitching operations.

[0011] Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. These operations include the method described above.

[0012] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

[0013] FIG. lAis a top view of an exemplary vehicle-trailer system having a tow vehicle towing a trailer.

[0014] FIG. IB is a top view of the exemplary vehicle-trailer system shown in FIG.

1 A at a trailer angle between the tow vehicle and the trailer.

[0015] FIG. 2 is a schematic view of the exemplary vehicle-trailer system of FIG. 1A.

[0016] FIG. 3 is a top view of the exemplary vehicle-trailer system of FIG. 1 in which camera views are identified;

[0017] FIG. 4 is a perspective view of an exemplary intermediate image generated by tow vehicle cameras of the vehicle-trailer system.

[0018] FIG. 5 is a schematic view of an exemplary arrangement of operations for a method that generates an image having a panoramic view of an environment of the trailer attached to the tow vehicle.

[0019] FIGS. 6-8 are resulting images having panoramic views displayed by the exemplary vehicle-trailer system of FIG. 2.

[0020] Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0021] A tow vehicle, such as, but not limited to a car, a crossover, a truck, a van, a sports-utility -vehicle (SUV), and a recreational vehicle (RV) may be configured to tow a trailer. The tow vehicle connects to the trailer by way of a trailer hitch. It may be difficult to drive the vehicle-trailer system in a rearward direction. Therefore, it is desirable to have a system that provides the driver with a panoramic view of the rear and side of the trailer which allows the driver to have a wider range of motion because of the area the driver is able to see.

[0022] Referring to FIGS. 1 A, IB, and 2, in some implementations, a vehicle-trailer system 100 includes a tow vehicle 102 attached to a trailer 106. The tow vehicle 102 includes a vehicle tow ball 104 supported by a vehicle hitch bar 105. The vehicle tow ball 104 is coupled to the trailer 106 by way of a trailer hitch coupler 108 supported by a trailer hitch bar 109 of the trailer 106. The tow vehicle 102 may include a drive system 110 that maneuvers the tow vehicle 102 across a road surface based on drive commands having x, y, and z components, for example. As shown, the drive system 110 includes a front right wheel 112, 112a, a front left wheel 112, 112b, a rear right wheel 112, 112c, and a rear left wheel 112, 112d. The drive system 110 may include other wheel configurations as well. The drive system 110 may also include a brake system (not shown) that includes brakes associated with each wheel 112, 112a-d, and an acceleration system (not shown) that is configured to adjust a speed and direction of the tow vehicle 102. In addition, the drive system 110 may include a suspension system (not shown) that includes tires associates with each wheel 112, 112a-d, tire air, springs, shock absorbers, and linkages that connect the tow vehicle 102 to its wheels 112, 112a-d and allows relative motion between the tow vehicle 102 and the wheels 112, 112a-d.

[0023] The tow vehicle 102 may move across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the tow vehicle 102: a transverse axis Xv, a fore-aft axis Yv, and a central vertical axis Zv. The transverse axis Xv extends between a right-side and a left-side of the tow vehicle 102. A forward drive direction along the fore-aft axis Yv is designated as Fv, also referred to as a forward motion. In addition, an aft or rearward drive direction along the fore-aft direction Yv is designated as Rv, also referred to as rearward motion. In some examples, the tow vehicle 102 includes a suspension system (not shown), which when adjusted causes the tow vehicle 102 to tilt about the Xv axis and or the Yv axis, or move along the central vertical axis Zv.

[0024] Moreover, the trailer 106 follows the tow vehicle 102 across the road surface by various combinations of movements relative to three mutually perpendicular axes defined by the trailer 106: a trailer transverse axis XT, a trailer fore-aft axis YT, and a trailer central vertical axis ZT. The trailer transverse axis XT extends between a right-side and a left-side of the trailer 106 along a trailer turning axle 107. In some examples, the trailer 106 includes a front axle (not shown) and rear axle 107. In this case, the trailer transverse axis XT extends between a right-side and a left-side of the trailer 106 along a midpoint of the front and rear axle (i.e., a virtual turning axle). A forward drive direction along the trailer fore-aft axis YT is designated as FT, also referred to as a forward motion. In addition, a trailer aft or rearward drive direction along the fore-aft direction YT is designated as RT, also referred to as rearward motion. Therefore, movement of the vehicle-trailer system 100 includes movement of the tow vehicle 102 along its transverse axis Xv, fore-aft axis Yv, and central vertical axis Zv, and movement of the trailer 106 along its trailer transverse axis XT, trailer fore-aft axis YT, and trailer central vertical axis ZT. Therefore, when the tow vehicle 102 makes a turn as it moves in the forward direction Fv, then the trailer 106 follows along. While turning, the tow vehicle 102 and the trailer 106 form a trailer angle a being an angle between the vehicle fore-aft axis Yv and the trailer fore-aft axis YT.

[0025] The tow vehicle 102 may include a user interface 120. The user interface 120 may include a display 122, a knob, and a button, which are used as input mechanisms. In some examples, the display 122 may show the knob and the button. While in other examples, the knob and the button are a knob button combination. In some examples, the user interface 120 receives one or more driver commands from the driver via one or more input mechanisms or a touch screen display 122 and/or displays one or more notifications to the driver. The user interface 120 is in communication with a controller 140. In some examples, the display 122 displays an image 133 of an environment of the tow vehicle 102 which includes the trailer articulation view.

[0026] The tow vehicle 102 may include a sensor system 130 to provide reliable and robust driving. The sensor system 130 may include different types of sensors that may be used separately or with one another to create a perception of the environment of the tow vehicle 102 and the trailer 106 that is used for the tow vehicle 102 to drive and aid the driver in make intelligent decisions based on objects and obstacles detected by the sensor system 130. The sensor system 130 may include the one or more cameras 132, 132a-d supported by the vehicle-trailer system 100. In some implementations, the tow vehicle 102 includes a rear vehicle camera 132a (i.e., a first camera) that is mounted to provide a view of a rear-driving path for the tow vehicle 102, or in other words, the rear vehicle camera 132a captures images 133 of a rear environment of the tow vehicle 102. Additionally, the sensor system 130 includes a trailer rear camera 132b (i.e., a second camera) that is mounted to provide a view of a rear-driving path for the trailer 106, or in other words, the rear trailer camera 132b captures images 133 of a rear environment of the trailer 106. Tow vehicle 102 may further include side view cameras 132c and 132d (i.e., third and fourth cameras) each of which captures images 133 which provide a corresponding side view of tow vehicle 102 and trailer 106. In some examples, the sensor system 130 also includes side trailer camera 132e and 132f (i.e., fifth camera and sixth camera, not shown in the drawings) each mounted to provide corresponding side images 133 of the side environment of the trailer 106. In some examples, additional one or more cameras 132 are positioned on a front of the tow vehicle 102 to capture a forward-driving path of the tow vehicle 102.

[0027] In some implementation, the rear vehicle camera 132a, the rear trailer camera 132b, and the side trailer cameras 132c, 132d include a fisheye lens having an ultra- wide- angle lens that produces strong visual distortion intended to create a wide panoramic or hemispherical image. Fisheye cameras capture images having an extremely wide angle of view. Moreover, images captured by the fisheye camera have a characteristic convex non-rectilinear appearance. Other types of cameras may also be used to capture the images 133.

[0028] The sensor system 130 may also include other sensors 134 that detect the vehicle motion, i.e., speed, angular speed, position, etc. The other sensors 134 may include an inertial measurement unit (IMU) configured to measure the vehicle’s linear acceleration (using one or more accelerometers) and rotational rate (using one or more gyroscopes). In some examples, the IMU also determines a heading reference of the tow vehicle 102. Therefore, the IMU determines the pitch, roll, and yaw of the tow vehicle 102. The other sensors 134 may also include, but are not limited to, radar, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), ultrasonic, HFL (High Resolution 3D Flash LIDAR), etc. In some implementations, the sensor system 130 may provide external sensor data received from other systems or vehicles, such as by way of V2X communication or any other communication.

[0029] The controller 140 includes a computing device (or processor) 142 (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory 144 (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s) 142. The controller may be supported by the tow vehicle 102, the trailer 106, or both the tow vehicle 102 and the trailer 106. In some examples, the controller 140 executes an imaging system 150 that provides the driver of the tow vehicle 102 with images having panoramic views of the environment behind or otherwise associated with the trailer 106. [0030] The imaging system 150 receives images 133 from a plurality of cameras 132 and provides an image having a panoramic view of the rear environment of the tow vehicle and trailer 106. The imaging system 150 solves the difficulties that the driver faces when backing up the vehicle-trailer system 100 by showing images having panoramic views of the rear environment of the trailer 106 on the display 122. The image having the panoramic view is based upon images 133 captured by the rear trailer camera 132b, the rear vehicle camera 132a and the side vehicle cameras 132c, 132d.

[0031] FIG. 3 illustrates the views provided by cameras 132 of imaging system 150, in which the views are grouped into four zones A1-A4. Zone Al corresponds to a view provided by side vehicle cameras 132c, zone A2 corresponds to a view provided by side vehicle camera 132d, zone A3 corresponds to a view provided by rear vehicle camera 132a, and zone A4 corresponds to a view provided by rear trailer camera 132b.

[0032] In general terms, the image having a panoramic view provided by imaging system 150 is created by overlaying a portion of the image 133b created by the rear trailer camera 132b onto a combined image created by rear vehicle camera 132a and side vehicle cameras 132c and 132d. The portion of the image 133b is overlaid and/or displayed within a boundary 124 of a trailer representation 126 in the combined image, as described in greater detail below.

[0033] FIG. 5 provides an example arrangement of operations for a method 500 of providing an image having a panoramic view of an environment behind or otherwise associated with a trailer 106 of a vehicle-trailer system 100. In an example embodiment, the blocks of method 500 are performed by the data processing hardware 140, 142. At block 502, the method 500 includes receiving, at the data processing hardware 140, 142, a first image 133, 133b from the rear trailer camera 132, 132b positioned on a rear portion of the trailer 106. The first image 133, 133b captures a rearward environment of the trailer 106. This block 502 also includes receiving a second image 133, 133a showing a captured rearward environment of tow vehicle 102 from the rear vehicle camera 132a; a third image 133, 133c showing a captured side view of tow vehicle 102 and trailer 106 from the side vehicle camera 132c; and a fourth image 133, 133d showing another captured side view of tow vehicle 102 and trailer 106 from the second side vehicle camera 132d.

[0034] At block 504, the images 133a, 133c and 133d captured by the tow vehicle cameras 132a, 132c and 132d, respectively, are combined. In particular, the second image 133a (from rear vehicle camera 132a), third image 133c (from side vehicle camera 132c) and fourth image 133d (from side vehicle camera 132d) are combined to create a single, combined image. The combined image is created with an assumed world geometry in the shape of a 3-D bowl. An example of the combined image is illustrated in FIG. 4, showing a representation 126 of trailer 106 therein. Combining these images 133a, 133c and 133d do not include image stitching or otherwise computationally demanding operations. In an alternative embodiment, image stitching is used to combine the images.

[0035] Block 506 is optionally performed if there are a plurality of cameras mounted onto trailer 106. Specifically, the images captured by the cameras of trailer 106 are combined to create a single combined image. This combined image from the trailer cameras is also created with an assumed world geometry in the shape of the 3-D bowl. If there is only a single camera located on trailer 106, then block 506 is not performed. Combining the images from the trailer cameras also does not include image stitching or other computationally demanding operations. Alternatively, image stitching is used to combine the images.

[0036] Block 508 includes determining, at the data processing hardware 140, 142, a trailer angle a based on cam era/ sensor system data 131 received from rear vehicle camera 132a and/or another camera 132 and/or sensor of a sensor system 130. As shown in FIG. IB, the trailer angle a is an angle between a vehicle fore-aft axis Yvand a trailer fore-aft axis YT. For example, the angle a may be determined based upon an image 133a generated by rear vehicle camera 132a of tow vehicle 102. The image 133a includes a representation of trailer 106 which data processing hardware 140, 142 uses to estimate the trailer angle a.

[0037] At block 509, a region of the combined image created in block 504 that is obscured by the representation of trailer 106 is determined. In particular, this region is determined based upon the angle between tow vehicle 102 and trailer 106 identified in block 508 and upon known dimensions of trailer 106.

[0038] At block 510, an image is created based upon the first image 133a generated by rear vehicle camera 132a and upon the region determined in block 509. In the event multiple cameras 132 are disposed on trailer 106, the image is created from the combined image from the trailer cameras created in block 506. The image created in this block is extracted from the first image 133a and/or the combined image 133 created in block 506 based upon the region determined in block 509.

[0039] In block 512, the image from the trailer camera(s) created in block 510 is combined with the combined image from the tow vehicle cameras created in block 504 to create a combined image 600, 700, 800 providing a panoramic view of images generated by the cameras of the tow vehicle 102 and the trailer 106. This combined image 600, 700, 800 is created by overlaying the image created from the trailer camera(s) in block 510 onto the combined image from the tow vehicle cameras so that the region of the combined image from the tow vehicle cameras determined in block 509 that is obscured by the representation of the trailer 106 is replaced by the image from the trailer cameras created in block 510. In one implementation, block 512 replaces pixels in the region of the combined image from the tow vehicle cameras (i.e., that portion of the combined image corresponding to the representation of the trailer 106) with pixels of the image from the trailer camera(s) created in block 510. No stitching together of images or other computationally demanding operations are performed. The resulting image created in block 510 has a panoramic view of the environment behind and/or associated with trailer 106, but is not one seamless image. For example, the region of the combined tow vehicle image appears as a window having an image of the rear view of the trailer 106 corresponding to the replacement pixels of the image from the trailer camera(s).

[0040] At block 514, the image of the panoramic view created in block 512, along with corresponding instructions, is sent to display 122 for display thereon, and the image is then displayed on display 122. FIGS. 6-8 illustrate displayed panoramic views of the environment behind or otherwise associated with trailer 106. FIGS. 6 and 8 are panoramic views of images 600 and 800, respectively, displayed by display 122 with trailer 106 being disposed at an angle a of 180 degrees (i.e., directly behind tow vehicle 102), and FIG. 7 is combined image 700 of a panoramic view with trailer 106 disposed at an angle a that is less than 180 degrees relative to tow vehicle 102. In each panoramic view image of FIGS. 6-8, the space previously occupied by the representation of trailer 106 is replaced with the image created in block 510, i.e., the portion extracted from image 133a generated by rear vehicle camera 132a. As more clearly seen in FIG. 8, the panoramic view image 800 includes region 802 determined in block 509 and having therein a portion of image 133b. This region 802 does not create a seamless image with the combined tow vehicle camera image 804 determined in block 504. Instead, the region 802 of image 800 is more akin to windows through which portions of image 133b captured by rear trailer camera 132, 132b are displayed.

[0041] An advantage of method 500 is that displaying the extracted portion of the first image 133a and the combined image created in block 504 is performed without stitching or other computationally demanding operations. As a result, the display of the image 600, 700, 800 having a panoramic view as described above is suitable for lower cost camera systems having limited processing capabilities and/or processing speeds. [0042] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. [0043] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. [0044] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Moreover, subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The terms “data processing apparatus”, “computing device” and “computing processor” encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus. [0045] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multi-tasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0046] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.