Field

The present application relates to a system and method for monitoring a connection between vehicles. In particular, the system monitors components of a connection between a vehicle and a trailer.

Background

Connections between vehicles are common, such as the connection between a car and trailer. Such connections typically comprise components to facilitate safe operation. Examples of such components include a coupler e.g. a tow-ball, a break away cable, a handbrake, a jockey wheel, and an electrical lighting connector. To ensure safe operation, a user must ensure to manually configure these components appropriately for each driving/towing scenario. For example, the jockey wheel should be stowed when the trailer is about to be towed to avoid damage to the jockey wheel when towing. Wrongly configuring one or more components could lead to personal injury or damage to the vehicle, trailer, or connection.

Known systems of monitoring vehicular connections rely on electronic signals and adaptations to the trailers and/or the connections.

US2019386438A1 describes a vehicle plug for transmitting power and signals between a vehicle and a trailer that includes a power conductor and one or more video signal conductors.

GB2429046A relates to an electrical system to monitor the connection of a safety cable to a towing vehicle. The monitor may also detect the application of the handbrake, sense the lights, and/or monitor a hydraulic connection to the trailer.

It is an object of the present invention to monitor a vehicular connection in a manner that addresses at least some of the deficiencies in known methods. Summary

The present invention relates to a versatile camera-based system to monitor components of a connection between vehicles. In some cases, the system may provide a visual and/or audio alert to the user if the states of components indicate a potential concern. Compared to known monitoring systems, the described system is reliable, expandable, and does not require contact to operate. The described system can also operate over a wide angular range and works with most existing trailers.

In some embodiments, the alert to the user may also immobilize a vehicle by providing a signal to the vehicle’s system bus (BUS), for example, a controller area network (CAN) bus.

Brief Description of the Drawings

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

Figure 1 shows a vehicle and a trailer coupled by a connection. The vehicle has a mounted camera orientated towards the trailer;

Figure 2 shows steps in a process for monitoring connection components;

Figure 3 shows a flow chart of example processes for monitoring connection components;

Figure 4 shows part of an image from a camera mounted on a vehicle orientated towards a trailer that is coupled by a connection to the vehicle;

Figure 5 shows part of the image from figure 4 with overlaid regions of interest for a jockey wheel and a handbrake; and

Figure 6 shows a display in a vehicle displaying a connection to a trailer.

Description Figure 1 shows an example of a vehicle 100 with a coupled trailer 110. A trailer is any vehicle that may be towed by another vehicle. (For example, a car that is being towed by another car may be considered as a trailer.) The trailer 110 is coupled to the vehicle 100 by a connection 105. It makes no difference whether the connection 105 is part of the vehicle 100, the trailer 110 or a separate item, provided it can couple vehicles together to enable towing. The connection 105 comprises one or more components to ensure the safe operation of the trailer 110. Examples of such components include couplers such as a tow-ball, breakaway cables, handbrakes, jockey wheels, and electrical connectors such as electric lighting connectors.

The vehicle 100 comprises a mounted camera 101. The mounted camera 101 is orientated such that its field of view 101 a covers at least part of the trailer 110 and the connection 105. The vehicle 100 does not need to be the source of motive force for towing the trailer 110. For example, the vehicle 100 on which the camera 101 is mounted can itself be a trailer e.g. a car-trailer-trailer configuration.

The field of view 101 a is wide and this is typically achieved by the camera 101 having a wide field of view lens, such a fisheye lens. A fisheye lens is preferable as these are generally cylindrically symmetric. In other applications of the invention, the field of view may be less or more than 180 degrees. Whilst a fisheye lens is preferred, any other lens that provides a wide field of view can be used. In this context, a wide field of view is a lens having a field of view over 100 degrees, preferably over 150 degrees and more preferably over 170 degrees.

The sensitivity of the camera used in the invention need not be limited to any specific range of wavelengths but most commonly it will be used with cameras that are sensitive to visible light. The camera 101 will generally be in the form of a camera module comprising a housing for a lens and a sensor, the lens serving to focus light onto the sensor. The camera module may also have electronics to power the sensor and enable communication with the sensor. The camera module may also comprise electronics to process the image. The processing can be low level image signal processing, for example, gain control, exposure control, white balance, denoise, etc. and/or it can involve more powerful processing for example, for computer vision. As such, not only can raw images or minimally processed images be employed in embodiments, but it is also possible to use transformed versions of an acquired image, so for example, some embodiments could produce histogram of oriented gradients (HOG) maps or integral images or any other form of feature descriptor from an acquired image for use in downstream processing.

In some embodiments, one component of the connection to be monitored is a handbrake of a trailer. A handbrake is typically an independent braking system mounted on the trailer. A user should disengage the trailer handbrake before towing the trailer, to prevent inconvenience to the user or, in more serious cases, damage to the trailer braking system or towing vehicle.

Another component of the connection that is monitored in some embodiments is a jockey wheel. A jockey wheel is a small wheel, generally at the front of a trailer, that can be deployed to assist in manoeuvring, or jockeying, the trailer when it is not coupled to a vehicle. Generally, a user should raise the jockey wheel into a stowed position before towing. Again, this will prevent inconvenience to the user or, in more serious cases, damage to the jockey wheel.

Various connections have different components. For example, small trailers (total maximum weight <750kgs) are not required to have a handbrake in Europe and thus rarely have handbrakes. In some embodiments, an initial assessment is made of what components exist on a considered connection. This initial assessment can be automatic, for example using a trained convolution neural network (CNN), and/or it can be manually performed. After this initial assessment, the state of the detected component is assessed.

In some embodiments, to improve user experience and efficiency, the system is configured to selectively activate or deactivate when relevant events are detected. The relevant events include at least one of changes in the component states, in the connection coupling state (coupled or not), and in the driving state (whether the vehicle is moving). The connection coupling state and driving state can be determined from the camera data, electrical signals, or other sensors on the vehicle or trailer. Selective activation or deactivation can be configured to occur a predetermined time after the relevant events are detected.

For example, the system can stop monitoring and deactivate if: the connection is coupling a trailer to a vehicle, the handbrake is disengaged, and the vehicle is moving. For further efficiency, the system may also only activate for a predetermined time-period unless alerts are issued. For example, the system may activate for a few seconds after a connection is coupled and the vehicle starts moving.

In one embodiment, the monitoring of the connection comprises the steps shown in figure 2. Firstly, an image from a vehicle mounted camera orientated toward a trailer is acquired. Part of this image shows a connection between a vehicle and a trailer. This part is extracted 210. A classification step 220 then follows. This classification step comprises two parts. In the first part, the extracted image is analysed to identify components of the connection, step 221. The identified components are then analysed to identify 222 the states of the components. The identified states are then recorded 230.

Optionally, the recorded states are tested 240 against expected predetermined states or with respect to each other. The expected predetermined states can be obtained from a database of expected states in various driving scenarios. Depending on the identified states or the result of the testing, an alarm may be issued 250. Alarms can be disabled or muted manually if a user is satisfied the components are correctly configured.

The flow diagram in figure 3 summarises processes that some embodiments of the system follow. In summary, firstly an image is acquired 310 from a vehicle mounted camera. As a trailer is typically connected to the rear of a vehicle, the camera that is used is typically one providing a rearward view of the vehicle. It does not matter whether this is a standalone rear-view camera, the rear camera of a surround view system, or a rearward mirror replacement camera (e.g. a camera that replaces the rear-view mirror). Fundamentally, any camera that has the requisite view of the connection can be used. Identification of the regions of interests associated with the components of the connection, within the image acquired from the camera, can be done in several ways. Preferably, to begin, a high-level region of interest, HL-ROI, is extracted 320 from the image, giving an image of the connection itself. The extracted part of the image does not need to be perfectly aligned to the connection but merely close enough to reduce irrelevant imagery to improve later processing efficiency.

There are several possible ways to obtain the HL-ROI. One way is to detect 320a the yaw angle of a connection, and the width and length of the connection using predominant vertical gradients. It is also possible to define the part of the image to be extracted using pre-defined values 320b. These pre-defined values can be supplied by a user, a trailer manufacturer, or obtained from a database. Several trailer specific HL-ROIs may be stored in the database from which a user may select. In some embodiments, there can be a mixture of camera-based detection and user selection. For example, the system may automatically detect the yaw angle of a connection and have the user enter the width and height of the connection. The user may, optionally adjust an HL-ROI if necessary. In some embodiments, the HL-ROI can also be evaluated 320c using a full-image based CNN. Regardless of the method used to determine the HL-ROI, it can be triggered or prompted to begin by a relevant event, such as coupling a trailer.

Figure 4 shows a part of an image 400 from a camera and the HL-ROI 401. The connection to the trailer is a tow-bar 405. The components of this tow-bar include a jockey wheel 402 and a handbrake 403, both of which are in the HL-ROI. In this case, the yaw angle of the tow bar 405 was automatically detected from the camera and used to orientate the HL-ROI appropriately.

Now that the HL-ROI has been found we return to figure 3 and proceed to step 330. This step 330 comprises detecting one or more regions of interest (ROIs) around the components of the connection. As with the HL-ROI, this can be done manually by the user and/or using predetermined values. In other embodiments, a machine learning approach is used, for example, a CNN or a support vector machine (SVM) may be used. If a machine learning approach is used for a component and fails or has low confidence in the output, the system may use predetermined values and/or prompt the user to enter the region of interest (ROI) for the component.

Figure 5 shows an image 500 of the HL-ROI extracted from the image 400 of figure 4, that has been processed to identify the ROI of the jockey wheel 502 and the ROI of the handbrake 503. A component ROI may also be called the bounding box of the component.

Once the component ROIs have been set, the system proceeds to determine the state of components. For most components, this is a binary assessment of whether they are engaged or disengaged. For example, the jockey wheel is engaged when it is deployed and disengaged when it is stowed, and the handbrake is engaged when it is preventing trailer movement and disengaged when it allows movement.

The state of the components can be determined in two ways. Firstly, an offline binary classifier can be trained. The ROI of the handle is passed 330a to the classifier which returns the state of “engaged” or “not engaged”.

Alternatively or additionally, to increase confidence, template extraction and comparison methods 330b may be used. As an example, when the vehicle is driving with the trailer attached, it can be assumed that the handbrake is disengaged (off) and the jockey wheel is disengaged (in a stowed position). Therefore, template images from the component ROIs when driving can be extracted and stored. The next time the vehicle starts driving, the current state of the component ROIs can be compared to the templates to confirm the component are in a driving state. There are many known techniques for comparing templates and images and these will not be described further here.

The state of the components is then recorded 340. Recording may further include sending the recorded states or a signal derived from the recorded states for storage or display and/or for use in another system. For example, if the recorded states indicate that the trailer is not ready to be towed (e.g. the handbrake is on), the system may signal this on the BUS which may alert the driver and/or immobilize the vehicle. In addition to the camera-based measurements of component states, electric signals may also be used to measure or confirm the component states. The electric signals may also determine the coupling of the connection or the states of other components that the camera cannot or does not measure.

In some embodiments, the electrical signal is taken into consideration when the system assesses if the identified component states justify issuing an alert to the user. As an example, when preparing to tow a trailer the user should attach an electrical lighting connector to the vehicle to enable the trailer lights. If the monitoring system detects at least one component is in state for towing and the electrical lighting connector is not connected it can provide this information to the user via an alert. Since the trailer lights are a key safety feature to warn other users of the user’s driving intention, the system can be configured to immobilize the vehicle by providing an output signal from the monitoring system to the BUS. As another example, the system can warn the driver if an electrical signal suggests the electric lighting connector is still attached but at least one other component is not in a towing state. This reduces the risk of the vehicle driving away without decoupling the electric lighting connector.

The alerts that are issued to the user can take many forms. They can be audio and/or visual alerts. As mentioned, they may also comprise a signal to affect the driving of the vehicle.

As shown in figure 6, in some cases, the driver of the vehicle may have a screen in the vehicle showing an image 600 of at least part of the connection to the trailer. The alert may be displayed on this screen. The system may be configured to indicate the one or more component of the connection that led to the alert being raised. This indication may be in text and/or it may indicate the ROI of the components.

In some embodiments, the alert can be manually overridden by the user. Thus, if the user is satisfied the components of the connection are correctly configured, they can continue. The previously mentioned machine learning algorithms including CNNs can be trained in known ways. Typically, training occurs in a training phase using labelled data. For example, when training the CNN to identify the HL-ROI, the CNN may be trained with images with the HL-ROI manually annotated. Similarly, images of connections with handbrakes that are labelled as engaged or disengaged can be used to train the binary classifier of the handbrake. As forming labelled data is labour intensive, it is possible to make additional training data by adjusting existing labelled data. For example, the labelled data can be processed to change the background and/or to adjust the image in various ways. This significantly increases the training data that is available, which makes the trained machine learning algorithms more accurate. Manually entered data or user adjustment can also be recorded and used to retrain and improve the machine-learning algorithms.