The present invention relates to a method for estimating a three-dimensional position of an object, wherein a computing system is used to receive two-dimensional image data generated by a non-rectilinear camera and depicting the object and for each of at least two characteristic points of the object, determine respective two-dimensional positions based on the two-dimensional image data. The invention is further related to a method for controlling a vehicle, to a system for estimating a three-dimensional position of an object, to an electronic vehicle guidance system and to a computer program product.

For various applications, in particular autonomous driving applications, it may be important to estimate the position of an object, in particular in an environment of a vehicle, in 3D. In particular, it may be necessary to calculate a distance between the vehicle and the object for such applications. However, a single 2D-camera can provide the position information of the object only in two dimensions, which may be insufficient. The situation may be even more complicated in case a non-rectilinear camera, such as a fisheye camera, is used, which is common in automotive applications due to their extremely wide field of view. The 2D-information provided by such cameras do in general not correspond exactly to distances or positions in the real world. It is therefore an object of the present invention to provide an improved concept for estimating a three-dimensional position of an object by means of a camera, in particular by means of a single 2D-camera, in particular by means of a non-rectilinear camera.

This object is solved by the subject matter of the independent claims. Further implementations and preferred embodiments are subject matter of the dependent claims.

The improved concept is based on the idea to identify characteristic points of the object and transform respective two-dimensional positions of the characteristic points obtained from camera data of a non-rectilinear camera in a way as if the characteristic points would have been imaged by a rectilinear camera. According to the improved concept, a method for estimating a three-dimensional position of an object is provided. According to the method, a computing system is used to receive two-dimensional image data generated by a non-rectilinear camera, wherein the two- dimensional image data depict the object. The computing system is used to determine respective two-dimensional positions for each of at least two characteristic points of the object based on the two-dimensional image data. The computing system is used to perform a coordinate transformation, in particular a predefined coordinate transformation, of the two-dimensional positions of the at least two characteristic points, wherein the coordinate transformation is designed to transform an image captured by the non- rectilinear camera into a virtual image of a virtual rectilinear camera. The computing system is used to determine three-dimensional positions of the at least two characteristic points based on the transformed two-dimensional positions and based on a predefined three-dimensional model of the object.

The three-dimensional positions of the at least two characteristic points can be considered to represent the three-dimensional position of the object.

A non-rectilinear camera can be understood as a camera with a non-rectilinear lens unit. A non-rectilinear lens unit can be understood as a lens unit, that is one or more lenses, having a non-rectilinear, also denoted as curvilinear, mapping function. In particular, fisheye cameras represent non-rectilinear cameras.

The mapping function of the lens unit can be understood as a function r(0) mapping an angle Q from the optical axis of the lens unit to a radial shift r out of the image center. The function depends parametrically on the focal length f of the lens unit.

For example, a rectilinear lens unit has a gnomonic mapping function, in particular r(0) = f tan(0). In other words, a rectilinear lens unit maps straight lines in the real world to straight lines in the image, at least up to lens imperfections.

A non-rectilinear or curvilinear lens unit does, in general, not map straight lines to straight lines in the image. In particular, the mapping function of a non-rectilinear camera can be stereographic, equidistant, equisolid angle or orthographic. Other examples for mapping functions of non-rectilinear lens units are polynomial functions.

The generation of the two-dimensional image data using the non-rectilinear camera is not necessarily but can be a part of the method according to the improved concept. To receive the two-dimensional image data, the computing system may receive a respective camera signal directly from the camera or may receive the image data from a storage device.

The image data can be understood, for example, to represent a raster image or pixel image, in particular a two-dimensional array of pixels. In particular, the image data do not contain any direct distance information regarding a distance of the object from the camera. In other words, the non-rectilinear camera is designed as a 2D-camera.

The two-dimensional positions of the at least two characteristic points can therefore be understood as corresponding positions in the pixel array, or, in other words, as two- dimensional coordinates in a camera coordinate system, wherein the camera coordinate system is, in particular, rigidly connected to the non-rectilinear camera.

That the transformation is designed to transform the image into the virtual image can be understood such that, if, hypothetically, the image captured by the non-rectilinear camera would be transformed according to the transformation, the result of the transformation would correspond to a virtual image or, in other words, an image of the virtual rectilinear camera. In other words, the transformed image corresponds to an image that would result from depicting the object by means of the virtual rectilinear camera. In other words, all straight lines in the real environment, in particular of the object, correspond to straight lines in the transformed image, in particular up to mapping errors. However, the method according to the improved concept does not actually require that the complete image or the full set of two-dimensional image data generated by the non- rectilinear camera is transformed according to the coordinate transformation. Rather, it is sufficient to transform only the two-dimensional positions of the at least two characteristic points. This may significantly reduce the required computational effort.

The three-dimensional model can be understood as a set of relations, in particular distances, between the at least two characteristic points, in particular between mutual pairs of the at least two characteristic points, in three dimensions. The relations or distances reproduce approximately the actual positions of the at least two characteristic points on the real object, that is the object in the real world. If, in addition to the three-dimensional model, the transformed two-dimensional positions of the at least two characteristic points are known, the three-dimensional positions of the at least two characteristic points can be determined.

Therefore, according to the improved concept, the three-dimensional model, which is based on real physical distances of the characteristic points of the object, can be used to estimate the three-dimensional positions from the two-dimensional image data, even though the camera is a non-rectilinear camera. If, on the other hand, the three- dimensional positions would be directly estimated from the two-dimensional image data in combination with the three-dimensional model, significant errors would result. Therefore, the improved concept improves accuracy of the estimation.

According to several implementations of the method, the computing system comprises one or more computing units of the camera and/or external to the camera, for example of a vehicle. The one or more computing units of the camera may comprise circuitry of an imager chip of the camera and/or an image signal processor, ISP, of the camera and/or a graphics processing unit of the camera. The one or more computing units of the vehicle may comprise one or more electronic control units, ECUs, of the vehicle.

According to several implementations, the computing system is used to apply a perspective-n-point algorithm, in particular to the transformed two-dimensional positions, to determine the three-dimensional positions of the at least two characteristic points based on the transformed two-dimensional positions and the three-dimensional model.

Such implementations allow to efficiently determine the three-dimensional positions of the at least two characteristic points based on the transformed two-dimensional positions and the three-dimensional model.

A perspective-n-point algorithm can be understood as an algorithm solving the mathematical problem of estimating the pose of a calibrated camera, given a set of n 3D- points in the real world and their corresponding 2D-projections in the image. While such algorithms are known in principle, they cannot be applied to image data obtained by a non-rectilinear camera without introducing significant inaccuracies. Such inaccuracies would render the result useless for automated driving applications. According to several implementations, the computing unit is used to apply an image processing algorithm to the two-dimensional image data to determine the two-dimensional positions.

The image processing algorithm may for example be based on a computer vision algorithm or a machine vision algorithm. A computer vision or machine vision algorithm may for example be understood as an image processing algorithm, which has been trained by means of machine learning, in particular as an image processing algorithm based on a trained artificial neural network, in particular a convolutional neural network, CNN.

The at least two characteristic points of the object may, for example, comprise corner points of geometric figures, for example polygons, which approximate the object according to the three-dimensional model.

Image processing algorithms, in particular such based on neural networks, have proven very effective and efficient to determine such characteristic points based on two- dimensional image data.

According to several implementations, the computing system is used to apply a further image processing algorithm to the two-dimensional image data to characterize the object and to select the three-dimensional model of the object from a plurality of predefined models based on a result of the further image processing algorithm.

In particular, by characterizing the object, the object may be identified or a type of the object may be determined. The type of the object may be more or less general. For example, the object type may be define the object as a vehicle, a person, a further object, for example a van, a limousine, a station wagon, a bus, an adult person, a child, a traffic sign, traffic lights, lane markings, lane boundaries and so forth.

Once the identity or the type of the object is known, the computing system can select the appropriate model to describe the object approximately.

The further image processing algorithm may be a part of the image processing algorithm or may be implemented separately. In particular, the further image processing algorithm may also be based on computer vision or machine vision and/or may be based on a neural network, such as a CNN.

In particular, the further image processing algorithm may be designed as an object detection or object classification algorithm.

According to several implementations, the non-rectilinear camera is used to generate the two-dimensional image data, in particular by capturing an image of the object.

According to several implementations, the computing unit is used to determine the three- dimensional positions by detecting respective three-dimensional coordinates in a vehicle coordinate system of a vehicle.

Therein, the vehicle coordinate system is, for example, rigidly connected to the vehicle.

The non-rectilinear camera may be mounted on or in the vehicle. Consequently, the vehicle coordinate system may also be rigidly connected to the camera.

In such implementations, the three-dimensional positions of the at least two characteristic points contain information relative to a position of the vehicle or the non-rectilinear camera, respectively. Such information may particularly suitable for controlling the vehicle.

According to several implementations, the coordinate transformation depends on, in particular is determined by the computing system depending on, a mapping function of the lens unit of the non-rectilinear camera.

The lens unit comprises one or more optical lenses combined such that they feature the mapping function as a common mapping function.

If the mapping function of the lens unit is known, the computing system can obtain the virtual image from the image of the non-rectilinear camera or, in particular, the transformed two-dimensional positions from the original two-dimensional positions of the at least two characteristic points. According to several implementations, the computing system is used to determine a distance of the object from the non-rectilinear camera depending on the three- dimensional positions.

In case the non-rectilinear camera is mounted in or on the vehicle, the distance between the object and non-rectilinear camera translates directly to a distance of the object from the vehicle. This distance may represent important input information for systems to control the vehicle, in particular for advanced driver assistance systems or systems supporting automated driving.

However, two-dimensional cameras cannot reliably determine the distance per se. The improved concept provides means to nevertheless estimate the distance based on the two-dimensional image data in combination with the three-dimensional model.

According to the improved concept, a method for controlling a vehicle at least in part automatically is provided. Therein, a non-rectilinear camera mounted on or in the vehicle is used to generate two-dimensional image data depicting an object in an environment of the vehicle. Three-dimensional positions of at least two characteristic points of the object are determined by carrying out a method for estimating the three-dimensional position of an object according to the improved concept. A control unit of the vehicle, in particular of the computing system, is used to control the vehicle at least in part automatically depending on the determined three-dimensional positions, in particular depending on the distance of the object from the non-rectilinear camera.

The computing system or one or more individual computing units of the computing system may be comprises by the vehicle and/or the camera.

The control unit may be comprised by the computing system.

Controlling the vehicle at least in part automatically may, in particular, comprise controlling a longitudinal velocity, a lateral velocity or a lateral position of the vehicle, controlling a steering system of the vehicle, a braking system of the vehicle and so forth.

According to the improved concept, a position estimation system for estimating a three- dimensional position of an object is provided. The position estimation system comprises a computing system configured to receive two-dimensional image data generated by a non- rectilinear camera, which depicts an object, in particular in an environment of the position estimation system, for example of the non-rectilinear camera. The computing system is configured to determine, for each of at least two characteristic points of the object, respective two-dimensional positions based on the two-dimensional image data and to perform a coordinate transformation of the two-dimensional positions of the at least two characteristic points, wherein the coordinate transformation is designed to transform an image captured by the non-rectilinear camera into a virtual image of a virtual rectilinear camera. The computing system is configured to determine three-dimensional positions of the at least two characteristic points based on the transformed two-dimensional positions and based on a predefined three-dimensional model of the object.

According to several implementations of the position estimation system, the computing system is configured to apply a perspective-n-point algorithm to determine the three- dimensional positions of the at least two characteristic points.

According to several implementations, the position estimation system comprises the non- rectilinear camera.

According to several implementations, the non-rectilinear camera comprises a lens unit with a mapping function, wherein the coordinate transformation depends on the mapping function.

Further implementations of the position estimation system according to the improved concept follow directly from the various implementations of the method for estimating a three-dimensional position according to the improved concept and the various implementations of the method for controlling a vehicle according to the improved concept and vice versa, respectively. In particular, a position estimation system according to the improved concept may be configured to or programmed to perform a method according to the improved concept or the position estimation system performs such a method.

According to the improved concept, also an electronic vehicle guidance system comprising a position estimation system according to the improved concept is provided. Therein, the computing system is configured to determine the three-dimensional positions by determining respective three-dimensional coordinates in a vehicle coordinate system of a vehicle.

An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle fully automatically or fully autonomously and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle conducts required steering maneuvers, braking maneuvers and/or acceleration maneuvers and so forth automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated June 2018.

According to several implementations of the electronic vehicle guidance system, the computing system is configured to control the vehicle at least in part automatically depending on the determined on the three-dimensional positions, in particular depending on the distance between the object and the non-rectilinear camera.

According to the improved concept, also a computer program comprising instructions is provided. When the instructions are executed or the computer program is executed, respectively, by a computer system, in particular by a position estimation system according to the improved concept, for example by the computing system of the position estimation system, the instructions cause the computer system to carry out a method according to the improved concept.

According to the improved concept, also a computer readable storage medium is provided, wherein the computer readable storage medium stores a computer program according to the improved concept.

The computer program product as well as the computer readable storage medium according to the improved concept may be considered as respective computer program products comprising the instructions.

Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which do not have all of the features of an originally formulated independent claim. Moreover, implementations and feature combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the feature combinations set out in the relations of the claims. In the figures,

Fig. 1 shows schematically a vehicle with an exemplary implementation of an electronic vehicle guidance system according to the improved concept;

Fig. 2 shows schematically an image captured by a non-rectilinear camera;

Fig. 3 shows schematically a transformation of the image of Fig. 2; and

Fig. 4 shows schematically an exemplary three-dimensional model of an object.

Fig. 1 shows a vehicle 7 and an object 1 in an environment of the vehicle 7.

The vehicle 7 comprises an electronic vehicle guidance system 9 according to an exemplary implementation of the improved concept. The electronic vehicle guidance system 9 comprises a position estimation system 8 according to an exemplary implementation of the improved concept.

The position estimation system 8 comprises a non-rectilinear camera, which may be designed as a fisheye camera 3, and a computing system 2.

The computing system 2 may comprise one or more ECUs and/or other processing units of the vehicle 7 and/or one or more processing circuits of the fisheye camera 3, for example circuitry of an imager of the fisheye camera 3, an ISP of the fisheye camera 3 and/or a graphics processor of the fisheye camera 3. The fisheye camera 3 comprises a fisheye lens 3’ with a non-rectilinear mapping function. The object 1 is, in particular, located within a field of view of the fisheye camera 3.

The vehicle 7, in particular the electronic vehicle guidance system 9, for example the position estimation system 8, may comprise a control unit 2’ which may, for example, be comprised by the computing system 2.

The electronic vehicle guidance system 9 is, in particular, configured to perform a method for controlling the vehicle 7 at least in part automatically according to the improved concept. To this end, the position estimation system 8 may for example be configured to perform an exemplary implementation of a method for estimating a three-dimensional position of the object 1 according to the improved concept.

For example, the fisheye camera 3 may capture an image 5 of the environment of the vehicle including the object 1 . Such an image 5 is shown schematically in Fig. 2.

Since the mapping function of the fisheye lens 3’ is a non-rectilinear mapping function, straight lines in the real environment of the vehicle 7 are in general not mapped to straight lines in the image 5, as can be seen in Fig. 2.

In Fig. 2, also several characteristic points 4 of the object 1 are indicated in the image 5.

In the example of Fig. 2, the object 1 is for example a van and the characteristic points 4 may comprise specific corner points of polygons approximately describing the van.

The computing system 2 applies an image processing algorithm to the image 5 to identify the characteristic points 4 and to determine their two-dimensional positions.

The computing system 2, in particular a memory unit of the computing system 2, stores a projection, for example, a projection function or a projection matrix, which would transform the image 5 into a virtual image of a virtual rectilinear camera. Therein, the virtual image 5’ would look as if it would have been captured by a rectilinear camera, in particular the virtual rectilinear camera. Flowever, the projection corresponds to a coordinate transformation. In particular, the virtual rectilinear camera does not actually exist and is not actually used to capture any image.

For the method according to the improved concept, however, the computing unit 2 does not necessarily transform the whole image 5 according to the coordinate transformation. Rather, it is sufficient to perform the coordinate transformation of the two-dimensional positions of the characteristic points 4.

Fig. 3 shows schematically the virtual image 5’ resulting from transforming the whole image 5 according to the coordinate transformation. In addition, Fig. 3 shows the characteristic points 4 at their respective transformed two-dimensional positions.

In Fig. 4 a three-dimensional model 6 of the object 1 is depicted schematically.

The model 6 comprises distances of mutual pairs of characteristic points 4 in three dimensions, such that the object 1 is approximated by the model 6.

In particular, for a plurality of pairs of the characteristic points 4, the model 6 comprises respective three-dimensional distance vectors. The model 6 is, in particular, stored on the memory of the computing system 2.

The computing system 2 may apply a perspective-n-point algorithm to the transformed two-dimensional positions of the characteristic points 4 using the model 6 in order to determine the three-dimensional positions of the characteristic points 4.

Optionally, the computing system 2 may determine a distance between the object 1 and the fisheye camera 3 or between the object 1 and the vehicle 7, respectively, based on the three-dimensional positions of the characteristic points 4.

The control unit 2’ may receive the three-dimensional positions and/or the distance and control the vehicle 7 at least in part automatically based on the three-dimensional positions of the characteristic points 4 and/or the distance.

As described, the improved concept provides a means to estimate three-dimensional positions of an object based on camera data generated by a single non-rectilinear 20- camera.

In particular, according to the improved concept, neither more than one 2D-cameras are required to retrieve depth information from stereographic algorithms nor a 3D-camera, a lidar sensor system or a radar sensor system. In particular, the improved concept enables the usage of perspective-n-point algorithm to the case of non-rectilinear cameras.

The improved concept may for example be considered as containing an approach to simulate a perspective or rectilinear camera around the object of interest without necessarily generating corresponding perspective images.

In some implementations, first a transformation is applied to the fisheye image to simulate an image from the virtual rectilinear camera looking at the object in order to apply the perspective-n-point algorithms afterwards. The virtual rectilinear camera is associated to a corresponding projection matrix. If the characteristic points are projected or the model is projected accordingly, the perspective-n-point algorithm may be used to find the three- dimensional positions of the characteristic points. In practice, there is no need to generate the full perspective image, which would be computationally expensive. Only the two-dimensional positions of the characteristic points in the virtual perspective image have to be determined. This may be done “on-the-fly” for every object in the field of view of the non-rectilinear camera.