Method for obtaining calibration Data

Description

The present invention is directed to an apparatus for providing calibration data, in particular, for calibrating images from a plurality of cameras. The present invention is further directed to a camera system, to a method for obtaining calibration data and to a non-transitory storage medium having stored thereon a computer program. The present invention is further directed to a multi-camera rectification for planar and non-planar camera arrays.

Multi-camera systems are the base for modern visual effects like refocus or novel-view- synthesis as well as geometric analysis and depth reconstruction. An important requirement for high quality and high efficiency image processing is precise camera positioning and orientation. Due to mechanical deviations, like small deviations from an ideal orientation, this requirement is, in general, not met.

The genera! task of camera calibration is to estimate values for the internal (intrinsic) and external (extrinsic) parameters of a (digital) camera. The most important intrinsic parameters are focal length and principal point of the camera. Extrinsic parameters denote position and orientation of a camera. In the context of media production, camera calibration is, for example, required when one likes to add computer generated (CG) objects to the scene: When the physical camera is moving through the scene, the computer generated objects shall move consistently with the real world objects. Therefore, the path of the camera through the scene needs to be known precisely to control the path of a CG camera in a 3D environment. This process is also known as camera tracking or match-moving. In case of stereoscopic productions, camera images need to be aligned such that corresponding image points are in the same image line. Such a precise alignment is, in general, not possible by mechanical setups and needs to be done by digital image processing. This process is also known as rectification. A typical stereoscopic production usually requires two identical cameras precisely mounted on a rig. This initial, mechanical calibration ensures that camera images are already close to the rectified state. A more general approach for stereoscopic production is presented in [1]. In this work, three - -

cameras on a line are used to virtually manipulate the interocular distance of the stereo system. Therefore, the stereo calibration process is extended towards a three camera system. A widely used camera model is the pinhole camera. It is typically described by up to 1 1 individual parameters composed in the projection matrix P. These 1 parameters are focal length (horizontal and vertical), skew, principal point (horizontal and vertical), orientation comprising 3 angular components and location comprising three components of a 3D coordinate. This matrix may also be represented as a product of the camera matrix K, the rotation matrix R and the position C of the camera given as a 3D world coordinate. More parameters such as to model lens-distortion can be added if required yielding more than 1 1 parameters.

Estimating intrinsic and extrinsic parameters for a single camera or a system of cameras can be done using calibration charts or based purely on images. In the first case, usage of calibration charts allows the reconstruction of camera parameters in metric units; for example, a focal length of the camera may be given in a mm unit, i.e., millimeters. Such a calibration needs to be done right before or after capturing a shot on a set. The calibration chart is placed in front of the cameras and each camera captures an image of the chart. It is important that the physical properties such as focal length, orientation or position of a multi-camera system do not change between the calibration shot and the shot one wants to capture and process. Otherwise, calibration might be unreliable. However, many algorithms known for calibrating multi-camera systems require that the calibration chart is completely visible in every image. For large camera arrays, this is challenging.

The second case is widely known as self-calibration or auto-calibration and typically requires at least two images from a set of stereo cameras or two images from the same camera taken at different positions. The advantage of self-calibration compared to methods using calibration charts is that the cameras can be calibrated even if the captured content was not specifically intended for calibration. This is, for example, the case when the content was not intentionally shot for integration of CG elements but integrating CG elements is required in post-production. In contrast to chart-based methods, camera parameters can also be re-estimated if physical parameters change within a running capture. Self-calibration methods estimate camera parameters in non- metric units. As self-calibration algorithms exploit the correspondence between several images, such algorithms typically require that at least two or more images have overlapping image areas. Therefore, parts of the captured scene need to be visible in more than one image.

It is an object of the invention to provide a concept that allows for rectification of images of a camera pattern that allows for computational efficiency.

This object is achieved by an apparatus, a camera system, a method and a non-transitory storage medium having stored thereon a computer program according to the independent claims. Advantageous improvements are the subject matter of the dependent claim.

A recognition of the invention consists of having detected that, by determining displacement information that allows for aligning images along a first direction and along a second direction and by determining offset information between principal points of cameras of the camera pattern, the offset information may be determined with respect to pre-aligned images such that it is sufficient to determine the offset information along one of the two directions such that the computational effort for determining the offset information along the second direction may be saved which allows for computational efficiency. This further allows for a high precision and reliability of obtained image information such as offset information in 3D picture by having at least two independent, but correlating measurements.

Embodiments provide an apparatus that comprises a first interface for receiving a plurality of partially overlapping images of an object from a corresponding plurality of camera positions imaged by at least one camera, the camera positions being arranged along a first and a second direction and being arranged according to a camera pattern. The apparatus comprises an analyzing unit configured for selecting at least one corresponding reference point in an overlap area of a set of overlapping images, and for determining displacement information along the first and the second direction of the reference point in each of the other images of the set of images, i.e., a displacement of the reference point in the different images. A misalignment of the plurality of images along the first and the second direction is compensated for by the displacement information so as to obtain aligned images. The apparatus comprises a determining unit configured for determining offset information between principal points at the plurality of camera positions, using at least three aligned images. The apparatus further comprises a second interface for providing calibration data based on the displacement information and based on the offset information, the calibration data allowing for calibrating the plurality of images so as to comply to the camera pattern. In a multi-camera system that comprises more than two cameras, more stereoscopic views may be obtained such as between a first and a second camera and between a second and a third camera. Those two stereoscopic views may be brought into consistency based on the offset information. Based on the aligned images, it is sufficient to determine the offset information along the first or second direction which may be advantageous for at least three cameras arranged in a single row but which is especially advantageous for a two-dimensional or even three-dimensional camera pattern. This further allows for a high precision and reliability of obtained image information such as offset information in 3D picture by having at least two independent, but correlating measurements.

According to an embodiment, the analyzing unit is configured to determine the displacement information using a set of parameters indicating a real or actual condition of the at least one camera at the plurality of camera positions, the parameters comprising a non-linear relationship with respect to each other. The analyzing unit is configured to use a linearized version of the set of parameters and to determine the displacement information by minimizing an error of the linearized version with respect to a desired, e.g., error-free condition of the plurality of camera positions. Linearization allows expansion of a low amount of computational power. Based on the linearization, the displacement information along the first and second direction may, furthermore, be obtained one after the other that allows for a high robustness of the indetermination.

According to an embodiment, the analyzing unit is configured to iteratively minimize the error of the linearized version. This allows for a stepwise reduction and, therefore, for an efficient determination.

According to an embodiment, the analyzing unit is configured to minimize the error based on a determination rule that is based on a Jacobian matrix system that corresponds to a vector of constant values when being multiplied with the linearized version of the parameters. This allows for a further reduction of computational effort.

According to an embodiment, the analyzing unit is configured to determine the displacement information pair-wise for a pair of the plurality of images. This allows obtaining the displacement information pair-wise and, thus, while considering a low amount of parameters. This may be done repeatedly so as to obtain the displacement information of one image with respect to a different image. According to an embodiment, the analyzing unit is configured to first determine the displacement information along one of the first and the second direction and to subsequently determine the displacement information for the other direction. This allows for a high robustness of the result against computational errors and non-linearity as only a subset of unknown values is determined at an instance.

According to an embodiment, the analyzing unit is configured to determine the displacement information along the first direction independent from the displacement information along the second direction. This allows for a small subset of unknown variables that may be determined in a robust manner as errors along one direction may have no or low influence on determinations of the other direction.

According to an embodiment, the calibration data is based on angles that describe a rotation of the cameras, to a focal length of the camera and to a principal point of the cameras. The calibration data does not contain information indicating the plurality of camera positions. This allows for a low computational effort as the positions of the cameras may be considered as being constant and may thus remain uncomputed as variables.

According to an embodiment, the apparatus further comprises at least one camera being configured to provide the plurality of images from the plurality of camera positions. The apparatus is configured to determine a depth map of an object region comprising a plurality of sub regions. The at least one camera is configured to provide the corresponding image by projecting one of the plurality of sub regions. This allows for a self-calibration.

According to an embodiment, the apparatus is configured to apply the calibration data to the plurality of images so as to obtain a plurality of rectified images and to provide the plurality of rectified images. This allows for precise images.

According to an embodiment, a camera system comprises a at least one camera to provide images from the plurality of camera positions and comprises a memory having stored thereon calibration information derived from calibration data generated from an apparatus according to one of the previous claims. This allows for rectifying the images - -

obtained from the plurality of camera positions even if the calibration data is not determined for an actual image but has been pre-determined.

According to an embodiment, the camera system comprises an apparatus for providing calibration data. This allows for an apparatus that is able to self-calibrate.

According to an embodiment, a method for obtaining calibration data comprises receiving a plurality of partially overlapping images of an object from a corresponding plurality of camera positions being arranged along a first and a second direction according to a camera pattern. The method comprises selecting a reference point in an overlap area of one of the images and determining displacement information along the first and the second direction of the reference point in each of the other images, such that a misalignment of the plurality of images along the first and the second direction is compensated by the displacement information so as to obtain aligned images. The method comprises determining offset information between principal points of the at least one camera at the plurality of camera positions using at least three aligned images. The method comprises providing calibration data based on the displacement information and based on the offset information. According to an embodiment, a non-transitory storage medium has stored thereon a computer program having a program code for performing, when running on a computer, a method according to embodiments described herein.

Preferred embodiments of the present invention are described herein and make reference to the appended drawings in which:

Fig. 1 is a schematic block diagram of an apparatus according to an embodiment.

Fig. 2a is a schematic top view of a plurality of cameras before mechanical calibration;

Fig. 2b is a schematic top view of the cameras of Fig. 2a after mechanical calibration;

Fig. 2c is a schematic diagram that illustrates parameters of embodiments described herein;

Fig. 2d is a schematic perspective view of a camera that may be used in embodiments; Fig. 3a is a schematic diagram of a camera pattern according to an embodiment;

Fig. 3b is a schematic view of picture positions of the pattern that has been described in connection with Fig. 3a;

Fig. 3c is a schematic diagram of transformed images according to Fig. 3b;

Fig. 4a is a schematic perspective view of nine cameras that are arranged according to an embodiment;

Fig. 4b is a schematic view of example images arranged according to the pattern of Fig.

4a; Fig. 4c is a schematic perspective view of an example camera array that is arranged according to a 2x8 pattern;

Fig. 4d is a schematic representation of an example 3D camera pattern; Fig. 4e is a schematic top view of a part of the pattern of Fig. 4d;

Fig. 5 is a schematic flow chart of method for obtaining calibration data according to an embodiment; Fig. 6a is a schematic flow chart of another method for obtaining calibration data according to a further embodiment;

Fig. 6b is a schematic diagram of a block of Fig. 6a, that illustrates a linearization strategy that may be implemented by a analyzing unit according to an embodiment;

Fig. 6c is a schematic representation of a graph that represents triple-matching correspondence points according to an embodiment; Fig. 6d is a graph similar to Fig. 6c in an incomplete state; Fig. 7 is a schematic block diagram of an apparatus according to an embodiment that comprises a plurality of cameras; and

Fig. 8 is a schematic block diagram of a camera system according to an embodiment that comprises a plurality of cameras and comprises a memory having stored thereon calibration information.

Equal or equivalent elements or elements with equal or equivalent functionality are denoted in the following description by equal or equivalent reference numerals even if occurring in different figures.

In the following description, a plurality of details is set forth to provide a more thorough explanation of embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other instances, well known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring embodiments of the present invention. In addition, features of the different embodiments described hereinafter may be combined with each other, unless specifically noted otherwise. In the following, reference is made to camera arrays comprising a plurality of cameras that are arranged at different positions. Such arrays are described herein as being arranged according to a two-dimensional pattern for illustrating the advantages of the embodiments. Although referring to a two-dimensional pattern, other patterns may also be used that comprise cameras being arranged according to a three-dimensional pattern. For example, this may be considered when allowing for divergences of the focal length of a camera. According to other embodiments, the camera pattern may comprise only one dimension, i.e., the cameras are arranged along a single line (or row). Some embodiments are described as one camera being arranged at each camera position such that embodiments that relate to a plurality of partially overlapping images of an object being received from a corresponding plurality of camera positions may be understood as the plurality of images being received from a corresponding plurality of cameras. Without limitation one camera may capture two or more images at different locations when being moved from a location to another such that images from different camera positions may be captured by a single camera. Embodiments described herein are explained in terms of stepwise rectifying images. This may relate to rectify or transform images so as to obtain transformed images and to use such transformed images for further steps or for further processing. Without any limitation, such descriptions rate to correction data that is obtained stepwise, i.e., the original images may remain partially or completely unchanged, wherein a transformation of the images is only theoretically performed, e.g., when determining displacement data. Thus, rectifying or transforming the images may be equivalent to determining the amount of the transformation that has to be applied to the images. Embodiments described herein relate to a displacement of images with respect to each other and/or with respect to a camera pattern. The displacement relates to a transformation according to a first and/or second direction parallel to an object plane but also relates to a rotation of the image in the three-dimensional space. In the following, reference is made to rectifying images, i.e., to calibration of images. Such rectification may comprise a transformation of images. Such transformation may relate to a shifting or a movement of the image along at least one image direction, to a rotation of the image in the image plane and/or to a scaling (skew) of images. Thus and unless stated otherwise, reference made herein that relates to a shift of an image also relates to a rotation and/or a scaling of the image.

Fig. 1 is a schematic block diagram of an apparatus 10 according to an embodiment. The apparatus 10 comprises an input interface 12 for receiving a plurality of partially overlapping images of an object or object area 16. The images 14 are provided from a corresponding plurality of cameras 18a to 18c that project the object 16 by projecting sub areas 22a to 22c of the object 16 wherein each sub area 22a to 22c may correspond to one of the plurality of images 14. An overlap of the sub regions or sub areas 22a to 22c may correspond to an image overlap of the images 14. The plurality of cameras 18a to 8c is arranged according to a pattern along a first direction x and along a second direction y, for example, in an x/y plane. The apparatus 10 comprises an analyzing unit 24 configured for selecting one or more reference points 26 in an overlap area 28 of one of the images 14 which is illustrated, by non-limiting example only, for reference point 26 ₁ in an image 14a of the plurality of images 14. The image 14a overlaps with another image 14b of the plurality of images 14 in the overlap area 28, i.e., in the overlap area 28, both images 14a and 14b show the same part of the object 16. The reference point 26 contained as reference point 26 _! in the image 14a is also contained as reference point 26 ₂ in the image 14b. For example, the image 14a is captured by camera 18a and the image 14b is captured by camera 18b. Due to a misalignment between the cameras 18a and 18b, the reference points 26i and 26 ₂ are displaced by a displacement 32 along the x direction and/or y direction. The analyzing unit 24 is configured to determine the displacement information 34 that indicates the displacement 32 along the x direction and the y direction. The misalignment of the images 14a and 14b along the x direction and the y direction is compensated by applying the displacement information 34 which allows obtaining aligned images. Determining the displacement information 34 so as to obtain information on how to transform the images and allowing for low or a minimum of remaining errors for all of the considered images may be a first optimization problem. The analyzing unit 24 may be configured to solve the optimization problem, as will be described later.

Further, the analyzing unit is configured for receiving the images 14 from the plurality of cameras, e.g., without or only coarse prior knowledge with respect to the content thereof and/or overlaps of the images. The analyzing unit may be configured for selecting a corresponding reference point, i.e., image content in the overlap area, e.g., by comparing images. The corresponding reference point or reference region may be a single point or region/object in the captured scene. By identifying similar or identical content, the overlap area may be identified and a set of cameras (pairs, triplets, etc.) may be identified having overlaps in their images at least at a region comprising the reference point 26i and/or 26 ₂ . The set of images may thus be configured such that the set of images is a subset of the plurality of images 14, i.e., at least one of the images is not contained in the respective set. In contrast each camera is contained in one or more, e.g., at least two sets. Based thereon relationships between the cameras may be determined as being described in connection with Fig. 6c. The analyzing unit 24 may be configured for determining offsets or shifts or other deviations of the reference point starting from one image in one (pair of images) tow (triplet of image) or a higher number of other images. The analyzing unit 24 may be configured for determining further reference points for the same set of cameras and/or for other sets of cameras.

Further images of the plurality of images 14 may be misaligned with respect to the image 14a and/or the image 14b. The analyzing unit 24 is configured to determine the displacement, to determine the displacement information 34, respectively, between the plurality of images 14. The analyzing unit 24 may be configured to determine the misalignment or displacement globally, i.e., to select all images of the plurality of images 14 that illustrate the same reference point 26 and to determine the displacement information globally for the selected set of images. Alternatively, the analyzing unit 24 may be configured to align two images such as the images 14a and 14b with respect to each other, i.e., to operate pair-wise. The analyzing unit 24 may be configured to select a set of images that comprise the same reference point 26 and to pair-wise align the images with respect to each other. The analyzing unit 24 may be configured to select more than just one reference point in the overlap area such as, for example, at least 5, 10, or 100. For determining the displacement information along one of the first and second direction, a first image of a selected pair of images may be rotated and scaled so as to be aligned along one image direction, i.e., such as lines or columns of pixels point towards the other camera position. Then, the second image may be transformed along the respective direction so as to align the at least one selected reference point. This may be repeated for the other direction. The machine has knowledge about the position of all cameras, it is even assumed that this information is reliable.

The apparatus 10 comprises a determining unit 36 configured for determining a offset information 38 that relates to principal points 42a to 42c of the plurality of images 14. The offset information may thus be a distance information that describes a distance between principal points and may thus be a relative information relative with respect to the images associated to the principal points. Based on a physical distance between cameras (or camera positions) 18a and 18b or between cameras (or camera positions) 18a and 18c or between cameras (or camera positions) 18b and 18c, a view on the scene or object area, i.e., the object 16 is different from camera to camera. This effect is known as disparity and is used for determining 3D images. For a stereoscopic view, two cameras are required, for example, cameras 18a and 18b. Through the use of camera 18b and 18c, 18a and 18c, respectively, a second stereoscopic view may be obtained that may allow for an increase in information. When considering now a first stereoscopic view being obtained by using cameras 18a and 18b and a second stereoscopic view being obtained by using cameras 18b and 18c, the stereoscopic information may differ from one another when distances 44a between principal points 42a and 42b and a distance 44b between the focal points 42b and 42c differ from each other. Based thereon disparity estimation and/or depth reconstruction might lead to different results, which may be prevented by the described embodiments. The determining unit 36 is configured to determine the offset information 38 so as to comprise information related to the distances 44a and 44b. The determining unit 36 is - -

configured to use at least three images, for example, the images of the cameras 18a to 18c, for determining the offset information 38. For determining the distances 44a and/or 4b the reference points may be used. The respective reference point may be the identical in a pair of images and may be identical or different from each other in different sets of images, i.e., in a picture that is part of different pairs of images, different reference points may be used. Determining the offset information 38 so as to obtain information on how to transform the images and allowing for low or a minimum of remaining disparity offset for all of the considered images may be a second optimization problem. The determining unit 36 may be configured to solve the optimization problem, as will be described later.

The apparatus 10 comprises an interface 46 for providing calibration data 48 that is based on the displacement information 34 and based on the offset information 38. The calibration data may allow for rectifying the images 14 including a compensation in differences between distances 44a and 44b between the focal points 42a to 42c. The apparatus 10 may comprise an optional directing unit 52 being configured to process and/or to direct the calibration data 48. For example, the apparatus 10 may comprise an optional memory 54, wherein the directing unit 52 is configured to store the calibration data 38 in the memory 54. The directing unit 52 may be implemented by a central processing unit (CPU), by a microcontroller, by a field programmable gate array (FPGA) or the like.

Based on the displacement information 34, the apparatus 10 has knowledge about the displacement of the images 14 and may compensate for the displacement either by transforming such as shifting the images with respect to each other or by considering the displacement during further operation. This allows determining the differences in the distances 44a and 44b using aligned images such that the distance may sufficiently be calculated along one direction such as x for obtaining sufficient, precise information. An alignment of the images 14 may allow for reconstructing one dimensional dependencies in the camera pattern such as lines, rows or diagonals of the cameras 18a to 18c. The alignment performed by the analyzing unit 24 may, for example, allow for each of the images 14 being in-line or in-row of the camera pattern. This may leave unconsidered a distance between two lines of the pattern, or two rows of the pattern, respectively. However, by determining the distance 44a and the distance 44b, differences in the disparity between different stereoscopic images may be compensated for. Thus, by pre- aligning the images, the compensation for deviations in distances of principal points may be performed along one direction and may, at the same time, provide sufficient - -

information. Thus, disparity estimation and/or depth reconstruction as well as rectification of images may be performed with low computational effort.

Fig. 2a shows a schematic top view of the cameras 18a to 18c before mechanical calibration. Optical axes 54a to 54c of the cameras 18a to 18c differ from each other.

Fig. 2b shows a schematic top view of the cameras 18a to 18c after mechanical calibration. The optical axes 54a to 54c are essentially in parallel but may differ from each other by the displacement described in connection with Fig. 1. In addition, distances b _{x 2} between a position Ci of camera 18a and position C ₂ of camera 18b may be different when compared to a distance b _x2 3 between the position C ₂ and the position C ₃ of camera 18c, wherein the distances b _x12 and b _x23 relate to the x direction, d , C ₂ and C ₃ are positions of the cameras 18a, 18b and 18c in the three-dimensional world. In other words, Figs. 2a and 2b show the top view of an array from the top before and after (mechanical) calibration. Before the calibration orientation of cameras d, C ₂ and C ₃ are not consistent and not in parallel. The calibration rotates the cameras such that their optical axes are parallel and perpendicular to the base line of the cameras. Thus, Fig. 2a shows a top view of the camera array before calibration. The optical axes of the cameras are slightly different. Fig. 2b shows a top view of the camera array after calibration. The optical axes are in parallel. Both Figs. 2a and 2b may show only a top row of the camera array.

Fig. 2c shows a schematic diagram that illustrates parameters of embodiments described herein. The three dimensional world may be described by the x direction, the y direction and the z direction that may be arranged perpendicular to each other. A rotation by an angle a around the x direction may be referred to as "tilt". A rotation by an angle β around the y direction may be referred to as "yaw". A rotation by an angle γ around the z direction may be referred to as "roll". The illustrated parameters may be part of a vector that is optimized for obtaining rectified images.

Fig. 2d shows a schematic perspective view of a camera 18 that may be arranged in the array of cameras and may thus be camera 8a, 8b or 8c. The camera 8 comprises and image sensor 58 that comprises a center point 62. Properties of an imaging lens 64 or an array thereof may determine or at least influence a focal length f of camera 18. The focal length f may face an error a _f . Furthermore, the location of the lens 64 may at least - -

influence the location of the principal point 42 on the image sensor 58 that may be different from the center 62. The principal point 42 may describe a center point of the image that is projected, wherein the point 62 may be a center point of the image sensor 58. A divergence or distance between the projected sub region of the object area and the center of the image sensor 58 may be described by a distance p _x along the x direction and by a distance p _y along the y direction. The parameters p _x and p _y may be assumed to be almost zero for calibrated cameras so as to denote that the optical center of the camera coincides with the center 62 of the imaging sensor. When referring again to the model of the pinhole camera, the projection matrix P may be formulated according to:

P = K ^■ R ^■ [I I - C] Formula 1

K is, for example, a 3x3 matrix consisting of the intrinsic parameters. These include the focal length, the principal point τ and the skew factor s.

Fig. 3a shows a schematic diagram of an example of a camera pattern 30 according to which cameras 8a to 18d are arranged, i.e., the pattern 30 is a 2x2 pattern in the x/y plane. The camera array may be in a state that may be obtained after a mechanical calibration that is explained in connection with Fig. 2b. Camera 18a may be arranged at position d, camera 18b may be arranged at position C ₂ , camera 18c may be arranged at position C ₃ , and camera 18d may be arranged at position C ₄ , that are, in an error-free state, arranged according to a rectangle or a square. Other numbers of cameras and/or other patterns are possible. Parameters that may be evaluated for determining the displacement between images may be a focal length. In Fig. 3a, the focal length a _f1 of camera 18a is different from the focal length a _f3 of camera 18c which results in different sizes 66 ₁ and 66 ₃ along the x direction and/or different sizes 68 ₁ and 68 ₃ along the y direction of pictures 14a and 14c. Furthermore, picture 14b may be rotated by an angle y ₂ . Thus, the displacement information may relate to a transformation such as a shift, a rotation and/or a scaling of the images 4a to 14d along lateral directions x, y and/or z as well as to a rotation of images around at least one axis as described in connection with Fig. 2c.

Fig. 3b shows a schematic view of pictures 1 a to 14d of the 2x2 pattern that has been described in connection with Fig. 3a. On the left hand side, the images 14a to 14d are displaced; in particular, the image 14d does not match the intended pattern. On the right hand side a scenario is illustrated that may be obtained when compensating for the displacement. Arrows 12^ to 72 ₃ indicate a direction along which the images 14a to 14c have to be transformed including one or more of a shift, a rotation and a scaling, with respect to image 14d such that the reference points 26i to 26 ₄ correspond to the camera pattern. The arrows 12 _\ to 72 ₃ may also indicate a distance along which the respective image 14a to 14c has to be transformed along the directions x and/or y and may therefore indicate the displacement information. The black parts in the images 14a to 14d schematically illustrate regions from which the image has been transformed away. The analyzing unit 24 may be configured to determine the displacement information along the x direction and subsequently along the y direction or vice versa. When compared to a determining of the displacement along x direction and y direction simultaneously, this may provide for a high robustness. The apparatus may have knowledge about the error-free pattern 30 and may determine that lines 74a and 74b should be in parallel to each other and that columns 76a and 76b should also be in parallel to each other. The analyzing unit 24 may be configured to determine that this is not true and that the images have to be transformed with respect to each other as illustrated on the right hand side of Fig. 3b where the lines 74a and 74b and the columns 76a and 76b are each in parallel with each other. The determination of the displacement information may also be executed independently from each other, i.e., lines 74a and 74b may be brought into a parallel condition independent of columns 76a and 76b or vice versa. This may refer to a first direction that may be corrected independent from the second direction. The second direction such as the columns 76a and 76b when first having transformed the lines 74a and 74b is preferably executed dependent on the other direction so as to maintain the corrected images.

Thus, the left hand side of Fig. 3b shows an initial state before calibration but after mechanical calibration. The bottom right hand image is clearly not aligned with the other three images. On the right hand side of Fig. 3b rectified images are illustrated. Corresponding pixels are in the same row for horizontal adjacent views and the same column for vertical adjacent views.

In other words, the left hand side of Fig. 3b shows a 2x2 camera array in a general state. Camera 2 is slightly rotated and camera 3 has a slight deviation in focal length. Fig. 3c shows a schematic diagram of the transformed images 14a to 14d according to Fig. 3b on the right hand side. Based on different positions Ci to C ₄ of the cameras, the structure that is projected as reference points 26·, to 26 ₄ on the images 14a to 14d is arranged on laterally different positions within the images 14a to 14d. On images 14a and 14c, the reference point or reference structure 26i may comprise a distance to a first side of the image, wherein distances indicated with u may be, for example, in parallel with the x direction. In pictures 14b and 14d, reference points 26 ₂ and 26 ₄ may face a distance u ₂ that is different from to a comparable side of the image. The same effect may relate to images 14a and 14b with respect to a different perpendicular side. The reference points 26i and 26 ₂ may face a distance to that side, wherein the distance may be in parallel, for example, along the y direction. The reference points 26 ₃ and 26 ₄ of images 14c and 14d may face a different distance v ₃ to the comparable side.

An angle <p ₁₄ indicates an angle by which image 1 (24a) has to be rotated around an anchor point, possibly around a center 27i of the image 24a, such that its direction u or alternatively v at the anchor is directed towards the comparable anchor 27 ₄ in image 4 (24d). An angle cp ₂₄ indicates an angle by which image 2 (24b) has to be rotated around an anchor point, possibly around a center 27 ₂ of the image 24b, such that its direction u at the anchor is directed towards the comparable anchor 27 ₄ in image 4 (24d). Such angles may provide for information indicating the displacement information, φ may therefore be a parameter fixed for a pair of images or views (p,q) as it only depends on the mechanical center/position of cameras p and q. Centers 24i to 24 may also indicate the used camera position d to C ₄ . Fig. 4a shows a schematic perspective view of nine cameras 8a to 18i that are arranged according to a 3x3 pattern.

Fig. 4b shows a schematic view of the pattern 40, wherein the images 4a to 14i are arranged according to the pattern 40 of the cameras 18a to 18i in Fig. 4a. Distances b _x and b _y may differ from camera to camera which may be compensated for when determining the offset information.

In other words, Fig. 4b shows a schematic view of a 3x3 camera array from the back side. Cameras in the array are consecutively numbered starting at the top left corner with camera d going on to the right, to the end of the row, and then continuing on to the next row in the left-most column and so on. The last camera in the bottom right corner is consequently denoted C _MxN , M indicating the row an N indicating the column of the pattern.

Fig. 4c shows a schematic perspective view of a camera array 40' that is arranged according to an 2x8 pattern. Any other patterns comprising an arbitrary number of cameras may be implemented. The concept may be implemented for a number of three or more images. The at least three images may be acquired at different locations with at least one camera. As the rectification is done on an image-basis, also a single camera may be used that is moved between taking the pictures. The distance between the centers of the cameras or the principal points may then relate to the distance when having moved from position one to position two. Accordingly, to be able to take at least three pictures, also two cameras may be used, wherein at least one camera is moveable. Three images may also be acquired by three cameras such that for any number of images a corresponding number of cameras or a lower number may be used.

Although having described patterns that comprise a square or rectangular shape, any other shape may be realized. Any number of cameras such as at least three, five, nine or sixteen may be used. Thus, known approaches are extended towards two dimensional arrays of cameras, for example, such as shown in Figs. 4a and 4c. Such two dimensional systems may comprise a set of cameras but may also be mounted on a gantry that is able to move the camera to a specific position and take a picture.

Apparatus 10 implements a concept that transforms the images such that the remaining deviation compared to a given model is minimized. This may be achieved by approximating a highly non-linear problem by a series of sub-problems that may be of a linear nature. The overall problem may be approximated and solved using iterative minimization methods minimizing the residual error. Alternatively, the apparatus 10 may aiso be configured to solve a non-linear optimization. Fig. 4d shows a schematic representation of a 3D camera pattern 40" that is arranged, by non-limiting example, on a curved surface such as on a sphere. The pattern may also be considered to have lines 74a-d and/or columns 76a, wherein the lines and/or columns may be occupied different from each other. Alternatively or in addition, the lines and columns may be arranged in a non-perpendicular manner with respect to each other, along curved lines and/or columns or according to other rules. - -

Fig. 4e is a schematic top view of a part of the pattern 40" of Fig. 4d. Objects 77 and 79 may be part of a scene to be captured. The pairs of cameras 18b/18e and 18d/18e of line 74c may each have different direction of view and different distances to the objects 79a and 79b that are seen by both pairs which may lead to additional information required when compared to a 2D-pattern.

In other words, typically, the precision of mechanical setups is limited. Therefore, the exact properties of one or several cameras need to be reconstructed. Such a reconstruction of camera parameters is also known as calibration. In this procedure, different measurements are made and based on a physical or mathematical model, unknown parameters are found using the analyzing unit and/or the determining unit. Once those unknown parameters are found, one can use these parameters, for example, to define a function that transforms a set of non-rectified images into a set of rectified images. Thus, the obtained calibration data may be applied to non-rectified images which allows for storing or transmitting the non-rectified images even if the displacement information has already been obtained.

The focal length f of a camera may differ from its intended / ideal value in horizontal and/or vertical direction via the error a _f . The skew factor s may be used to describe the skew of a pixel. In the case of modern imaging sensors, this factor is typically 0, meaning that pixel cells are rectangular. The principal point describes the point of intersection between the camera's optical axis and the sensor. If the sensor is positioned exactly beyond the central point of the optical axis, τ _χ , deviation along the x direction and r _y , deviation along the y direction, both are zero. Therefore, the properties of a camera may be formulated as a matrix as:

Formula 2

By convention it may be formulated that, in a default state, an ideal camera is oriented such that it is strictly pointing down the z axis of a coordinate system such as the one illustrated in Fig. 2d. Image lines may be parallel with the x axis while image columns may be parallel with the y axis, respectively and by non-limiting example only. It should be understood that different coordinate systems may be used that may be transferred from one into another without limitation, in such a case, the rotation matrix may be the identity matrix I. This is not always the case and therefore the rotation matrix R is a general rotation matrix that is constructed from three angular components α, β and y, denoting rotations about the primary axes of the coordinate system as described in connection with Fig. 2c. The camera position C may be given as a three dimensional coordinate. The projected points or image points m, from a set of 3D world points Mi ... M _N may then be computed according to the determination rule:

Formula 3 In this case [M , lj ^T , denotes a homogenous 4D coordinate. The resulting image coordinate m, may then be a three-element vector. This point represents an image coordinate such that the vector may be normalized to its third component. After this normalization, u and v denote the horizontal/vertical position of a point in an image plane which may also be understood as distance (in pixels) from a side of the image.

For each vector m, the following rule may be applied:

Formula 4 In the context of multi-camera self-calibration, bundle adjust (BA), as described in [2], chapter 18, and which is also known as bundle-adjustment, is a known and proven algorithm. This algorithm uses iterative optimization methods like the Levenberg- Marquardt algorithm to minimize the following problem: Formula 5 wherein d(.) denotes the geometric distance between an image point x- and a re-projected point P ^l · Mj ^■ P ^! . P ⁱ denotes the estimated camera matrix for camera / ^' and Mj denotes an estimated world point. Variants of this algorithm are known that are designed to improve efficiency for specific applications and working points. In [3] a method is presented to calibrate a similar camera system consisting of 25 cameras in total. A well- known chart-based calibration method presented in [4] is used and the result is refined using global optimization. Other works as presented in [5], [6] and [7] use additional means like projected patterns or laser pointers to calibrate a set of cameras or require actively controlling the camera orientation or zoom. A calibration procedure as described in [8] whilst making references to [9] requires the user to provide a calibration object (such as a checkerboard) in front of the multi-camera system. Furthermore the system is limited to two-dimensional (2D) camera setups and the calibration object needs at least to be partially visible in all cameras at once. The properties of the calibration object need to be known to the algorithm. Thus, the calibration object cannot be selected arbitrary. Thus, in known systems, properties of the calibration object are known to the algorithm. I.e., the number of patches on the checkerboard or the fact that all patches are squares. This is directly exploited in the algorithm. The algorithm will fail if the calibration object cannot be detected properly. Camera systems such as the ones illustrated in Figs. 4a and 4c and schematically illustrated in Fig. 4b are arranged so as to share one common plane and their optical axis may be roughly, i.e., at most ± 15%, ± 10% or ± 5% deviation is possible, perpendicular to that plane. Additionally, all cameras form a rectangular grid. A grid may be understood as comprising lines 74 and columns 76 that may be, but do not need to be, perpendicular to each other. The apparatus 10 has knowledge about the physical layout of the camera setup, i.e., the analyzing unit 24 and the determining unit 36 have such knowledge. The model or knowledge includes knowledge about a camera position C and each camera's ideal orientation. The ideal orientation is thus a specified condition. The starting point of the calibration procedure that may be performed with apparatus 10 may be seen on the left hand side of Fig. 3b. It illustrates a subsystem of four cameras that is arranged on a 2x2 grid and captures a scene from slightly different positions. Corresponding image points, i.e., the reference points 26 _! to 26 ₄ , are connected with lines. A set of corresponding points denotes image points, regions or objects that refer to the same 3D scene point. Thus, even if defining reference points, embodiments are not limited to single points but refer to image regions or objects. In other words those individual image points correspond to a single scene point. This point / object is observed in two or more individual cameras. Corresponding points can be detected by means of feature detection and matching algorithms. Those algorithms assume that the projection of an object into different cameras results in similar image patches. Due to different distortions such as non-exact mechanical alignment, corresponding points typically form a non-regular quadrilateral. This is not very descriptive and establishing a simple relationship between those corresponding points 26 is intuitively difficult or impossible. Therefore, the images are transformed such that the connecting shape is rectangular for all corresponding points in the images. By aligning the reference points 26i to 26 ₄ , other selected reference points may also be aligned. An aligned version is illustrated in Fig. 3b on the right hand side. Once this requirement is met, further processing steps like stereo- matching may be executed efficiently.

Based on the pinhole camera model and the projection matrix P, the situation on the right hand side of Fig. 3b may be met if the cameras' focal length is identical and the optical axis of the cameras are parallel and perpendicular to the plane formed by the cameras' centers.

In the remaining part of the document subscripts as in x _p are used to indicate that a vector x denotes properties or information belonging to a specific element such as a camera p. If a variable encodes information belonging to a pair or tuple of cameras we denote this using a two-element subscript such as <p _pq . In this example cp _pq encodes an angular relationship between to cameras p and q. Similarly, relationship or information comprising three cameras is encoded using a three element subscript such as in A _pqr .

The calibration task can be formulated as measuring the deviation from its ideal state and finding a set of functions H(x _pi C _p , C _q ) and H(x _qi C _pi C _q ) that, given a pair of images, transforms each image point such that corresponding image points are located in the same image row. For example, matrix H is a 3x3 homography matrix. Mathematically, this may be expressed as: i H { x _p , C _v , C _q ) P _p M) ~ ii ( x _q , C _p , C _C{ ) P _r; - M ) - 0

Formula 6 v(H^ x _p , ^{( '} , C _q ) · m _p ) - v(H {\ _q , C _p , C _q ) ^■ m _q ) = 0

left view right view

Formula 7 M denotes a 3D world coordinate, P _p denotes the projection matrix for camera p. Typically, M as well as P are unknown. Assuming that the world point M can be seen in both cameras p and q, this point forms one pair of corresponding image points m _p and m _q . H(xp, C _p , C _q ) is a function of each camera's position C _p and C _g and several intrinsic parameters x _p . The row vector x may be formulated according to the rule:

x := [α β γ a _f p _x p _y κ]

Formula 8

H(x _p , Cp, C _q ) may be modeled using the pinhole camera model. The outer function v(.) is used to extract the vertical component of an image point. The equation given above may be implemented for all possible pairs of cameras for a given subset which is relevant for the further processing. Although Formulas 6 and 7 and other descriptions made herein differentiate between left view and right view, this may be changed with respect to each other or manipulated, without any limitation. When referring, for example, to Fig. 3b, x direction and y direction are illustrated as travelling from left to right or from bottom to top and therefore define positive and negative values of Formulas 6 and 7. By redefining the coordinate system such as when letting x travel from right to left, the left view and the right view are interchangeable. It is therefore important to differentiate between the views of a stereo setup in view of the coordinate system. For example, p may be related to the left view and q may be related to the right view or camera. In other words, camera q is to the right of camera p. For this reason, the ordering of the camera positions C is p, q in the left part of Formula 7 as well as in its right part.

The problem discussed above is typically valid for the case of a pinhole camera that does not suffer from side effects like lens-distortion. If lens-distortion is relevant for a specific camera and lens in an implementation Formula 7 may be extended:

v (H{X _v , Cp, C _q ) · undist(m _p , x _p † _j - v (Η{Χ _ψ C _p , C _q ) ^■ undist(m _pi x _q )J

Formula 9

UndistQ denotes a function that reverts lens-distortion according to some model and parameters contained in x as κ (Formula 8). UndistQ may be defined as: undistim, x)

Formula 10

L(r, K) ~ l + k ₁ - r + k ₂ - r ² + -

Formula 1 1

Function L(r, κ) actually models the distortion and is controlled by κ. Different models may be selected as well as a different length of parameter vector κ. Here, the discussion is limited to two elements in κ. In the ongoing part of this text, explicit elements in κ will be referred to as k such that κ may be expressed as: κ = [k ₁ , k ₂ , ... ] .

In principle, this problem may be solved using stereo calibration methods on each pair of cameras. In general, such a method will estimate a parameter set x _p , x _q for images p and q, but a different parameter set x _p , x _r for images p and r, wherein image r is located, for example, below picture p such as image 14c with respect to image 14a. In contrast, it is an object to obtain only one set of parameters x per camera. The desired calibrated situation may also be expressed using the camera matrices P. In the calibrated case, all cameras have identical camera matrices K. All cameras are orientated identical with their optical axis perpendicular to the camera baseline. In this situation, the camera matrix P only depends on the camera position C as described by:

P _p = K ^■ [I I - C _p ] Formula 12 In the case of independent stereo calibration, the obtained solution is slightly different and doesn't match all criteria: Self-calibration methods can only estimate a relative difference between two cameras. This is, of course, also true for the principal point. In case of a horizontal calibrated stereo rig with corresponding image points on identical image lines, both horizontal principal points r _Xi1 and τ _{χ 2} may be selected and manipulated arbitrarily. Both images still form a stereo system. This fact is, in example, also exploited in stereoscopic productions to manipulate the convergence plane.

In case of complex image processing such as multi-camera disparity estimation, one likes to ensure that corresponding image points across more than two views can be correlated by a simple relationship. Based on the simplified projection matrices given in Formula 9, one can deduce the following formula:

disp _1;i disp _1>3

^{Ι Γ} · ¹ ' ^{: « !} Formula e

Given a triplet of matching feature points, one can see that stereo disparity directly corresponds to the distance between the cameras' centers. The denominator of Formula 10 may be the distance 44 or b _x , b _y , respectively.

In a real setup, true camera positions will slightly differ from the given values C and known calibration methods are in general not applicable. However, the concept according to embodiments described herein assumes that the positional error is very small and can be - -

ignored which allows for a reduced space of parameters and therefore for a lower complexity of the problem. The concept according to the embodiments described herein may be applied but may slightly increase back-propagation error. It is therefore one aspect of the embodiments described herein to first align the images and afterwards determine the distance between the principal points by using a different number of images. For determining the displacement information, images may be used pair-wise, wherein the offset information may be determined using three cameras.

The displacement information as well as the offset information may be determined without user-interaction. For each pair of images / triplet of images robust corresponding points, i.e. , reference points, regions or objects are determined in an automatic manner, especially without a user providing special calibration patterns such as checkerboards or similar objects. In contrast to existing systems (such as [8]) it is in addition not required that a single (possibly known) object or point can be observed in all cameras at one time.

Thus, the number of cameras is adapted to the problem to be solved. Furthermore, the underlying non-linear optimization problem may be reformulated. The analyzing unit and/or the determining unit may be configured to solve the non-linear optimization problems. According to an embodiment, the analyzing unit and/or the determining unit may be configured to linearize the non-linear problem for some distinct points and to solve the linearized versions using standard iterative minimization algorithms. Instead of solving one large non-linear problem, such embodiments set up a list of local, piece-wise linear problems that are used to formulate the global problem. This global problem may then be solved iteratively using linear methods. When assuming that the mechanical design of the system is known, including camera position and ideal orientation, the calibration procedure may enforce camera views to obey this model. Additionally, the determination of the displacement information may be split into two parts. By this, highly correlated parameters leading to unstable results can be separated and optimized sequentially. Known approaches such as BA require estimating 3D world points from image correspondences. This is, in itself, a complex, erroneous procedure. In embodiments, 3D world points are not required, increasing the stability and efficiency of the approach. By example, BA optimizes camera parameters as well as position of 3D world points. The total number of degrees-of-freedom can be computed as 7xN + 3xP. In this case, N denotes the number of cameras while P denotes the number of 3D world points. According to embodiments, a focus is set on the intrinsic parameters and the camera rotation. This, initially, yields, in an example, number of degrees-of-freedom of 8xN. When referring again to Formula 8, the vector x comprises a number of 6 parameters. When formulating a problem based on the 6 parameters that rely on a single camera, the number of cameras Nx8 values per camera yield in the value above. Thus, embodiments described herein focus on obtaining a result, i.e., the displacement information and the offset information, that makes subsequent image processing steps efficient. In the first step, correspondences between image pairs are sufficient. The second step requires correspondences in three, possibly adjacent, views. This solution may spread an error in the camera array among all cameras, which is a specific property of embodiments described herein. Thus, an apparatus such as apparatus 10 may be configured to impinge on an error-free image with an error so as to compensate for inter-image errors. The concept is designed to fit a set of images to a given model, i.e., the camera pattern. The physical model, as well as the mapping between an image and its position in the array, is specified, by a user. Fig. 5 is a schematic flow chart of method 500 according to an embodiment that may be used to obtain calibration data. The method 500 comprises a set 510 in which a plurality of partially overlapping images of an object is received from a corresponding plurality of cameras being arranged along a first and a second direction according to a camera pattern. In step 520, a reference point is selected in an overlap area of one of the images, and displacement information along the first and the second direction of the reference point is determined in each of the images such that a misalignment of the plurality of images along the first and the second direction is compensated for by the displacement information so as to obtain aligned images. In step 530, offset information between principal points of the plurality of cameras is determined, using at least three aligned images. In step 540, calibration data is provided based on the displacement information and based on the offset information. The calibration data allows for calibrating the plurality of images so as to comply with the camera pattern when being applied to the images.

Fig. 6a is a schematic flow chart of another method for obtaining calibration data according to a further embodiment. The method may be organized in three sub-methods 602, 603 and 608. Block 602 may be considered to be an initialization. The initialization 602 may comprise steps 612, 614 and 616, wherein in step 612 L images are obtained, for example, from a L=MxN camera array with L=MxN cameras. In another exampie the L images are obtained from a sphere or cylinder like structure with L cameras. Two or more of the L cameras may be substituted by a single camera that is moved between different positions and configured to provide more than one picture. For simplicity, the method 600 - -

is first described for the special case of an L=MxN camera array with cameras mounted on a rectangular grid.

Sub-method 603 comprises block 604 that is a linear optimization procedure and block block 606 that is performed alternatively or in addition to block 604 and is a non-liner optimization procedure. Only one of the blocks 604 and 606 needs to be executed in order to obtain a set of calibration data 607. Block 608 generates, in a block 612, a set of calibrated images from the calibration data obtained in 607 so as to obtain L rectified images in a block 614 or stores the calibration data in a block 616 for future use. Method 600 allows for use of linear strategies as well as of non-linear strategies and is consistent for planar camera array and non-planar camera array.

In the following reference will be made to a use of a linear optimization, for example, in block 604 using steps 612, 614 and 616 of method 600. In the MxN camera array (M-1 )xN + (M-1 )xN, individual stereo pairs can be formed using directly adjacent camera positions. When referring, for example, to Fig. 4b, such pairs may be Ci/C ₂ , C^C _A , C ₂ /C ₃ , C ₂ /C ₅ , C ₃ /C ₆ , C4/C5, C4/C ₇ , C ₅ /C ₆ , Cs Ca, C ₅ /C ₉ , C ₇ /C ₈ , and C ₈ /C ₉ . I.e., the analyzing unit may be configured to determine the displacement information pair-wise for a pair of the plurality of images, e.g., direct neighbors. Alternatively, any other pairs of images may be used such as Ci/C ₅ , C-i/Cg, C ₂ /C ₆ , or the like. The apparatus is configured to select a sufficient number of pairs that allows determining a required number of parameters. Such a number of parameters may be, by non-limiting example only 6 as described above. Such a number is required for each camera such as 9 such that in total 54 parameters are required. Each camera is advantageously part of a number of different pairs of images that corresponds to the number of dimensions along which the camera pattern is arranged. A single line is considered as one-dimensional, camera positions (centers) differing in two directions are considered as two-dimensional and camera positions (centers) differing in three directions are considered as three-dimensional. For the present example, this is true for each camera being part of at least two pairs which is fulfilled for the list given above, wherein a number of 8 pairs (each of the 9 cameras may for a pair with 8 other cameras and has to be a part of two pairs equals in 2x8/2), wherein additional information may be obtained by additional pairs.

Although description was made in connection with a number of six parameters that are to be determined in order to calibrate images, also a different number of parameters or different parameters may be used, e.g., in case one or more of the above mentioned is considered to be within a tolerance range, calibration of that parameter may be skipped. Alternatively or in addition other parameters may be considered such as a distortion of lenses used in the cameras which may lead to low picture quality. The apparatus 10 may be configured to calibrate for the distortion, e.g., during determination of the displacement information. Different reference points in the image may allow for compensating the distortion of the lens.

In this way, each view is at least involved in two individual stereo pairs. The set of stereo pairs is further referenced as the set Ω. The set Ω may in example comprise pairs (p,q), (p,r), (q,r) and others. In general, the method is not limited to directly adjacent cameras. Each possible pair of views with overlapping image areas that allow for a common reference point may be formed. In step 614 interest points or reference are detected on each of the images and these interest points are matched on the mentioned stereo pairs in step 616. This may be done using any arbitrary feature detection and matching algorithm like SIFT, SURF or BRIEF. As a result, sets of corresponding, matched image coordinates

rrip, _q - between two images p and q is obtained.

Up and v _P denote vectors containing / horizontal/vertical positions of feature matches between view p and view q respectively. In the example of Fig. 3b, / would have dimension 1. If only one set of feature points belonging to a set of cameras is needed, this is denoted by an additional superscript in round braces. I.e., denotes all feature points in image p obtained from the pair (p,q). The third (homogenous) component is neglected here but is considered in further steps. In such a case, the third homogenous component may be set to a value of 1 .

In block 604 disparities along a direction, e.g., vertical disparities, for ail stereo pairs contained in Ω are reduced or minimized. For example, this may be done along the vertical direction such as the y direction. Formally, for one pair (p,q) this can be written as:

0 = V (H(X _P , C _P , C _Q ) ^■ m<¾) - v {E { _t C _p> C _Q ) · m¾)

Formula 14 - -

As before, H(x _p , C _p , C _q ) denotes a function returning a 3x3 homography matrix for view p. The argument x _p of this function is a vector of length k. Function v(x) extracts the vertical component of a feature point coordinate: v{x) = ~^ V i e I where x is a 3x/ matrix.

Consequentially, v(x) returns a vector of length /. In the present consideration x is a variable unrelated to other variables named x herein and shall simply denote an argument of function v. The overall minimization task can now be formulated as: argmin T 11 v (H(X _V , C _P , C _Q ) ^■ m¾) - v H(x _q> C _P , C _Q ) · m¾)

(P,q)en

« \ \x\ \ =∑ _{l Xl} ²

Formulas 15 & 16

The result thereof may allow the analyzing unit for obtaining a solution for all images p, q, r and s forming pairs, the pairs being comprised of the set Ω, i.e., a common solution for all x may be obtained.

In Formula 15 the square norm is selected in order to model the error between two views. Other choices are possible. The notation ∑( _Ρ(ί7 ) _εΩ \ \arg\ j indicates that the tuple (p,q) is one tuple exemplariiy selected from the set Ω but also that the summation is performed for all tuples contained in Ω. H(x _p , C _p , C _q ) is defined as a product of three functions R ₀ , K and

Rf.

H(X _v , Cp, Cq) = R ₀ · K · Ri = R ₀ · H{xp

with H{x _p ) = K - R _t

Formula 17 This formulation can be interpreted as follows: The right-most rotation matrix R, corrects a camera's orientation such that it corresponds to its ideal orientation as given by the ideal camera setup. The center matrix K corrects parameters like focal length and principal point. The left-most rotation R ₀ rotates a camera p such that it forms a stereo system with a camera q. Only this matrix is individual for each pair of views.

K may model the standard camera matrix as given above. R ₀ and R _t denote the outer and inner rotation matrix. More precisely, R ₀ is computed from the relative position and orientation of two cameras p and q. Because each camera position is known, R ₀ is always known. In some arrangements, all these cameras share one common plane, such that the outer rotation R ₀ can be modeled using a single rotation around the optical axis. This may - -

be referred to as angle φ that is explained in connection with Fig. 3c. As illustrated in Fig. 3a, a 2x2 camera array in a general state may comprise camera 18b that is slightly rotated by angle γ ₂ , wherein camera 18c has a slight deviation in focal length. For the pair d , C ₃ , ₀ ,13 needs to rotate d and C ₃ by - to form a rectified system assuming that no further distortion occurs. The angle φ may be computed with respect to left/right viewpoint. In the example of Fig. 3a, d is exactly above C ₃ . In order to be rectified to camera 8c and being the left view in this setup, d may be rotated by -90°. In contrast, C ₃ forms the right view and is consequently also rotated by -90°. This is also expressed in the argument ordering of H. The second argument denotes the left view while the third argument denotes the right view.

Alternatively, Formula 17 may also be defined as:

H(X _v , C _p , Cq) = R ₀ · Ri ^■ K = R ₀ ^■ ( p)

with H(x _p ) = R _r K

Formula 17b

In the more general case when the cameras do not share a common plane, the apparatus, in particular the analyzing unit is configured to determine R ₀ from the position Cp and C _q of two cameras p and q. R ₀ is a 3x3 matrix with elements as listed below:

Formula 18 - -

The parameter vector x _p contains all unknown parameters for a camera p and is defined according to Formula 8. Based on this definition H(x _pi C _p , C _q ) may now be completely defined as:

Formulas 19 & 20

The integer number subscript such as in C _p,2 denotes the second element in the vector C _p . Please note that in Formula 19 subscripts in the elements of x _p such as a have been neglected. It should however be clear that in any formula comprising elements of x, those elements need to be interpreted with respect to specific camera p. In Formula 19 the ideal orientation of all cameras is identical and perpendicular to the baseline. If this is not the case, H can be extended by an additional rotation matrix R(p,o, λ) describing the specific orientation with respect to the coordinate frame:

H O _p , C _p , C _q , p, σ, λ) = R _o (0,0, φ _ρΛ ) ^■ R (p, a, X ^{" ■} K {a _f , _Px , p _y )■ R^c , β, γ)

Formula 21 In the ongoing description the case expressed in Formula 14 is considered. If required, calculations can be redone including R(p,o,A).

With respect to the alternative definition of Formula 17 as given in Formula 17b, Formulae 19, 20 and 21 can be adapted by switching the order of the two rightmost matrices R, and K.

According to embodiments, the apparatus is configured to consider the lens-distortion. The functionality of the apparatus may be expressed by including Formula 10 in Formula 1 1 and Formula 12 as to obtain an undistorted version of reference points m.

The term of Formula 14 forms a non-linear system. This formula can be solved using iterative least-square minimization techniques like Levenberg-Marquardt algorithm or comparable methods. However, this solution is inefficient. This non-linear minimization is depicted in block 606 of Fig. 6a.

The problem in Formula 12 may be reformulated in order to find the values in H efficiently: Therefore, it may be assumed that all variables in x are small. This corresponds to the - -

block 604 in Fig. 6a. This block is depicted in more detail in Fig. 6b that illustrates a linearization strategy that may be implemented by the analyzing unit.

With the complete definition of H, Formula 12 may be evaluated for one pair of cameras p, q and for one pair of point correspondences.

In the next step such a formulation may be expanded, for example, using first order Taylor expansion of several variables. This expansion may be defined as:

f{x, a) := Tf(x, a) = TF _x>a (l,0) = F _X (0) + ¾ _e (0)

F _{x a} : m→ R := t→ f(a + t · (x - a))

Formula 22 & 23 In Formulae 21 and 22 F _XA is a function of t that evaluates the function f(x) at the point a + t · (x - a) . f(x) is the function to approximate at some point a. Parameters to the function f(x) are contained in the parameter vector x. t is a scalar value, a is a vector with identical size as x. First order Taylor approximation T of a function f(x) at some point a may be expressed as Taylor expansion of a scalar function F _XA (t) . F _{x a} is expanded at point t ₀ = 0 by computing the function value of F _X with t _Q = 0 and the value of the first order derivative F _X = F _XA ^ also at t ₀ = 0. Evaluating the resulting term at t = 1 yields the first order Taylor approximation of function f(x) at point a.

The expansion may exemplary be computed for a single pair of images p and q and a single pair of reference points. Then, Formula 16 (here also including lens-distortion as given in Formula 9) may be simplified as written in Formula 24.

Formula 24

The apparatus may be configured to determine the Taylor expansion using Formula 22 for Formula 24 for all variables contained in x _p and x _q forming a parameter vector χ like X = [½ _< ·½]■ X is argument to Formula 22 as well as an expansion point a = 0. Without limitation, other expansion points may be selected.

This yields an approximation for the point x _p =0 and x _q =0. The resulting equation can be differentiated with respect to every unknown variable in x _p and x _q . As a result, one obtains the Jacobian matrix of this system. Each column in the Jacobian matrix contains an - -

approximation for each parameter in the system. These approximations may also be expressed as:

In this case, k _{g, h} denotes the parameters to model lens-distortion of view g being either q or p according to Formula 10 and 1 1 and the definition of vector κ.

The remaining constant part may be formulated as:

Formulas 26

All elements in J and also the constant part b _p , _q can now be evaluated for point correspondences. The resulting system can be re-written as linear system according to:

^T X ^T

p,q = -b

PA

X ^T

Formulas 27 & 28

In Formulas 27 & 28, J _{p q} is a lx(2*m) matrix where I denotes the number of point correspondences and m denoting the number of variables in x. b is a column vector with length I.

As the Taylor expansion only approximates the function value at one point, the true parameter vector x _p needs to be replaced by an approximated value x _p . In Formula 27, - -

J _{p q} and b _p,q denote the Jacobian matrix and constant part of two views (p, q) evaluated for all point correspondences. As before, in Formula 28 denotes the columns of J _{P q} belonging to unknowns of camera p.

For the set of all stereo-pairs Ω, the overall minimization task may be formulated by the analyzing unit 24 as follows: Each pair (p, q) in Ω forms one line in the global constraint matrix A that is generated in a step 621 of method 600. For a 2x2 camera array with four individual stereo pairs, the problem can be written as:

A ^■ Δ = b

Formulas 29 & 30 The number of feature matches in each stereo pair may be greater than the number of unknown variables in x, such that the overall system is overdetermined. To solve the linearized system and thus finding a set of parameters approximating the real condition of the plurality of cameras as given by Formula 15, the system can be solved for Δ by computing the Pseudo-inverse of A. In total, the solution for Δ can be obtained as:

Δ = -{A ^{T ■} AT ¹ · A ^{T ■} b

Formula 31

Thus, the analyzing unit may be configured to minimize the error based on the determination rule that corresponds to Formula 31. The analyzing unit may be configured for determining the displacement information using a set of parameters, that indicates a real condition of the plurality of cameras. The parameters may comprise a non-linear relationship with respect to each other. The analyzing unit may be configured to determine a linearized version of the parameters in the vector Δ and may be configured to determine the displacement information by minimizing an error of the linearized version Δ with respect to a desired condition of the plurality of cameras. This is indicated as step 622, i.e., the system is solved for Δ. Alternatively, the analyzing unit may solve the non-linear problem. The at least two elements in x show the approximation for τ _χ and τ _γ . As both do not depend on image coordinates and only depend on φ which is constant, the linearization does not provide valuable information of these parameters. For this reason, both parameters τ _χ and z _y are excluded in this step. This works by dropping the value in - -

x _p and x _q as well as the appropriate columns in J _{p q} . As a consequence, deviations in a camera's principal point will be approximated by a small additional rotation around the y axis and x axis respectively. I.e., the calibration data may be based on angles that describe a rotation of the cameras, to a focal length of the camera and to a principle point of the cameras, wherein the calibration data may be in absence to a position of the cameras of the plurality of cameras, i.e., does possibly not contain information indicating the camera positions. This may be possible as the camera position is considered as known variable in the system. The elements in Δ now contain an approximated solution for the overall non-linear problem. At this point, the overall problem can now be solved using an iterative optimization algorithm like gradient descent. In every iteration a small portion of ε · Δ is added to the previous Xp ^t-1 forming a new i.e., the solution vector is updated in a step 623. The current x^ is then used to generate an intermediate mapping homography H for each camera. This intermediate homography is used to update all feature sets (p, q) 6 Ω. Those updated feature sets m _ViQ form new initial feature sets in the next iteration, i.e., the feature points are transformed in a step 624. Ideally, iteratively approaches the desired solution. As soon as Δ is smaller than threshold e ₀ or the remaining error cannot be reduced anymore, the algorithm stops. I.e., the analyzing unit may be configured to iteratively minimize the error of the linearized version. This is indicated in step 625 checkin if congruence is reached. This may also be expressed as:

and »i <¾ = ¾ · m¾

Formulas 32 & 33 Once this iterative minimization algorithm has converged, the second part of this linearized calibration procedure can be started, i.e., block 627 may be started, e.g., by the determining unit. At the end of block 625 a result according to Fig. 3c may be obtained, i.e., after the first rectification step, images may be pair-wise rectified. Corresponding pixels are shifted horizontally only for views in the same row and shifted vertically in the same column respectively. Both, the transformation of the images in step 622 by the analyzing unit and the estimation of the principal points in step 627 by the determining unit may be executed by determining a solution for a minimization problem that may be linear - -

or non-linear. I.E., four problems may be formulated, two for solving the minimization criteria with respect to the transformation that may be referred to a first (vertical) disparity and two for solving the second (horizontal) disparity. The problems may be re-formulated for a 2D and a 3D-pattern of cameras.

The final state after the converge criteria is this illustrated Fig. 3c for a 2x2 array. The position of the marked objects in C† is denoted by Ui and v In C ₂ , the object has slightly moved to the left but the vertical coordinate is still v _? . A similar condition holds for the vertical pair d, C ₃ : the object is shifted vertically but not horizontally.

As formally expressed in Formula 13, corresponding disparity values in three cameras may depend on the array geometry only. Up to this point, this has not been taken into account in the method 604. Consequentially, it is possible that Formula 13 is not met. Simplified, a distance between columns in pairs may differ or vary. This condition can be re-written in order to formulate a condition similar to Formula 15. For a triplet of three cameras this may be formulated as:

1

(u(R _o (0,Q, <p _Piq ) - 7h _q ) - u(R ₀ (Q,Q _{> (} p _M ) · m _q j)

- (u(fl _o (0,0, φ _ρ ) ^■ M _p ) - u(R _o (Q,0, φ _ρ ) ^■ = 0

with

m = [fi v If = H(x) ^■ m

Formulas 34 & 35 H denotes the mapping homography that has been obtained in the last iteration of the previous steps, i.e., block 625. H(.) may be defined as given in Formula 17 or 17b with respect to a parameter vector x. The function u(.) extracts the horizontal component of image point. d(C _p - C _q ) denotes the Euclidian distance between two camera C _p and C _q . Using in instead of m, it may be assumed that optical axes are parallel and that all cameras have identical focal lengths. The remaining error can be modeled by a shift in the cameras' principal point. Formula 34 can now be used to define a second minimization problem which is subject to be minimized and formulated as follows:

argmm (u (H(X _P , C _{V >} C _Q ) · m¾, _r ) - u (H(x _q , C _{P I} C _Q ) ^■ m¾, _r )) ^A

Xl,X > ^x 3- ^x N (H{X _P , C _P , C _r ) · m¾ - u (H(x _r , C _P , C _R ) · m¾, _r )) -~ _{c c} - -

Formula 36

Formula 36 refers to the general case when the cameras are not in a calibrated state. The analyzing unit may be configured to solve this problem so as to determine the displacement information. Alternatively or in addition, the analyzing unit may be configured to solve a linearized version thereof that may be expressed by

Formula 37

Function R(x _pi C _p , C _q ) may be defined as:

H(x _pf C _pi C _q ) = Η _ο (0,0, φ _ρ,η ) · , τ^, τ^)

χ = [τ _χ , fy]

Formula 38

Function R as given in Formula 38 models a transformation that performs a shift in a cameras' principal point τ and as before rotates two images such that they may form a stereo pair. Thus, by comparing two individual stereo pairs minimizing the remaining error as given in Formulas 36 and 37 by finding optimum values for f may be performed.

As an additional condition, Formulas 36 and 37 may be minimized with respect to the minimization criteria given in Formula 15. Being consistent to the definition of sets of feature point pairs m _p,q given before, m _{p q r} defines a set of triple matching feature points. All triplets (p, q, r) of cameras sharing overlapping image areas are contained in the set Ψ.

Formulas 36 and 37 may be evaluated for various camera configurations with the requirements as given above. Depending on the camera configuration and the resulting image overlap the set of equations that is obtained will have a different shape. It is possible that the resulting system can be solved exactly with linear methods, and is over or under determined. - -

In matrix notation, problem 37 may be written as follows: For one triplet of cameras (p,q,r) this may be formulated as:

[A% _r A _qr A^ _r } - [X _P X _q \ ^T + b _pqr = 0

A ^■ \x _p X _q X _r ] ^T + b _M r = 0

Formula 39 wherein

- cos(<p _pr ) · d _pq + cos((p _pq ) · d _pr sin(cp _pr ) · d _pq - sm(<p _pq ) · d _pr

dpq d _pr

^Ι ΡΨ ^' sin(<p _p(? )

sin(<p _pr ) sin(<p _pr )

0 0

Formula 40

^■

Formula 41

In matrix A the first line arises from Formula 36 evaluated for a triplet (p,q,r). Lines 2 to 4 may arise from the additional requirement as given by Formula 15. As transformed points in are used here, Formula 15 simplifies and only R ₀ (or the elements comprised in R ₀ ) remain.

For a camera pattern comprising more than 3 cameras, and so more than one camera- triplet in the set Ψ, each triplet forms one line in a global matrix A _G .

Formula 42

In Formula 42, the set Ψ comprises four triplets denoted as pqr, prs, qrs and prt. A solution for the equation system may then be found as:

Formula 43 - -

A critical setup needs to be discussed in the case when only three cameras or images are available. The presented strategy can easily be extended to such a scenario. Evaluating the sum of Formula 36 for the camera triplet (C _A , C ₂ , C ₃ ) as given in Fig. 3a yields:

Ui + τ _χΛ ~ u ₂ - τ _Χι2 -v - i _yA - v ₃

= 0

d(C _lt C ₂ ) d(C C ₃ )

Formula 44

Evaluating Formula 15 for the pairs (C _f C ₂ ), (C^ C ₃ ) and (C ₂ , C ₃ ) yields three additional conditions:

V _X + f _{V l} - V ₂ - fy _i2 = 0

-u _x -i _{x l} + u ₃ + τ _Χι3 = 0

-(u ₂ + v ₂ + i _Xi2 + Ty _iZ - u ₃ - v ₃ - t _X;3 - f _y>3 ) = 0

Formula 45 Rewritten in matric notation, Formula 39 becomes:

Formula 46

Formulas 44, 45 and 46 may be evaluated for one triplet-point correspondence m _i:i,3 = {u ₁ , v ₁ , u ₂ , v ₂ , u ₃ , v ₃ }. The resulting system has six unknown variables but only four formulas. It is therefore under-determined. Additionally, the matrix A does not depend on the values of u and v. This means that by adding additional triplet-point correspondences the matrix A cannot obtain full row rank. Any line added to A will only be a duplicate of an already existing line. Therefore, this system cannot be solved without further assumptions or without reducing the number of unknown variables. One possibility to reduce the number of unknowns is to define Cj as reference camera with τ _Χι1 = _y = 0. This allows removal of the first and second column of matrix A. The remaining linear system can then easily be solved using linear methods. This is shown by:

Formula 47 This strategy can be extended towards a general planar camera array. Depending on the field of view of each camera, their position and orientation, there may exist cameras that are unable to be formed into triplet matches over all possible camera combinations. It is, in general, therefore not possible to predict how many triplets and pairs can be formed. The amount of linear independent equations one obtains to solve for each principal point is unknown. This needs to be taken into account when setting up an array.

Alternatively, the linearization of Formula 36 may also be expressed as: argmin J [/¾, /JJJ, J + h p.q.r

Wherein the argument of Formula 36 is approximated

(u (H(X _P , C _V , C _Q ) · m¾. _r ) ~ u (H{x _q , C _P , C _Q ) · m¾,)) j^—^

- (« {" (Xp- Cp- Cr) ^■ m<¾ _r ) - u (H(x _r , C _p , C _r ) ^■ m¾, _r ))

Alternatively, the analyzing unit may be configured to determine the displacement information, to minimize the error respectively based on a non-iinear concept. This may lead to a slightly increase in computational effort but may in return allow for usage of common available implementations as in software libraries or the like. In case of non- linear optimization both problems as given in Formula 15 and Formula 36 are subject to be minimized with respect to vectors x.

Based on x _p , x _p , C _p and C _q and as part of as 612 and / or 616 the total calibrating function H being referred to as a homography matrix can be computed as: ll{x _v ,x _p , C _v , C _q ) R ₀ {Qfi, <p ) ^■ K _i (a _{f l} f _{x l} T _y ) - R _i (a,l Y)

view dependend constant

Formula 48

H(.) may thus be a function providing a homography matrix undistorting points according to a parameter vector x _p and camera positions C _p and C _q .

Formula 48 relates to the definition of H as given in Formula 17. Alternatively, the definition as given in Formula 17b may be used. - -

One can see that the right part is constant for a camera p while the left part depends on the relative position of both cameras. An application of the calibration data, i.e., the homography in step 612, allows for obtaining rectified, i.e., calibrated images in a step 614. In the special scenario of rectangular arrays as used as an example herein, this can be exploited to increase performance. In a rectangular system matrix, R ₀ is either the identity matrix or a rotation around 90°. It is therefore beneficial to apply the constant part once for an image and execute the second part on demand. Within a computer system, a rotation around 90° can easily be executed by swapping rows and columns of an image without further computations. In other words, Fig. 6 shows a schematic working principle of a calibration method. The left column covers initialization, the central column corrects focal length and orientation whereas the right column corrects deviations in the principal points. For the following calibration procedure, there may be assumptions that all cameras in the array are mechanical mounted on a strict, static grid. Therefore, it may be assumed that it has a good pre-calibration. This mechanical setup may be, in general, not aligned at pixel level and comprise slight deviations that remain in the images. As illustrated in Figs. 2a and 3b, the orientation of the cameras may differ as well as their focal length. Due to this mechanical pre-calibration, the camera position may be considered as known and pairs of cameras may be close to the rectified state. Therefore, the camera matrix K for a camera may be modelled as:

Formula 49 wherein f denotes a normalized focal length and may be set to 1. a _f denotes a small deviation of focal length. Furthermore, the focal length may be considered as constant in horizontal and vertical direction, i.e., s,≡ 0 which means that square pixels are present and a _f « f = 1. As the camera's principal point, one can further assume that τ _χ « 0 and T _y « 0. The orientation is modeled using a standard rotation matrix R(a,p,y) with a, β and Y denoting tilt, yaw and roll respectively. In this case, it may be assumed that |α|, |β|, |γ| « 1 , wherein all angles may be given in radiants). At the end of the calibration, deviations from the ideal model are known. - -

Details of above embodiments may relate to a 2D camera pattern. When determining calibration data for a 3D camera pattern, the same or at least related considerations may be taken into account.

In a general case when cameras do not reside on a common plane, i.e., form a 2D- pattern, Formula 13 cannot be applied and needs to be replaced by a more general formulation that fits to a 3D-pattern. This formulation is known as the Trifocal-Tensor and is presented in [2]. The basic equation is also expressed in Formula 45. The Formula states that given three corresponding image points m _p , m _q and m _r visible in three cameras the function results in a 3x3 zero matrix. This also requires that camera properties like position, orientation and intrinsics are known precisely.

3

l ^m q] _x ^m P · ^T (p,q,r),i ' [ ^m rlx = 0 _3x3

Formula 50

Formula 50 may be expanded as:

l ^m q] _x ^' ( ^U P ^{' T} (p,q,r),l + ^V p ^{' T} {p,q,r),2 + ^T (p,q,r),3) ^' rlx = 0 _3x3

Formula 51

In Formula 45 and Formula 46 [a] _x denotes a skew symmetric matrix corresponding to some vector a as:

Formula 52

The three elements Τ( _{ΜιΓ)ι1 ι} T _{(p q )} .2 and T( _P , _q , _r ), ₃ denote a representation of the Trifocal Tensor. Each element Τ _(ΜιΓ μ may be represented as a 3x3 matrix and its entries may be computed (according to [2]) from three projection matrices P _q , P _q and P _r . Based on the ideal layout and ideal extrinsic and intrinsic camera parameters it may be assumed that all projection matrices and consequentially all Tensor elements may be computed. Due to mechanical misalignment, measured points in images p, q and r will not obey this ideal setup and consequentially computing Formula 51 for some triplet of corresponding points and idealized Tensor elements may result in values different from a zero matrix.

As in the planar system Formula 46 may be modified as follows: - - ] _χ · Op · ¾,,r),i + v* · r _(p , _q , _r)(2 + r _(p , _(? , _r) , ₃ ) · [m = 0 _{3 3}

Formula 53 g(x _p , x _q , x _r ) := [/7(x _£ ,) · · ( j · ( _{ρΑιΐ ι1} + v* ^■ T _{iPiq 2} + Τ _(ΡιηΛ3 ) ^■ [H(x _r ) ^■ πι _γ ] _χ

Formula 54 wherein

m ^# = [ _u * v ^# lY = # O _p ) ^■ m

Formula 55 A calibrated image point m _p * may be obtained by transforming an uncalibrated image point m _p by a mapping homography as m _p = fi(x _p ) · m _p . H(x _p )may be again be constructed from a camera matrix K and a rotation matrix R as in Formula 17 or 17b with parameters contained in a parameter vector x. As before, an additional element to compensate for lens-distortion may be integrated in order to obtain calibrated image points m _p .

mJOp): = H(x _p ) ^■ undist(m _p , Xp)

Formula 56

Formula 54 may therefore also be represented as function of x _p , x _q and x _r :

g(x _p , x _q , x _r ) := [m«] _x ^■ (u* ^■ T _(M , _r + v* ^■ T _(Pj£Jir);2 + T _(P:qx)i3 ) · [m*] _¾

Formula 57

For a general set of cameras, the apparatus may be configured to solve two optimization problems simultaneously, wherein one of the optimization problems may be subject to be minimized according to: argmin

Formula 58

In Formula 53, Ψ may denote a set containing all sets of camera triplets relevant for analyzation. As before, it is an example target to find vectors x that simultaneously optimize Formula 15 and Formula 53, e.g., in a linearized case. The problem of formula 53 may be transferred to a linearized version and solved as such a linearized version, such as defined in formula 56. Again, the apparatus may be configured to execute a non-linear optimization method or a linearized optimization method. This is identical as presented for the planar case as depicted in Fig. 6a-b. The apparatus may be configured to use a non-linear optimization method so as to minimize the error. In case of a non-linear optimization method, the analyzing unit may be configured to optimize the results indicated in Formulae 15 and 58 simultaneously yielding optimum vectors x.

For the linearization, a local sub-problem consisting of three cameras p,q and r is evaluated for each element of Formula 57. This yields 9 functions of x _p , x _q and x _r . Each of these functions can be approximated using Taylor expansion as defined in Formulae 22 and 23. Formally, this can be expressed as:

Formula 59 & 60 The approximated elements g _it j may be composed to a vector and subsequently rewritten in matrix notation yielding a Jacobian matrix and a column vector b. As a result a linearized system is obtained with unknowns in χ. As before, a is a zero-vector with the same length as χ.

0 = [Τ 9 _1Λ (χ, α), Τ _9ΐι2 χ, α) T g _3,3 ( _X , a)] ^T

= J - _X + b

Formula 61 & 62

T gi ix, a) denotes the Taylor approximation of a function g and should not be mixed up with the Tensor elements in Formula 57. The indices in g denote an entry in the matrix returned by function g. (Compare also for Formula 53 and 54). However it should be clear the Tensors are elements of Formulae 58 to 62 and further on.

Alternatively, the analyzing unit may be configured to linearly minimize the error. In case of a linearized optimization method as shown in Fig. 6b Formula 57 may be linearized using first-order Taylor expansion resulting in a linearized approximation of the overall non-linear problem for some distinct, approximated point. As before, the obtained linearized equations may be represented as an Jacobian matrix J and a constant part b. This Jacobian matrix is composed from three submatrices as: Jp.qx Up,q,r Jp,q,r Jp.q.rl

Formula 63

In the same fashion as presented in Formula 29, the linearized global problem can be constructed from a series of local, linearized problems. Therefore, each camera triplet contained in Ψ may be represented so as to form one line in a matrix A _T and a vector b _T containing the constant part. The apparatus may be configured to form the Jacobian matrix accordingly, but according to other embodiments, it may form a matrix differently. In particular, the apparatus may be configured for determining an approximation for solving the first and second basis problems using the coordinates C of the cameras and of the reference points m on the basis of Jacobian matrices. The matrix A _T may then be concatenated with matrix A (further referenced to as A _B ) as obtained from Formula 29 for all stereo pairs (p,q) contained in the set Q . In the same fashion, vectors b (also referenced to as b _B ) and b _T may be concatenated.

Formula 64 b b

be. =

b _T

Formula 65

A _G · [x _p , x _q , x _r , ... ] ^T = -b _G

Formula 66

In a gradient-descend like manner the overall problem may be solved by solving A _G for a set of approximated vectors x and using a portion e of these values to update the true solution vector x as stated in Formula 32. This true solution vectors x may subsequently be used to update initial points m. With those updated points m matrix A _G may be recomputed. The process continues until congruence is reached.

In other words, the analyzing unit may be configured to solve a first and a second optimization criteria for determining the displacement information. In other words, the analyzing unit may be configured to solve a first and a second optimization criteria for the case of having a 2D camera pattern and for having a 3D camera pattern, wherein the criteria may differ between the 2D pattern and the 3D pattern.

For the 2D pattern, a possibly non-linear first minimization criteria, may be based on the determination rule in Y 11 v (//(x _p , C _PI C _Q ) · m¾) - v (Η(Χ _Ψ C _P , C _Q ) · m¾) with |x| \ =∑ _{l X}

(Formula 15) as described above. The possibly non-linear second minimization criteria may be based on the determination rule

argmin ^ II I" ( ^w (½< ^c v ^c q) ' « ¾ ^{~ u} (# (*q. C _p , C _q ) · m¾, _r ))— - - (u (H(X _p , C _p , C _r ) · m¾ _,r ) - u (tf(x _ri C _p , C _r ) · m¾, _r ))

(Formula 36)

as described above. For the 3D pattern, the possibly non-linear first minimization criteria may be based on the determination rule

argmin Y \ \ v (H(X _p , C _p , C _q ) ^■ m¾) - v (Η(Χ C _p , C _q ) ^■ m¾) with fx! \ =∑ _{l X}

(Formula 15)

and the possibly non-linear second minimization criteria may be based on the determination rule

(Formula 58) wherein

g{x _{pi Xq} , x _r ) [H(x _q ) ^■ m _q ] _x ^■ (u* · T _(Piq,rll + v* · T _{(M l2} + r _(Pi(?(r)>3 ) · [H(x _r ) ^■ πι _Ί ] _χ

(Formula 54) wherein

(Formula 55) wherein argmin denotes the minimization criteria, and wherein C _p , C _q denote camera positions of cameras p and q, d(C _p -C _q ) is a distance between the camera positions, H is a homography matrix, x _p , x _q and x _r are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, m _q , m _r and m _p,q refer to a respective minimization result of a previous minimization iteration, Ω and Ψ denote a set of images used for minimization, N is a number of parameters, v is an image direction, u* and v ^# denote a calibrated position of an image point along direction u and v obtained as m _p =H _p m _p and T, represents element i of a Trifocal Tensor. The explanation of the global optimization problem is represented in formula 58, wherein formula 58 may be built from (local) sub-problems given by formula 54.

Alternatively or in addition, the analyzing unit may be configured to solve linearized minimization criteria, i.e., to use a linearized version of the above mentioned criteria. The analyzing unit may thus be configured to determine the displacement information minimizing an error of a linearized first minimization criteria. Accordingly, the determining unit may be configured to determine the offset information minimizing an error of a linearized second minimization criteria. For the 2D camera pattern the first linearized minimization criteria may be based on the determination rule

(v (H{X _pi C _pi C _q ) · m¾ - v (H{x _q , C _pi C _q ) ^■ m^))" » J _PA + b PA

Form wherein this approximation rule relates to the non-linear minimization problem as: argmin v (Η(Χ _Ψ C _p , C _q ) ^■ m¾)|

+ b VA

. .

^— JB ' i ^x l> ^x 2> ^x 3> ·· ^■ I ^X NV + bf)

Formula 68 wherein

Formula 69 and

b _B = [b _pA b _Pir b _qiS b _TiS ...J ⁷

Formula 70 and for the 2D camera pattern the linearized second minimization criteria may be based on the determination rule

Formula 71 wherein this approximation rule relates to the non-linear minimization problem as: argmin (u (H{X _p , C _p , C _q ) ^■ m¾. _r ) - u (H(x _q , C _pi C _q )

- (« ("(½< Cp- Cr) ^■ m¾, _r ) - « (H(x _r . C _p> C _r ) ^■ m« _r )) ^ _C - -

argmi

Formula 72 wherein

Formula 73 and

b _B = [b _Pi q b _PiTiS b _{q s t} b _PiS>t ...J ¹

Formula 74

A solution may be determined, for example as:

[x ₁₍ x ₂ , xi- , ... , x _N f = -{Al ^■ AGY ¹ - A ^T _G - b _G

wherein

Formula 75

For the 3D camera pattern the linearized first minimization criteria may be based approximation rule

(v (H(X _P , C _p , C _q ) ^■ m¾) - v (HO,, C _p> C _q ) ^■ m¾)) « /. + h

Formula 76 wherein this approximation rule relates to the non-linear minimization problem as: argmin Y | j v (H(X _p , C _pt C _q ) · m¾) - v (H{x _qi C _p , C ) · m¾)

_Xl ,x ₂ ,x ₃ ..,x _N , _(p ^ _en i - -

« argmin + h PA

Formula 77 wherein

Formula 78 and

b _B = [ ^¾ p,<? ^b p,r bq,s b _r ,s -] ^T

Formula 79 wherein for the 3D camera pattern the linearized second minimization criteria is based on the approximation rule

= H(x _q ) ^■ m _q ] _x ^■ (u _p ^n■ T _(M>r)il + v* _p ^■ Τ _(Μ(Γ))2 + Τ _(ΜΛ3 )

Formula 80 wherein this approximation rule relates to the non-linear minimization lem as:

^— JT ' l ^x l> ^x 2> ^x 3< ··· > ½ + b _T

Formula 81 - -

wherein

Formula 82 and

b _T — [bp^q bp,s,t ^■ ]'

Formula 83

A solution may be determined, for example as:

[x x ₂ , x-i, ... , x _N Y = -(Ac' · A _c ) Al

Formula 84 wherein

Formula 85

2 Elements in J _Pi q, _r and b _{p q r} are formed from Taylor approximation of g{x _p ,x _q , x _r ) according to Formula 58 to Formula 62, and wherein argmin denotes the minimization criteria, and wherein C _P , C _q denote camera positions of cameras p and q, d(C _p -C _q ) is a distance between the camera positions, H is a homography matrix, x _P , and x _Q are vectors with linearized parameters to be determined so as to minimizing the error for camera p and q with respect to a constant b _q,p , J denotes a Jacobian matrix T represents a Trifocal Tensor. Subscripts p, q, r, s and t denote indices referring to a specific camera.

Thus, the apparatus may be configured to linearly and/or non-linearly performing optimization, wherein the selection is up to an implementation. Non-linear methods may allow for using a standard library, wherein the non-linear method may result in a higher computationai effort.

The first optimization problem or criteria may be solved by the analyzing unit so as to pre- align the images. The second optimization problem or criteria may be solved by the determining unit so as to reduce or compensate for remaining differences in disparities. - -

According to an embodiment, only a coarse setup of the camera system is known to the algorithm and the exact amount of overlapping image regions will be approximated or estimated. The analyzing unit is therefore configured to automatically detect corresponding points in all possible pairs and triplets of images.

A minimum requirement for proper execution according to an example is shown in Fig. 6c. The example refers to an 3x3 example setup of Fig. 4b. Accordingly, the given explanation does not limit the embodiments of the present invention. Fig. 6c shows a schematic representation of a graph that represents triple-matching correspondence points and, by way of example, comprises nodes N,. Each node index i relates to a triplet of cameras Ci , wherein the numbers displayed identify a respective camera of the camera array 40. For example, node Ni represents a corresponding point 1 which is captured by cameras d, C ₂ and C ₃ . Each triplet Ni to N _a is characterized in having a with respect to other triplets possibly individual corresponding part of the respective image, i.e., an overlap in the images.

Edges ΕΞ, _> between Nodes N, and _j of the graph and the corresponding labeling indicate cameras that are incorporated in both sets of triplet matches. For the sake of simplicity, Ey = Eji. For example, cameras 1 & 2 are both incorporated in the triplet set {1 , 2, 3} corresponding to node and the set {1 , 2, 4} corresponding to node N ₂ . Thus, cameras Ci and C ₂ have a (first) overlap containing a (first) correspondence point/object with camera C ₃ and have a second overlap containing a (second) correspondence point/object with camera C ₄ . Thereby, a correlation between camera C ₃ and camera C ₄ may be determined with the analyzing unit even if cameras C ₃ and C ₄ would not have a common overlap.

It is possible but not required and even unlikely that a common corresponding point is found in each pair of images. However, the embodiments described herein are not limited to a corresponding point being common in each of the cameras or pictures thereof. It is sufficient that each camera that is considered in the minimization problem is comprised in at least one pair of cameras contained in the set Ω and/or in at least one triplet of cameras contained in Ψ. I.e., it may be sufficient, that for each pair and/or triplet a possibly individual corresponding point may be identified. - -

The analyzing unit may be configured for determining the triplets N, from the pictures taken with the Cameras C, by identifying correspondence points in the respective pictures. The analyzing unit may be configured for the determining the triplets and/or the graph 60 or a representation thereof such that the graph 60 is fully connected.

As shown in Fig. 6c, all cameras (Ci , C ₉ ) incorporated in the system are represented in at least one node and the graph is fully connected. In other words there is a path from any node/triplet to any other node/triplet directly or indirectly by possibly crossing other nodes.

In contrast Fig. 6d shows a similar graph 60' but with the nodes N ₈ {5,6,8} and N ₉ {5,6,9} being removed such that node N ₈ is unconnected to the rest of the graph. For example, the determination of the analyzing unit is incomplete at the illustrated state. As can be seen, the node N ₈ {6,8,9} is not connected to any other node though all cameras (d ...C ₉ ) are represented in the graph. In this case the graph is not fully connected. Hence it is not guaranteed that the condition as specified in Formula 36 or Formula 58 is met. The feature-detection & matching step in the analyzing unit is implemented to build-up a fully-connected graph or a data representation thereof as shown in Fig. 6c. I.e., it is not necessary that the analyzing unit or the determining unit generates a graph or image thereof. It may be sufficient to generate a data representation thereof, e.g., a matrix or tabular containing the respective information.

A similar condition may also to be met for pairs of cameras. However any set comprising 3 cameras can be seen as 3 individual pairs of cameras. Therefore any camera that is incorporated in a triplet set is also incorporated in at least 2 pairs of correspondence points. Thus, the explanation given directly relates to 2D and to 3D camera patterns. The example given in Fig. 6c and Fig. 6d may be referred to as a minimum case. The apparatus may increase its precision by determining additional pairs or triplets as long as every camera is represented in the graph and the graph is fully connected. For improving the results, the apparatus may be configured for searching for a higher number of reference points. - -

The analyzing unit may determine the reference points so as to be contained in pairs and/or triplets of images.

According to embodiments, the analyzing unit selects or determines reference/correspondence points in overlapping image regions of the pairs and/or triplets. It is possible that not every possible pair of images overlaps. Further, the analyzing unit may be configure for searching or identifying at least one or up to a specific number of reference points, e.g., 2, 3, 5, 10 or more. When the specific number is reached, the analyzing unit may be configured for stopping the search and to thereby determining less than a possible number of reference points in the overlap area. This allows saving time and computational effort. Further, it may be sufficient to determine a specific reference point in the images of cameras of a pair or triplet only, even if the same reference point is also seen in further images. When compared to known concepts, it is therefore neither required to have a known reference object, nor to have a specific object to be contained in every image. It is of advantage to search for corresponding content/points in pairs and/or triplets of images as, e.g., at spherical or cylindrical arrays it becomes increasingly unlikely to have an object being imaged with more than three cameras when increasing the number of cameras of the array.

Fig. 7 shows a schematic block diagram of an apparatus 70 according to an embodiment that comprises, for example, apparatus 10 and comprises the plurality of cameras. The apparatus 70 is configured to apply the calibration data to the plurality of images 14 so as to obtain a plurality of rectified, i.e., calibrated images 82. For example, the apparatus 70 may comprise a processor 84 for applying the calibration data 48. The processor 84 may be configured to combine the calibration data 48 with the images 14. The processor 84 may be, for example, the element 52 of Fig. 1. Thus, not only calibration data may be provided but also rectified images may be provided. Alternatively or in addition, the apparatus 70 may be configured to determine a depth map of an object region such as the object 16. As described in connection with Fig. 1 , the object region may comprise a plurality of sub regions, wherein each camera 18a to 18c of the plurality of cameras may be configured to provide the corresponding image by projecting one of the plurality of sub regions and to provide an image based thereon. - -

Fig. 8 shows a schematic block diagram of a camera system 80 that comprises a plurality of cameras 18a to 18c and comprises a memory 86 having stored thereon the calibration information 48', being derived from the calibration data 48. For example, the calibration information 48' may be the calibration data 48 or may incorporate at least part thereof. The apparatus 80 may be configured to obtain a plurality of rectified images 82 and to provide the plurality of rectified images 82, e.g., at an output port of apparatus 80 or to store them in the memory 86.

The camera system 80 may further comprise an apparatus 10 that is configured to provide for the calibration information 48'.

Some explanations given herein relate to mathematical expressions of actions implemented by an apparatus and/or a method. It is to be pointed out that such expressions are used for explanatory reasons only so as to clearly define what an apparatus according to the respective embodiment does. Therefore, the embodiments do not relate to the mathematical expressions but to the actions implemented by the embodiments.

Some embodiments are described in connection with a 2D camera pattern, wherein other embodiments are described in connection with 3D camera patterns. Further, some embodiments are described in connection with solving a possibly non-linear optimization, wherein other embodiments are described in connection with solving a linearized versions thereof. Although being described as separate solutions, further embodiments may also provide for combinations. E.g., an analyzing unit may receive information whether the camera pattern is of 2D type or 3D type and may perform different actions responsive to said information. Further, the analyzing unit may be blind for said pattern-information and may perform both, 2D determination and 3D determination and may determine a result of higher quality and may take that result. Further, the analyzing unit may perform a 3D optimization and may take this result even if the pattern is a 2D pattern because the result is also valid.

Accordingly, the analyzing unit may implement one or both of the non-linear and linear approach, e.g., by first solving the linearized formulation and in case of having insufficient results solving the non-linear formulation. - -

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods - -

described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

- -

Literature

Ziily F., Riechert C, Muller M., Waizenegger W., Kauff P., Determination of rectifying homographies for a camera array, EP2917895 A1 , 2015

Hartley R., Zissermann A., Multiple View Geometry, Cambridge University Press,

2003

Wilburn B., Joshi N., Vaish V., Talvala E., Antunez E., Barth A., Adams A., Levoy M., Horowitz M., High Performance Imaging Using Large Camera Arrays, Proc. of ACM SIGGRAPH 2005, Vol. 24, No. 3, pp. 765-776, 2005

Zhang Z., A flexible new technique for camera calibration, Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(1 1 ), 1330-1334

Aerts M., Tytgat D., Macq J., Lievens S., Method and arrangement for multi- camera calibration, EP 2 375 376 B1 , 20 3

Zilly F., Method for the automated analysis, control and correction of stereoscopic distortions and parameters for 3D-TV applications, TU Berlin, 2015, http://dx.doi.org/10.14279/depositonce-4618

Zilly F., Riechert C, Muller M., Waizenegger W., Sikora T. , Kauff P., Multi-camera rectification using linearized trifocal tensor, 21st International Conference on Pattern Recongition (ICPR), 2012

Kurillo, G., Baker, H., Li, Z., & Bajcsy, R. (2013, June). Geometric and color calibration of multiview panoramic cameras for life-size 3D immersive video. In 3D Vision-3DV 2013, 2013 International Conference on (pp. 374-381 ). IEEE.

Li, Z., Baker, H., Kurillo, G., & Bajcsy, R. (2010). Projective epipolar rectification for a linear multi-imager array. In 3DPVT.