FIELD OF THE INVENTION

The invention relates to the field of electronic processing of images (photographs) and specifically - to the enhancement of the electronic image resolution by using the data of several overlapping low-resolution images.

DESCRIPTION OF RELATED ART

This invention provides the possibility to create automatically an image of enhanced resolution from several partially overlapping low-resolution images captured at different positions and angles of cameras. During the process the camera positions and the spatial surface, which is used for the interrelation between the low-resolution images, are reconstructed from the initial low-resolution images. After geometric corrections of initial low-resolution images, the intensity of high-resolution pixels is calculated based on the projections of spatial surface points rendered by them in low-resolution images.

Below is considered the state of the art, i.e., information, which is publicly available at the moment.

There is known the patent US8582922B2 (published on 12 November 2013), in which the method for the image resolution enhancement using two or more low- resolution images is presented. Method for the resolution enhancement is based on the calculation of the high-resolution pixel values of the low-resolution pixels using a principle of weighted interpolation calculation. This calculation method makes no reference about the images that were captured at different camera rotation angles in respect of the object, three-dimensional surface is not applied to enhance the resolution, therefore, images where the object is rendered from one plane or planes, which are only at few-degree angles to the said object plane, can be used for such a method. There is also known the patent application US20040170340A1 (published on 2 September 2004), which presents the method of image resolution enhancement, where the image of enhanced resolution is obtained from at least two lower-resolution images using a Bayesian approach. However, such resolution enhancement method can be used only when the orientation of the original low-resolution images and their turning at an object are very close to each other.

There is also known the document US20030012457 (published on 12 January 2003), where a flat higher resolution image is formed from multiple low-resolution images of panels. The basic principle of this method is that all information is collected into a single image and later unnecessary details are cropped according to a certain pattern.

In most cases, at the technical level documents most often associated with image resolution enhancement techniques that are oriented to computing and further processing of multiple low-resolution images applying different techniques are found. Also, the documents usually consider images restored in two-dimensional plane, which coincides with the image plane. Because the faster development of image processing and composition technologies as a science and application trend started only in the last decade (especially in the last years), so it can be said that at the moment image processing technologies are going through a breakthrough. This is due to the fact that the current computer hardware is able to process quickly large amounts of information without oversimplification of the original images, to say, in pursuance of better results it allowed to work with images in a complex way by applying various sophisticated methodologies.

The vast majority of image resolution enhancement techniques in the current technical level are based on the reconstruction of small changes in the position of the camera (shift and turn). Search for positions is carried out in 2D space coinciding with the image plane. Once the optimum positions have been found the increased resolution has been calculated for the entire area of the restored image at the same time. However, in case of major changes in camera position and locking direction, homography-based methods do not work because the images can no longer overlap due to the prospective transformation of objects captured in them at a different angle. The method described in the present invention performs entry of images during the photogrammetry process. With the use of the key-points and 3D model bundle adjustment the image interrelationships and camera positions in 3D space are found. Due to a different locking position of photographs the objects captured in each overlapping image are visible in different places of the image, i.e., it is not possible to carry out the increase of the resolution for the entire image at once, because the images can not overlap even after the necessary transformations. This problem is solved by calculating each pixel of increased resolution separately. Sequence of steps applied in the present invention restores all points of higher resolution by finding to each of them a respective point on the 3D surface, marking a local surface plane in it and projecting this small square area to all low- resolution images where this 3D surface location is locked. According to the obtained projections, in each selected image are found the intensities of the projection points, which are then used for the conclusion of the system of equations. Having solved the system of equations, the obtained central environmental intensity of the investigation point is saved in an image of increased resolution.

SUMMARY OF THE INVENTION

In the course of 3D image conversion, the 3D surface of a captured object (the earth surface, buildings, crowns of trees, etc.) used for finding of image projections captured in photographs is restored from the initial low-resolution images. Reconstruction of higher resolution images is calculated using the overlapping low-resolution areas of photographs. Higher resolution photographs are obtained following all the processing steps. These photos correspond to the locking position and angle of the original photos (i.e. they are contain the same images) but has a higher image resolution so the image captured in them becomes more detailed. The image will be more detailed in those areas where the greater number of photographs overlapped.

When forming a higher-resolution image three main stages are applied:

In the first stage, applying the method of photogrammetry a 3D surface model captured in photographs is found: positions and turning angle of cameras are restored, as well as 3D mesh of the object captured in photographs is calculated. In the second phase, the radial undistortion of the lens for all low-resolution images is performed. The size of the correction required is determined using the geometric distortion coefficients obtained during photogrammetry.

In the third stage, the calculation of the intensities of higher resolution pixels is carried out. To this end, for each new higher-resolution point a 3D point located on a 3D mesh of the captured object is found. When projecting the 3D point to all low-resolution images, the projection point is found. A quadrangular area of the point environment, the angles of which are also projected to all low-resolution images, is marked around the 3D point. Only those low-resolution images, where the wanted point is well distinguished, are selected for further calculations. From the low-resolution points located in the projections of the 3D point environments an equation system is formed, by resolving which the wanted intensity of the increased resolution point is found.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1. Overlap of (a) the low-resolution, (b) the high-resolution, (c) the high- and low-resolution environment points of the wanted point environment.

Fig. 2. Environment of the wanted point in different low-resolution images.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

For the purposes of this description the term "resolution" means a parameter of an image reproduced in electronic equipment and describing the amount of image forming points within a length/width/space unit.

For the purposes of this description the term "pixel" or "point" refers to a square/cubic area of the image occupied by the point.

For the purposes of this description the term "pixel intensity" or "intensity" is understood as the pixel brightness (for monochrome images) or as the brightness of the original colours of the pixel colour (red, green and blue) (for colour images). For the purposes of this description the term "low-resolution image" or "lower- resolution image" means a photograph of an object, where the captured object is clearly visible but fine details of the object are rendered abstractly.

For the purposes of this description the term "high-resolution image" or "enhanced resolution image" means an improved photograph of the same object, in which fine details of the object are rendered in greater detail (eg., you can read the note, which was not possible to read in the original low-resolution images).

In case of the present invention, rendering and processing of images can be carried out in electronic devices capable of working with images, such as computer, video camera, video player, video rendering device (e.g., TV, monitor) . For the presented processing of the image electronic devices capable of processing of digital information (various memory modules, at least one processor, input and output devices, etc.) are needed.

High-resolution images render the captured object in greater detail and can therefore be used solving more sophisticated tasks. However, capturing of the objects in high resolution is limited by various reasons: technical (optics of photographic device does not zoom the image to the correct level), legal (resolution of satellite and infrared photographs is limited by law due to their strategic importance), etc. There are a lot of different image resolution enhancement techniques using the lower-resolution images. Part of the methods enhance the image resolution by using a single low-resolution image, other methods do the same by using two or more low-resolution images. In the first case, the image resolution is increased by various means of interpolation, and in the second case, low-resolution images are identified by the found angular points and there are attempts using algorithmic means to identify minor changes in camera position and possibilities of their compensation. Such methods may additionally use other image processing techniques, such as noise removal, sharpness enhancement, etc.

Previously discussed resolution enhancement methods treat the captured image as a plane and is therefore required that the original low-resolution images would be captured from a single point (the unchanging position of the camera). Method presented in this description for the increase of the image resolution uses the original low-resolution images captured from different positions of the camera and at different angles of inclination in respect of the captured object. In the present invention the formation of enhanced resolution images consists of three enhancement stages.

In the first stage of the resolution enhancement the 3D mesh of the object is restored from the captured photographs. The 3D mesh is restored using the methodology of photogrammetry by identifying the position and turning angles of the camera in respect of the captured object.

In the second stage, the correction of optical distortions is performed using the values of geometric distortions obtained during photogrammetry.

In the third stage, the intensities of points forming the image of enhanced resolution are determined.

The present invention is new because by the use of the said three image resolution enhancement stages the images of enhanced resolution are obtained, when original low- resolution images were captured at different positions and inclination angles of the camera. The descriptive method works better if the rays are drawn through each point of the surface and positions of the cameras seeing these points are at the angle greater than 5 degrees.

The image resolution enhancement stages are described below in more detail:

The enhancement of the resolution is performed from the N low-resolution images forming a set of low-resolution images L = ty ₀ ,V _l , ... ,V _N ). Each of the original low- resolution images V _t is changed to an image of enhanced resolution V/ . When changing the image resolution, the geometric dimensions of the image remain unchanged, while the number of image-forming points increases. The enhancement of the resolution is performed by a freely selected value but it is recommended not greater than 1.5 times, because when increasing more than 1.5 times, the results obtained at the time of the resolution enhancement calculation can be unstable.

Fig. 1 shows a possible enhancement of the resolution by the aforementioned 1.5 times: a) part shows the side of an image consisting of two low-resolution points; b) part shows the side of an image consisting of three points of enhanced resolution; c) shows the overlap of low-resolution and higher-resolution images. At every stage of resolution enhancement each pixel of enhanced resolution is being processed separately, i.e., the entire functional processing sequence is applied to each formed pixel of enhanced-resolution V/ .

Below the separate phases (steps) of functional processing sequences are described in more detail:

1. Radius R _t crossing the point Q ^l of a higher-resolution image, which we desire to calculate, is drawn through the center C _t of the lower-resolution image V _t . At the intersection of the radius and the reconstructed three-dimensional surface the point X is marked.

2. An auxiliary Z plane is drawn through the point X. The direction of the normal of the plane Z is opposite to the direction of the radius R _t .

3. Around the point Q ^l , located in the calculated high-resolution image, a square environment S of the point Q ^l is created. The square environment S of the point Q ^l is made of high-resolution pixels W _u ^l _v e S . The number of pixels, i.e. the size of the environment, is a matter of free choice. Fig. 1, b) shows the square environment of the point, the side length of which is equal to three high -resolution points.

4. Radiuses are drawn through the neighbouring points W _u ^l _v e S of the point Q ^l , at the intersection of which with the auxiliary plane Z three-dimensional points T _uv are marked.

5. At this stage, superposition of a two-dimensional image with three- dimensional image takes place. The auxiliary plane Z is tilted/turned so that at the point X its normal N _z would coincide with the normal N _x of the three-dimensional surface. Points T _uv are turned together with the plane Z .

6. Using projection matrices calculated during at the time of photogrammetry, three-dimensional points T _uv are designed and their projections W _M * in each lower-resolution image V _k e L are derived. Because of the tilt of the plane Z in respect of the three-dimensional surface, in each image the projections of points W^ _v will be rendered as irregular quadrangles, which are the best correspondent with the projection of the area of a three- dimensional surface in each image.

7. A new set of images K is formed from the former set of images L. The image from the set L is transferred to a new set of images K, if two conditions are met:

a) All projections W _v fall into the image V _k , i.e., VW «, v : 0 < u < w _v v _k ,0 < v < h v, _k , here w v, _k and h v _k is the width and height of the image V _k , respectively.

b) The normal N _x of the point X forms an angle of less than 60° with a radius R, of the image V, , i.e., arccos <— . Where

IHIHJ ³

n - the normal N _x of the point X written in a vector form, and r - radius of the image V _k - R _k written in a vector form.

8. The intensity values G _u ^k _v are calculated in the projection points W _v for all lower-resolution images V _k e K . For the determination of the intensity bicubic interpolation is used.

9. According to the description given in chapter "Equation System Composition" and obtained intensity values G _u ^k _v of low-resolution images a system of normal equations for the calculation of the intensities G _u ' _v of the enhanced resolution pixels is formed.

10. Having solved the equation system, the medium element x _i i = M /2 corresponding to the intensity G _u ' _v of the central enhanced resolution point is selected from the vector x and entered into the point Q' in the enhanced resolution image V/ .

11. The processing mechanism is repeated for other remaining points of the image V/ .

Fig. 2 shows the environment of the wanted point in different low-resolution images.

Below is provided an explanation concerning (composition) of the aforementioned equation system:

Each pixel is understood as the mean intensity of the four-square surface area occupied by this pixel. Lower-resolution pixels overlap imperfectly, so several high-resolution pixels fall into the foursquare of each pixel (some of which fall partially). In general, the value of each lower-resolution point can be calculated using the weighted average formula:

Here g _tj - the intensity of the low-resolution pixel, m - the value of the side of the point environment S (Fig. 1. shows the example, the side of which is equal to 3), g _u ' _v - the intensity of the enhanced resolution pixel, w _u ' _v - the weight of each high-resolution pixel is calculated as the ratio of g _u ' _v overlapping area with g _{j .

Value of high resolution point Q _x ^l _y is calculated from the projection of this point environment S in each low-resolution image. Fig. 1 shows the environment of the point in low-resolution image (part a) and high-resolution image (part b) as well as the overlapping areas of high- and low-resolution points (part c).

Using the formula (1), the normal equation system is concluded, which is written using matrix form:

Ax = b

Here x - the intensities of all wanted points of high-resolution image, b - the intensities of the original low-resolution pixels, A - weighting matrix corresponding to the area of each high-resolution pixel overlapping with low resolution pixels.

Taking into account the amount of high-resolution points (the unknowns) falling into the environment of the point the required quantity of equations in the system is determined. The number of equations must be equal to or greater than the number of wanted points (the unknowns). Fig. shows the example where the environment of high-resolution points consists of 9 unknown points and, therefore, the equation system must consist of at least 9 equations. The environment of low-resolution points consists of 4 points. To achieve the required amount of equations, we have to use the data of at least 3 low resolution images, thus forming a system of 12 equations, each of which will have 9 unknowns.

On the basis of the example presented in Fig. 1 and the formula (1), as well as the available intensity values G _u ^k _v of low-resolution pixels of the wanted point, a normal equation system is concluded for the calculation of the intensities G _u ' _v of the high-resolution pixels. Matrix A is filled with the ratios of the overlapping point areas. Each equation corresponds to the intensity estimate of a single low-resolution point made up of the estimates of unknown high-resolution points according to the formula (1). Vector b is filled with the intensity values G _u ^k _v of low-resolution points. In the example shown the first 4 equations describe the environment points of the first low- resolution image, the next 4 - of the second, and the last 4 - of the third image:

Having solved the equation system, in the vector x we find the intensities G _u ' _v of the enhanced resolution points. x— [ _j J , G ₁₂ , G ₁₃ , G ₂₁ , G ₂₂ , G ₂₃ , G ₃₁ , G ₃₂ , G ₃₃ ]

The matrix A will be the same for all calculated points (or it may be extended if more than 3 images will be used for calculations), and the vector b will be filled with the values of the environment points of the wanted point. Below are presented restrictions for the resolution enhancement mechanism.

Resolution enhancement in individual locations of a formed large high-resolution photograph will be different. The level of detail of the enhanced resolution image strongly depends on the amount of overlapping photographs and the total point density in the environment of the wanted point i.e., the surface of the object in the photographs taken from a closer distance has a higher resolution than in the photographs taken from a greater distance and, therefore, in an attempt to increase the resolution of the photographs taken from a closer distance by using the overlapping photographs taken from a greater distance, the result will be unstable. If the coordinates of the points of the overlapping photographs will highly correlate with each other the photographs might complement each other insufficiently because there will be no new information that could be used to enhance the resolution of the restored/formed photograph. However, due to the roughness of the investigated surface and the change in the position of the camera in other areas of the same photograph, correlation of the points will be lower and as a result the image of the enhanced resolution will be more detailed.

In the areas of the photographs abstract objects (sky, shiny surface, rippled water, etc.) are captured or there are no sufficient amount of overlapping photographs, the resolution is increased from the original low-resolution pixels using a bicubical interpolation method.

Although this paper describes the principles for the enhancement of the monochrome image resolution, but this method can also be used for colour images. In the case of colour images the colour of each enhanced resolution pixel is determined by adapting the described method to each original colour (red, green and blue) individually.

The above description of the preferred embodiments is provided in order to illustrate and describe the present invention. This is not an exhaustive or limiting description, seeking to determine the exact form or embodiment. The above description should be considered more like an illustration, rather than a limitation. It is evident that numerous modifications and variations may be obvious to the specialists of that field. Embodiment is chosen and described so that the experts of this field would clarify in the best way the principles of this invention and the best practical application for various embodiments with various modifications suitable for a particular use or application of the embodiment. It is intended that the scope of the invention is defined in the claim appended thereto and its equivalents, where all of the said terms have meaning within the widest range, unless indicated otherwise.

In the embodiment options described by the specialists of this field changes can be created without deviations from the scope of this invention as specified in the following claim.