FIELD OF THE INVENTION

The present invention relates to a method of removing fog from the images as well as videos independent of the density or amount of the fog and free of user intervention and also to a system for carrying out such method of fog removal from images and videos. More particularly, the present invention is directed to said method and system for removing fog from images and video involving airlight estimation and airlight map refinement based restoration of foggy images and videos. Advantageously, removal of fog from images and videos of this invention would require less execution time and yet achieve high perceptual image quality with reduced noise and enhanced contrast. The method and the system for carrying out the same would be effective even in case of deep fog and enable real-time implementation of fog removal alongwith video encoding or decoding and thus be user friendly and supposed to have wide application and use.

BACKGROUND OF THE INVENTION

Poor visibility degrades the perceptual image quality as well as the performance of the computer vision algorithms such as surveillance, object detection, tracking and segmentation. Poor visibility in bad weather such as fog, mist and haze caused by the water droplets present in the air. These droplets are very small(l - 10 μηι)[ K. Garg and S. K. Nayar, \Vision and Rain", International Journal of Computer Vision, Vol. 75, No. 1, pp. 3-27, 2007.] and steadily float in the air. Due to the presence of fog, mist and haze light scattered in the atmosphere before it reaches the camera. Here onwards the word fog will be used for all fog, mist, and haze. Two fundamental scattering phenomena which cause the scattering are attenuation and airlight. A light beam travels from a scene point through the atmosphere, it gets attenuated due to the scattering by the atmospheric particles, this phenomena is called attenuation which reduces the contrast in the scene. Light coming from the source is scattered towards the camera and leads to the shift in color. This phenomena is called airlight. Airlight increases with the distance from the object. It is noted that the fog effect is the function of the distance between the camera and the object. Hence removal of fog requires the estimation of the depth map or the airlight map. If input is only a single foggy image then estimation of the depth map is under constrained. Generally estimation of depth requires two images. Therefore many methods have been proposed which use multiple images. Schechner et al [ Y. Y. Schechner, S.G. Narasimhan, and S.K. Nayar, \Instant dehazing of images using polarization", IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 325-332, 2001.] proposed a method based on polarization. This method removes the fog through two or more images taken with different degrees of polarization. But this method can not be applied on existing databases. In past few years many algorithms have been proposed for the removal of fog which use single image.

Fattal [R. Fattal, \Single image dehazing", International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH, pp. 1-9, 2008.] proposed a method which is based on the independent component analysis (ICA). This method estimates the optical transmission in hazy scenes. Based on this estimation, the scattered light is eliminated to increase scene visibility and recover haze from scene contrasts. Here restoration is based on the color information, hence this method can not be applied for the gray image. This method fails when there is dense fog because dense fog is often colorless.

Tan [R. T. Tan, \Visibility in bad weather from a single image", IEEE conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.] proposed a method based on spatial regularization from a single color or gray scale image. Tan removed the fog by maximizing the local contrast of the image but restored image looks over saturated.

Kopf et al [J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski, \Deep photo : Model-based photograph enhancement and viewing", ACM Transactions on Graphics, Vol. 27, No. 5, pp. 116: 1-116: 10, 2008.] proposed a method based on the use of a 3D model of the scene. This method is application dependent and requires the interactions with an expert.

He et al [K. He, J. Sun, and X. Tang, \Single image haze removal using dark channel prior", IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1956-1963, 2009.] proposed a method based on the matting and dark channel prior from a single color or gray scale image. But when the scene objects are bright similar to the atmospheric light, underlying assumptions of this algorithm do not remain valid.

Tarel et al [J. P. Tarel and N. Hautiere, \Fast visibility restoration from a single color or gray level image", IEEE International Conference on Computer Vision, pp. 2201-2208, 2009.] proposed a fast visibility restoration algorithm. This method assumes the airlight as a percentage between the local standard deviation and the local mean of the whiteness. This method based on linear operations but requires many parameters for the adjustment for optimal result. It is thus evident that there exist systems to remove fog from images captured by multiple cameras viz. stereoscopic imaging and there are also systems that do the same job using only one camera. However, no existing system attempted to remove fog from videos. The present invention involves a system and method using single camera approach for removing fog from images as well as videos which save cost and computation.

OBJECTS OF THE INVENTION It is thus the basic object of the present invention to provide for technical advancement in the field of removing fog from the images as well as videos for enhancing the perceptual image quality involving airlight estimation technique which would be user friendly and would not require any user intervention. A further object of the present invention is directed to providing for a method and system of removing fog effects from the images and/or videos which would effectively improve perceptual image quality for both RGB and gray image/video.

Another object of the present invention is directed to for a method and system of removing fog effects from the images and/or videos which would be adapted for RGB Colour model and advantageously also for HIS colour model involving reduced , computational requirements.

A still further object of the present invention is directed to providing for a method and system of removing fog effects from the images and videos which can efficiently improve perceptual image quality even in conditions of dense fog.

A still further object of the present invention is directed to providing a method and system of removing fog effects from the images and/or videos which would provide for the much required flexibility to focus on the restoration quality at a particular depth and can be effectively applied for restoration of images which are not degraded by fog.

A still further object of the present invention is directed to providing a method and system of removing fog effects from the images and/or videos adapted to limit noise while enhancing contrast.

A still further object of the present invention is directed to providing a method and system of removing fog effects from the images and/or videos which would be simple, cost-effective and advantageously involve high perceptual image quality.

A further object of the present invention is directed to providing for a method and system for fog removal from images and/or videos which would aid the performance of various computer vision algorithms especially those which use feature information such as object detection, tracking, segmentation and recognition.

Yet another object of the present invention is directed to a method and system for fog removal from images and/or videos which could be implemented for various hand held embedded systems. Another object of the present invention is directed to a method and system for fog removal from images and/or videos which could be advantageously implemented and provided for variety of applications such as navigational systems using visual cues, consumer electronics, entertainment industry and film post production.

SUMMARY OF THE INVENTION

The basic aspect of the present invention is thus directed to ά method for removing fog from images and/or videos comprising of: (a) carrying out histogram equalization whereby the foggy image is equalized to increase the contrast;

(b) providing an airlight map of the foggy image/video;

(c) Refining the airlight map involving anisotropic diffusion and/or bilateral filtering; and (d) Restoring the foggy image based on said refined airlight map and optionally, when the restored image is having low contrast, if required, performing histogram stretching of the restored image.

In the above method for removing fog from images and/or videos the said images and/or videos include multi component colour images/videos including RGB, HSI images/videos and/or single component including gray images/videos.

A further aspect of the present invention is directed to a method for removing fog from images /videos wherein after airlight map A is estimated, carrying out restoration of each color component of de-foggy image following :

wherein ce (r, g, b), I ₀ (x,y,c) is image intensity in absence of fog, I(x,y,c) is observed image intensity at pixel (x,y) and I^c) is global atmospheric constant or sky intensity, factor emphasize the restoration at particular depth and ki ,k ₂ are extinction coefficients. A still further aspect of the present invention is directed to said method for removing fog from images /videos wherein after airlight map A is estimated, since fog has no effect on the hue, carrying out restoration based on HSI (Hue, Saturation and Intensity) colour model involving processing carried out only over the saturation and intensity plane with reduced computation following :

where S _w is saturation in absence of fog and S is saturation of foggy image, A(x,y) is the airlight at pixel (x,y), min(I(x,y)) is the minimum image intensity of foggy image at pixel(x,y), I _int (X _/ y) is the intensity component (average of R,G and B) of a colour vector I(x,y) representing [R,G,B] . W 201 A still further aspect of the present invention is directed to said method for removing fog from images /videos wherein said airlight map is estimated iteratively involving anisotropic diffusion or non iterative bilateral filter.

Yet another aspect of the present invention is directed to said method for removing fog from images /videos wherein said airlight estimation involving anisotropic diffusion or bilateral filter is carried out perform the smoothing of the intra region and edges remain stable over a very long time whereby diffusion and edge detection is done under a single process adapted to reduce the diffusivity at those locations which have a larger likelihood to be edges.

A still further aspect of the present invention is directed to said method for removing fog from images /videos wherein said foggy image estimated airlight map depends upon the distance of the scene points from the camera. According to yet another aspect of the present invention is directed to said method for removing fog from images /videos comprising flexibility to focus on the quality of the image at required depth involving said factor 'q' adapted to control the visual quality of restored image such that for lower value of q(q< l), objects in lower depth are more enhanced, and on the other hand for higher value of q(q > l), the images at further end are more enhanced.

Importantly, also in said method for removing fog from images /videos involves different extinction co-efficients (ki & k ₂ ) for attenuation and airlight. A further aspect of the present invention is directed to said method for removing fog from videos comprising involving temporal redundancy, wherein I frame is spatially encoded and P frame is temporally coded, said temporal coding carried out using motion vectors. Yet another aspect of the present invention is directed to said method for removing fog from videos wherein the airlight for the I frame is estimated by suitable methods including the anisotropic diffusion and airlight of the P frame is essentially estimated using previously calculated motion vectors. A further aspect of the present invention is directed to a method for removing fog from videos as above comprising step of regularizing motion vectors for inaccuracies stemming from 3D to 2D mapping of real motion, finite aperture, motion discontinuities and temporal discontinuities. Another aspect of the present invention is directed to a method for removing fog from videos as above wherein for faster computation when the frame rate is high and the relative motion between the scene and camera is low ,the temporal correlation between constitutive frames is high ,the airlight map of the previous frame is used as an estimate of the airlight of the frame under consideration.

According to a further aspect of the present invention is directed to a system for removing fog from images/videos for implementation of the method as described above comprising of: (a) means for carrying out histogram equalization whereby the foggy image is equalized to increase the contrast;

(b) means for generating said airlight map of the foggy image;

(c) means adapted for refining the airlight map involving anisotropic diffusion and/or bilateral filtering; and

(d) means for restoring the foggy image based on said refined airlight map and, optionally ,if required, when the restored image is having low contrast adapted to perform histogram stretching of the restored image.

The invention is further discussed hereunder in greater detail in relation to the following non-limiting exemplary illustrations as per the following accompanying figures:

BRIEF DESCRIPTION OF THE ACCOMPANYING FIGURES

Figure 1 : is the schematic Block diagram of the proposed fog removal method for images.

Figure 2: show the surface plot of the brightness (a) before, (b) after the anisotropic diffusion and (c) after the bilateral filtering. Figure 3: shows airlight map estimation (a) original foggy image, corresponding airlight map using(b) anisotropic diffusion and (c) bilateral filtering. W 201

Figure 4: is the schematic Block diagram of the proposed fog removal algorithm for the videos.

Figure 5: is the images of (a) Original foggy lonavala 01' image, (b) restored ' lonavala 01' image by the proposed 1 algorithm, (c) restored ^' lonavala 01' image by the proposed 2 algorithm; (d) Original foggy ^' Ionavala02' image (e) restored ' Ionavala02' image by the proposed 1 algorithm, (f) restored ^' Ionavala02' image by the proposed 2 algorithm, (g) Original foggy ^' yellow mountain' image, (h) restored ^' yellow mountain' image by the proposed 1 algorithm, (i) restored ' yellow mountain' image by the proposed 2 algorithm.

Figure 6: (a)-(c) 3 frames of ^' fogiitcampusOl' video, & (d)-(f) removal of fog in corresponding above frames by proposed 1 algorithm.

DETAILED DESCRIPTION OF THE INVENTION WITH REFERENCE TO THE ACCOMPANYING DRAWINGS

The present invention is directed to an efficient fog removal method and system especially for the effective removal of fog from images and/or video taking care of the effect of both the scattering phenomena; airlight and attenuation. Importantly, the present invention addresses the issue of fog removal from images and/or video involving advancement in the estimation of airlight in a frame following either anisotropic diffusion or bilateral filtering. For P/B frames the airlight estimate can be derived from reference frames. Proposed method of the invention can be applied for the RGB and gray scale images/videos. The method and the system for carrying out the same is effective even in the case of dense fog. The invention is adapted to achieve significant improvement in the enhancement of the foggy images in less time and is directed to pave the way for realtime implementation of fog removal along with video encoding or decoding. The system and method of the invention involves a single camera approach to remove fog from images and also for the first time to remove fog from videos and enhancing perceptual image quality using temporal redundancy characteristics of videos. The system and method of the invention can be advantageously used for the HSI color space. Use of the HSI color space helps to reduce the complexity further and maintains the color fidelity. W In accordance with the present invention , fog attenuation is represented as ι ₃ κ(χΛ = ΐο(χ,γ) ^β -ν ^(Χ ° (1) where, I _a tt(x,y) is the attenuated image intensity (gray level or RGB) at pixel (x; y) in presence of fog and I ₀ (x,y) is the image intensity in absence of fog. ki is the extinction coefficient and d(x; y) is the distance of the scene point from the viewer or camera.

Airlight is represented as

A(x,y)= I _∞ (l- e - ^k ₂ ^d(x ' ^y) ) (2) where I _∞ is the global atmospheric constant. It is also called sky intensity. K ₂ is the extinction coefficient. Hence according to the Koschmieder's law, the effect of fog on pixel intensity is represented as

where, I(x; y) is the observed image intensity at pixel (x; y).

I(x,y)= I ₀ (x,y) e-y ^(x<y) + I _∞ (1- e - ^k ₂ ^d(x ' ^y) )

(4) wherein the right hand side first term is the direct attenuation and second term airlight.

According to Equation (2)

Thus,

where q = ki/k ₂ , in general q= l i.e. kj = k _{2 ;} the factor q can emphasize the restoration at particular depth. The airlight depend on the depth of the scene. Airlight is also dependent upon various assumptions of the underlying estimation procedure. Thus the method according to the present invention suggests different extinction co-efficients^ & k ₂ ) for attenuation and airlight in Koschmeider law which gives more flexibility to emphasize different depth at the time of restoration and to handle different situations. Different objects may be at different distance from camera and thus airlight should be different for different objects. Also it must be smooth for an object except along edges. Airlight map undergo intra- region smoothing preferentially over inter-region smoothing. Thus diffusion takes place in the interior region without affecting the region boundary. Accompanying Figure 1 schematically illustrates the method of the present invention with a representative block diagram wherein the details of each blocks are as follows:

Histogram equalization : First the foggy image is equalized to increase the contrast. Initial estimation of airlight : To restore the foggy image, there is a requirement of the airlight map of the foggy image. Here initial estimation of airlight map is based on the dark channel prior.

Airlight map refinement : Proposed techniques estimate the airlight map by the anisotropic diffusion and bilateral filtering. Proposed technique with anisotropic diffusion is proposed 1 algorithm and proposed technique with bilateral filter is proposed 2 algorithm. Generated airlight map preserves the edges and performs the smoothing over the object region. The procedure can be applied for the RGB and gray scale images and is effective even in the case of dense fog.

Airlight map estimation using anisotropic diffusion :

Anisotropic diffusion can be represented as. dA

Έ (8) where div is the divergence operator and a is the conduction coefficient. V and Δ are the gradient and Laplacian operators respectively. If a (x,y,t) is constant over time then equation (8) reduces to heat diffusion equation (9)

Heat equation describes the distribution of heat in a region over time. Heat equation states that if a body is kept inside the box of cold water, how the temperature of the body will decrease and finally attain the temperature of the surrounding water. To encourage the smoothing within a region instead of the smoothing across the edges, a should be 1 in the interior region and 0 at the edges. Hence if E(x,y,t) is the estimation of the boundaries, then according to Perona-Malik equation the conduction coefficient should be chosen as

where g(.) is a nonnegative monotonically decreasing function with g(0) = l. Thus diffusion take place in the interior region without affecting the region boundaries.

Here g(.) is assumed as where k is a constant which is fixed. Hence according to equation (9) airtight map can be estimated iteratively as follows

(12) where λ (0<λ< 1) is a smoothing parameter.

Air light map estimation using bilateral filter:

The bilateral filter is intuitively developed as a noniterative alternative to anisotropic diffusion, achieving satisfying results with a single iteration. The discrete version of the kernel of a bilateral filter can be expressed as

Where ξ is the pixel position, A(x) is the estimate of the current pixel A ₀ (x), η _χ represents the neighborhood of x(where ξ is a member of that neighborhood), w(x) and c(x) are the spatial domain and intensity domain kernels, respectively. The kernel w(x) basically averages the neighboring pixel intensity values with decreasing weights for pixels at larger distances. The kernel c(x) takes into account the intensity differences between the center pixel and its neighboring pixel; the larger the pixel value difference is, the smaller the pixels contribution is during smoothing. In this way, a pixel is simply replaced by the weighted mean of its neighbors.

Initial estimation of An

Implementation of equations (12) and equation (13) require the initial estimation of the An. It is known that airlight map A is a scalar image which is always positive, hence A ₀ >0. Taking minimal component of equation (7)

According to dark channel prior, dark channel is denoted as the minimum intensity across the red, blue and green channels. Natural outdoor images are usually full of shadows and colourful objects(viz green grass, trees, red or yellow plants and blue water surface). Thus dark channels of these images are really dark. For the fog free image except for the sky region, intensity of the dark channel is low and tends to be zero.

Hence ^¾ 0 mid Positive. Thus _rt& ¾(%y)) i i > 0 (14)

Thus initial estimate of A ₀ can be assumed as ** = ^β . rSRWW ⁾ (15) where β is a constant and 0<β< 1. If the input image is the gray scale image then initial estimation of the A ₀ can be assumed as

Ao = ? I( x, y) (16) Thus it is evident from the above methodology for the airlight estimate that anisotropic diffusion and bilateral filter perform the smoothing of the intra region and edges remain stable over a very long time which is illustrated in the accompanying Figure 3. These results look very impressive. It is due to the fact that instead of considering the diffusion and edge detection as two independent process, here both process interact in one single process. Here an inhomogeneous process is applied that reduce the diffusivity at those locations which have a larger likelihood to be edges. Airlight map estimation for the real foggy image is shown in the accompanying Figure 4. It is observed that in foggy image estimated airlight map depends upon the distance of the scene points from the camera. Estimated airlight map is able to capture the discontinuities across the edges and smoothing over the objects.

Restoration : Once airlight map A is estimated then each color component of de-foggy image can be restored as

where ce (r, g, b).

The initial estimation of airlight is based on the minimum channel prior denoted as the minimum intensity across the red, blue and green channels and which is not the exact one. Bilateral filtering and Anisotropic diffusion also introduce some error in the approximation of airlight map. To reduce this error effect, the present invention introduced the factor 'q' .e. ki/k ₂ ). This factor 'q' controls the visual quality of restored image.

The above factor q control the visual quality of the restored image. It is possible following the findings under the present invention that factor q control the restoration process. For lower value of q(q< l), objects in lower depth are more enhanced, and on the other hand for higher value of q(q> l), the images at further end are more enhanced. This gives a flexibility to focus on the quality of the image at the required depth. It is known that color image can also be represented in more perceptually relevant HSI (Hue, Saturation, and intensity) color space. Hue is a color attribute that describes a pure color, whereas saturation gives a measure of the degree to which a pure color is diluted by white light. This color space decouples the intensity component from the color carrying information (hue and saturation) in a color image. It is observed that fog has no effect on the hue of scene. Other two components saturation and intensity are affected by fog. Hence to restore a foggy image, processing is needed only over the saturation and intensity plane. Thus unlike RGB color space, computation is reduced by 1/3 in HSI color space. Post processing : Restored image may have low contrast. Thus there is a requirement of some post processing. This post processing step is optional and required only when restored image has low contrast. To increase the contrast, histogram stretching of the restored image is performed. Extension to videos: Temporal redundancy removal is an essential step of any video coding standard viz. MPEG. Above fog removal procedure can be extended to videos with the use of the temporal redundancy, wherein I frame is spatially encoded and P frame is temporally coded. Temporal coding is achieved using motion vectors. The airlight for the I frame is estimated by the anisotropic diffusion and airlight of the P frame is estimated using previously calculated motion vectors. While alternative methods other than anisotropic diffusion to estimate airlight of I frame is possible to be adopted, estimation of airlight for P frame is essentially carried out using motion vectors according to the present invention. Accompanying Figure 2 illustrate the proposed fog removal procedure extended to videos. The method according to the present invention is also capable of enhancing non-foggy images without producing any degradation in the visual quality.

In order to evaluate the performance, contrast gain (C _gain ) and percentage of number of saturated pixels(a) are calculated and compared with existing techniques. Contrast gain is the difference of mean contrast of the enhanced image and the mean contrast of the original image plagued by the fog. The contrast gain value should be positive and greater is the value of the contrast gain, stronger is the resulting contrast enhancement. Contrast gain should not be so high that the pixels of the output image become over saturated. On the other hand, a low value of σ indicates the better performance of the method of the present invention.

Simulation of the method of fog removal according to the present invention is carried out in various foggy images and videos. Results of the simulation for images are illustrated in the accompanying Figure 5 and the results of simulation for the videos are illustrated in accompanying Figure 6. Accompanying Figure 3 shows the images of (a) Original foggy ^' lonavala 01' image, (b) restored ^' lonavala 01' image by the proposed 1 algorithm, (c) restored ^' lonavala 01' image by the proposed 2 algorithm; (d) Original foggy ^' Ionavala02' image (e) restored ^" Ionavala02' image by the proposed 1 algorithm, (f) restored ^' Ionavala02' image by the proposed 2 algorithm, (g) Original foggy ' yellow mountain' image, (h) restored ^' yellow mountain' image by the proposed 1 algorithm, (i) restored ^' yellow mountain' image by the proposed 2 algorithm. Accompanying Figure 4 shows the images (a)-(c) 3 frames of ^' fogiitcampusOl' video, & (d)-(f) removal of fog in corresponding above frames by proposed 1 algorithm. Results show that the proposed method is capable to restore the images/videos with substantially enhanced perceptual quality. It is observed that as regards the fog removal technique followed as per the present invention, anisotropic diffusion is superior in terms of restoration quality and speed compared to the bilateral filtering and similar other techniques known in the art.

The present invention thus provide for the desired above method of removal of fog from the images and/or videos constitutes an advancement in the related art and the advantages derived include: (a) capable of removing fog from images as well as videos; (b) can be applied to any existing fog removal technique for enabling fog removal of videos.

(c) Can be applied for both color as well as gray images/videos.

(d) Can be applied for pre processing and post processing of video.

(e) Can provide the desired flexibility to focus on the restoration quality at a particular depth.

(f) Can be used for HSI color space too, which maintains color fidelity and reduces the computation in great extent.

(g) Can be applied also for restoration of images which are not degraded by fog.

(h) Can improve the performance of various computer vision algorithms which use feature information such as object detection, tracking, segmentation and recognition.

(i) Is found to be suitable for real time application for handheld embedded system, (j) Has the potential for application in navigation systems using visual cues, consumer electronics, entertainment industries and film post production.