OF IMAGES ACQUIRED IN A DISTORTING MEDIUM

PRIORITY CLAIM AND REFERENCE TO RELATED APPLICATION

[001] The application claims priority under 35 U.S.C. § 119 and all applicable statutes and treaties from prior provisional application serial number 62/220,306, which was filed September 18, 2015.

STATEMENT OF GOVERNMENT INTEREST

[002] The invention was made with support under grant number CCF- 1160832 awarded by the National Science Foundation. The government has certain rights in the invention.

FIELD

[003] Fields of the invention include camera systems and image reconstruction, particularly for images and videos acquired in a distorting medium. Particular applications of the invention include underwater photography and video and reconstruction of images and videos acquired underwater. BACKGROUND

[004] Underwater images or videos often suffer from visible degradation and color distortion due to the propagated light attenuated with distance from the camera, primarily resulting from absorption and scattering effects. Acquired images and videos often suffer from poor contrast and details of images and video frames are often lost, even after efforts toward reconstruction of the images or videos. Similarly, other distorting media, such as fog, haze and other liquids created visible degradation and color distortion from absorption and scattering effects.

[005] Prior methods for reconstruction of underwater images estimate depth based upon particular color channels. Some previous methods use the image formation model (IFM) to estimate the background light and the transmission map based on depth estimation. These methods still provide images with poor contrast. Some of these methods use a dark channel prior (DCP) to derive scene depth and remove haze in natural terrestrial images by calculating the amount of spatially homogeneous haze using the darkest channel in the scene. H. Yang, P. Chen, C. Huang, Y. Zhuang and Y. Shiau, "Low complexity underwater image enhancement based on dark channel prior," Int. Conf. Innov. in Bio-inspired Comput. and App. (IBICA), pp. 17- 20, Dec. 2011. 17-20, (2011); L. Chao and M. Wang, "Removal of water scattering," in Proc. IEEE Int. Conf. Comput. Engin. and Tech. (ICCET) vol. 2, pp. 35-39 (Apr. 2010). With the DCP approach, points in the scene closer to the camera have a shorter path over which scattering occurs, and dark pixels therefore experience less brightening from scattered light. This observation is used to estimate depth.

[006] Some DCP approaches use only the blue and green channels. See, e.g., H.

Wen, et al., "Single underwater image enhancement with a new optical model," in Proc. IEEE Int. Symp. Circ. & Syst. (ISCAS), pp. 753-756 (May 2013); P. Drews, et al, "Transmission Estimation in Underwater Single Images," in Proc. IEEE Int. Conf. Comput. Vis. Workshops (ICCVW), pp. 825-830 (Dec. 2013). The theory in these approaches is that red light attenuates faster due to its longer wavelength and the blue and green channels therefore provides necessary priors for image restoration.

[007] The inventors have determined that light absorption and different lighting conditions existing in underwater images and other distorting media can create exceptions to those priors. When such exceptions exist, the resulting image restoration based on the priors can be poor.

SUMMARY OF THE INVENTION

[008] An embodiment of the invention is a method for depth estimation in image or video obtained from a distorting medium. In the method, a pixel blurriness map is calculated. A rough depth map is then determined from the pixel blurriness map while assuming depth in a small local patch is uniform. The rough depth map is then refined. The method can be implemented in imaging systems, such as cameras or imaging processing computers, and the distorting medium can be an underwater medium, haze, fog, low lighting, or a sandstorm, for example. Preferred embodiments determined the blurriness map by calculating a difference between an original image and multi-scale Gaussian-filtered images to estimate the pixel blurriness map.

BRIEF DESCRIPTION OF THE DRAWINGS

[009] FIG. 1 illustrates a preferred embodiment system for depth estimation of images acquired in a distorting medium; [0010] FIG. 2 provides pseudo code for selection of separate BL (background light) for separate color channels that is used in a preferred embodiment depth estimation method of the invention; and

[0011] FIG. 3 shows a set of weighted combination curves illustrating (from left to right) curves plotted using different s = 8, s = 16, and s =32 for calculating the BL estimate.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Preferred embodiments of the invention include camera systems that acquire still images and/or video. The systems include software stored on a non- transient medium for reconstructing images and videos to enhance the images and videos from distortion caused by a distorting surrounding medium. Systems and methods of the invention are applicable, for example, to underwater photography and video acquisition, and to photography and video acquisition from other distorting media, such as fog, haze, dimly lit images and sandstorm images. Methods of the invention can be performed within cameras or video acquisition equipment, and can also be performed by image/video processing and adjustment software on non-transient medium executed by a computer.

[0013] Preferred methods for depth estimation of distorted medium images in video use image blurriness to estimate scene depth. A preferred embodiment leverages a recognition of the inventors that larger scene depth causes more object blurriness for distorted medium images, e.g. underwater images. Preferred methods estimate background light first, and then perform depth estimation and transmission map estimation based on the estimated background light so as to adaptively produce the more accurate scene depth and transmission map for an underwater image. The inventors have determined that background light is a key individual factor to decide the effectiveness of the depth estimation methods based on color channels for underwater images. This depth estimation can enhance image reconstruction.

[0014] Additional preferred methods for reconstruction provide a unified medium depth estimation based on image blurriness and light absorption. The extended method provides unified underwater depth estimation based on image blurriness and light absorption that works well in different lighting conditions and hues of water. Unlike other methods, preferred methods of the invention perform depth estimation based on background light estimation so as to adaptively produce the more accurate scene depth for underwater image restoration.

[0015] Preferred methods of the invention have been shown to work well in different lighting conditions and hues of water. Preferred methods use background light estimation to estimate scene depth and can therefore adaptively produce an accurate scene depth for image restoration, of images acquired in a distorting medium such as water.

[0016] A preferred depth estimation method makes a pixel blurriness estimation. A difference between an original and a multi-scale Gaussian-filtered image is calculated to estimate the pixel blurriness map. A rough depth map is generated by applying a maximum filter to the pixel blurriness map while assuming depth in a small local patch is uniform. Closing by morphological reconstruction and guided filtering are then used to refine the depth map.

[0017] Those knowledgeable in the art will appreciate that embodiments of the present invention lend themselves well to practice in the form of computer program products. Accordingly, it will be appreciated that embodiments of the present invention may comprise computer program products comprising computer executable instructions stored on a non-transitory computer readable medium that, when executed, cause a computer to undertake methods according to the present invention, or a computer configured to carry out such methods. The executable instructions may comprise computer program language instructions that have been compiled into a machine- readable format. The non-transitory computer-readable medium may comprise, by way of example, a magnetic, optical, signal-based, and/or circuitry medium useful for storing data. The instructions may be downloaded entirely or in part from a networked computer. Also, it will be appreciated that the term "computer" as used herein is intended to broadly refer to any machine capable of reading and executing recorded instructions. It will also be understood that results of methods of the present invention may be displayed on one or more monitors or displays (e.g., as text, graphics, charts, code, etc.), printed on suitable media, stored in appropriate memory or storage, etc.

[0018] Preferred embodiments are discussed with respect to the RGB color space.

Artisans will appreciate that the invention is applicable to arbitrary color spaces.

[0019] Preferred embodiments use image blurriness to estimate scene depth. One technique for estimating blurriness is provided in S. Bae and F. Durand, "Defocus magnification," in Proc. Comput. Graph. Forum., vol. 26, no. 3, pp. 571-579 (2007). This is a scale edge detector to estimate pixel blurriness, and then defocus map is generated by using edge aware interpolation. See, A. Levin, D. Lischinski, and Y.Weiss, "Colorization using optimization," ACM Trans, on Graphics 23(3), pp. 689-694 (2004). In constructing the defocus map, the blurriness at non-edge pixels is interpolated and propagated by neighbouring edge pixels based on the similarities of luminance. If matting Laplacian [A. Levin, D. Lischinski, and Y. Weiss, "A Closed-Form Solution to Natural Image Matting," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 2, pp. 228 - 242, (Feb. 2008)] is applied to perform the interpolation, noise and low contrast may cause incorrect blurriness propagation, especially for underwater image. The edge-aware interpolations are computationally expensive because the methods require solving high-dimension linear equations. Preferred methods therefore construct the defocus map using closing by morphological reconstruction (CMR). See, P. Soille, "Morphological Image Analysis: Principles: and Applications," Springer- Verlag, pp. 173-174 (1999).

[0020] Preferred embodiments of the invention will now be discussed with respect to the drawings and with respect to experiments that have been conducted to demonstrate the invention. The drawings may include schematic representations, which will be understood by artisans in view of the general knowledge in the art and the description that follows. Features may be exaggerated in the drawings for emphasis, and features may not be to scale. Artisans will understand broader aspects of the invention from the experiments and experimental data.

[0021] FIG. 1 illustrates a preferred embodiment system of the invention. A camera

10 obtains images or video. The camera 10 is an underwater camera in preferred embodiments, and includes a sealed housing 12. The camera further includes memory 14 for storing images, and a processor 16 for controlling camera functions. The processor 16 also can run software that processes images or video that is obtained prior to and after storing, and can control transfer of images via wired or wireless connection to another devices, such as a computer 18. The computer 18 can include image processing/adjustment software. Either or both of the processor 16 and the computer 18 can perform present methods for estimating depth and image restoration. The camera 10 also includes a lens 20, light meter 22 and operator controls 24.

[0022] The camera 10 is illustrated in a distorting medium of water 26 that separates the lens 20 from a scene 28. This makes the observed intensity I a distorted version of the scene radiance J. Specifically,

[0023] [0024] where I is the observed intensity of the input image at pixel x, J is the scene radiance, B ^c is the background light (BL), and t is the is the transmission medium map that describes the portion of the scene radiance that is not scattered or absorbed and reaches the lens 20 of camera 10. In this model, the transmission map is the depth map, i.e., smaller t means the scene point is farther from the camera.

[0025] Prior DCP methods are focused upon outdoor scenes taken above ground and recover the scene radiance J with a statistical observation that about three quarters of the pixels in non-sky pates of outdoor images have zero values in the dark channel. This can be written as follows (taken from K. He et. al, "Single image haze removal using dark channel prior," IEE Trans. Pattern Anal. Mach. Intell. vol. 33, no 12, pp. 2341-53 (Dec. 2011).

[0027] where Ω(χ) is the local patch centered at x, and is the intensity of the

scene radiance in channel c, where c is one of the R, G, B channels. The minimum operator of Eq. 2 can be applied to both sides of Eq. 1, which roduces:

[0029] The depth map has block-like artifacts. These can be filter via techniques such as image matting (A. Levin et al, "A Closed-Form Solution to Natural Image Matting," IEEE Trans. Pattern. Anal. Mach. Intell., vol. 30, no. 2, pp. 228-242 (Feb. 2008) or guided filtering (K. He, "Guided Image Filtering,". IEEE Trans. Pattern. Anal. Mach. Intell., vol. 35, no. 6, pp. 228-242 (Oct. 2012).

[0030] The dark channel of the input image can be defined as

To estimate the BL B ^c , the top 0.1% brightest pixels in

a ^re selected. The corresponding intensities in the input image are averaged. This effectively obtains BL from the 0.1% most distant scene points. If S _0.1 is the set of the 0.1 % brightest pixels, then BL can be calculated as

[0031]

[0032] Then, the scene radiance J can be recovered by placing I, t, and β into Eq. 1, as follows:

[0033]

[0034] where r ₀ is typically set ranging from 0.3 to 0.5. The value is chosen empirically to increase the exposure of J for display. This prior DCP method is effective in some cases but fails in others. The failure case can produce images missing a great amount of detail, especially when a foreground object (close object) has very bright features, which are mistakenly treated as being distant.

[0035] Preferred methods of the invention avoid such issues. In a preferred method, a blurriness estimate is obtained. The blurriness estimate can be determined from prior technique for blurriness, including those mentioned above. Preferred methods include multi-scale low-pass filtering to determine blurriness and multi-scale Gaussian-filtered images to determine blurriness. Another approach is to measure blurriness in the frequency domain. The frequency domain approach. See, Kanjar and Masilamani, "Image Sharpness Measure for Blurred Images in Frequency Domain," International Conference on Design and Manufacturing, Volume 64, Pages 149-158 (2013).

[0036] The preferred Gaussian-filtered images will be discussed as a detailed example. In the example, a difference between an original and multi-scale Gaussian-filtered images is calculated to estimate a pixel blurriness map. A rough depth map is then generated by applying a maximum filter to the pixel blurriness map while assuming depth in a small local patch is uniform. The rough depth map is refined, such as by guided filtering or image matting. See, K. He, "Guided Image Filtering,". IEEE Trans. Pattern. Anal. Mach. Intell., vol. 35, no. 6, pp. 228-242 (Oct. 2012); Levin et al, "A Closed-Form Solution to Natural Image Matting," IEEE Trans Pattern Anal Mach Intell. 30(2):228-42 (2008). Experiments compared the present method to the dark channel prior methods (DCP) maximum intensity prior (MIP) discussed in the background. Qualitative and quantitative measures were evaluated in different underwater lighting conditions. The experiments showed that the depth estimation based upon blurriness worked well for a variety of images, and provided better results with reasonable computation costs, whereas the prior methods sometimes fail to distinguish between background and foreground and sometimes cause color distortion and poor global contrast.

[0037] The pixel blurriness map can be calculated as follows. is the input

image filtered by a k x. k spatial Gaussian filter with variance σ ² . The pixel blurriness map Pmit is computed as:

[0038]

[0039] where r _t = 2 ^l n + 1 and I _g is the grayscale version of the input image I ^c . As an example, we can assign n = 4. This assignment can be altered based upon factors that include image resolutions and image characteristic (quality, sharpness, etc.). The assignment can also be dependent upon the precision of blurriness estimation. There is a tradeoff between precision and computational cost. Generally, a larger n results in a more precise estimation and higher computation cost than a lower n. A preferred range of n for a 1280 x 720 image is 1-6.

[0040] The max filter can be applied to B to calculate a rough depth map P _r as:

[0041]

[0042] where Ω(χ) is a z x z local patch centered at x. As an example, we set z =

7. More generally, a patch size of z = 7 to z— 31 works well for image sizes ranging from 800 x 600 to 1280 x 720. The value of z is can be determined by image resolutions and image contents, such as the size of objects in the image. For example, a larger z does not work well for small objects, which could cause halo effects around such objects in the restored image. According to experiments that were conducted, we found a patch size of z=7 to z=31 works well for image sizes ranging from 800 x 600 to 1280 x 720. Generally, an image with a higher resolution requires a larger z, i.e., z preferably increases with image resolution. The rough depth map P _r can then be refined. One option for refinement is hole filling, which can fill holes caused by the flat regions in objects during the closing by morphological reconstruction (CMR).

[0043] Specifically, refine by filling holes caused by flat regions in objects using

morphological reconstruction followed by soft matting or guided filtering applied to generate a refined blurriness map

[0044]

[0045] where C _r is a hole-filling morphological reconstruction and F _g represents soft matting or guided filtering.

[0046] Preferred embodiments modify the depth estimation above. This modification improves depth estimation, especially when red light content is significant. In this modification, a candidate selection process chooses from a plurality of candidate blurriness regions. There can be many regions that consist of a top predetermined percentage of blurry pixels, which are global to the image. A preferred candidate selection process selects from a top predetermined percentage of blurry pixels in the input image, a variance region within a predetermined lowest variance range and a blurriness region within a predetermined largest blurriness range. A specific preferred candidate selection process selects from among a predetermined number of background light (BL) candidates that are within a predetermined percentage of the top most blurry pixels, the lowest variance region and the largest blurriness region. The BL candidates are average values of the candidate regions. If three BL candidates are chosen, these correspond to three average pixel values of three BL candidate regions. For example, a preferred embodiment selects three BL candidates from the top 0.1% blurry pixels in the input image, the lowest variance region and the largest blurriness region. These candidate regions are preferably of equal size. For example, a preferred technique divides the input image I into four equally sized regions

The lowest variance region and the largest blurriness region (which may or may not be the same) can be determined, for example, by quadtree decomposition, which iteratively divides the input into a predetermined number of equal size blocks according to variance or blurriness, for example four equal sized blocks. The blurriness of a region can be obtained by averaging the corresponding region in the blurriness map.

[0047] After selection of the predetermined BL candidates, in this example, separate

BL is then selected for each color channel according to the input image. FIG.

2 shows pseudo code for the selection. The code of FIG. 2 describes a preferred background light estimation based on image pixel variance and blurriness. The sigmoid function S in FIG. 2 is given by

[0049] where s is an empirical constant. In an example, s was set to 32. If s is small, the BL estimate is calculated using a smoother weighted combination. FIG.

3 shows a set of curves illustrating (from left to right) curves plotted using s = 8, s = 16, and s =32. Fixed thresholds are set with reference to the input image. For example, v is a threshold (we assign v = e _n ) that determines whether the input image was taken under sufficient lighting. According to the percentage of bright pixels represents the

image has sufficient lighting. The value of can be determined empirically.

Example threshold used in experiments were The quadrant selection lowest variance (QUAD-SELECT-LV) is similar to the quadrant selection with the largest blurriness (QUAD-SELECT-LB), except that largest blurriness is replaced with the lowest variance and there is no need to consider

[0050] The function BL-ESTIMATE in FIG. 2 determines BL for each color channel between the darkest and brightest BL candidates according to the percentage of bright pixels, which for the example is the top

0.5). When the percentage of bright pixels in the input image is high enough then the image is deemed to have sufficient lighting, and BL

estimated as being brighter is more suitable. Generally, BL should be estimated as being brighter (with regard to the channel k) than most (at least a majority and preferably 50-90% or more) of pixels in the input image. The image is determined as having insufficient lighting with and

BL is estimated as darker. BL should be estimated as being darker (with regard to the channel k) than most (at least a majority and preferably 50-90% or more) of pixels in the input image. Between these extremes, the BL estimate can be calculated by a weighted combination of the darkest and brightest BL candidates. This selection process for the BL estimate followed by restoration has demonstrated visually pleasing results. Objective measures also reveal good performance.

[0051] In another modification, both image blurriness and light absorption are leveraged for the depth estimation. This method can be considered to leverage three different depth estimations that are then sigmoidally combined in view of lighting and image conditions. [0052] A first estimate of depth is obtained directly from the red channel. The red channel map R is defined as:

[0053]

[0054] From this, an estimate of depth is obtained:

[0055]

[0056] where F _s is a stretching function. The stretching function F _s can be defined as:

[0057]

[0058] where V is a vector.

[0059] A second estimate of depth is:

[0061] This estimation is based upon maximum intensity prior (MIP). The MIP can be determined through the different between the maximum intensity of the red channel compared to the green and blue channels, as follows:

[0062]

[0063] Large values of represent closer scene points having red light that

attenuates less than that of farther scene points. This concept was applied by others to estimate the transmission map, while it is adapted here to estimate the depth directly.

[0064] A third estimation uses the image blurriness P _r of Equation (7) to estimate depth. The third depth estimation based upon blurriness is defined as:

[0065]

[0066] Combining the three estimates of questions (1 1), (13) and (15) provides for the estimation of depth in a distorted medium based upon light absorption and image blurriness according to the estimated BL B ^c and the average input red value according to the following:

[0067]

[0068] where are determined

by the sigmoid function of Equation (9). After that the depth map is refined and smoothed, such as by soft matting or guided filtering, as discussed above. The resultant estimated depth map d _n E [0,1] can be considered as a map of normalized relative distance for the scene points of the image for any other image enhancement or adjustment processes to be conducted.

[0069] With this third variation, when the image has some reasonable level of red content overall and the background light is relatively

dim, avg alone provides a good depth representation.

In such case,

[0070] As BL gets brighter, the possibility that fails to represent scene depth

gets higher. Because the BL accounts for more of the observed intensity for a scene point farther from the camera, far scene points may still have large values in the red channel and be wrongly judged as being close according to Equation (11). When an underwater image has a brighter BL, then d _D is more reliable for scene depth. The red light of a farther scene point is absorbed more compared to the green and blue light. For this reason, when the image has some reasonable level of red content overallavg

and the background is relatively bright is itself a good

measure of depth. In such case,

[0071] When there is very little red light g( ) then both Equations (13) and (15), which directly use red channel values are likely to fail to estimate scene depth properly. In this case, and mean

that the depth estimation reverts to using the blurriness map alone, between these various extremes, the depth map comes from a weighted combination of the three approaches.

[0072] With the depth map determined, TM can then be calculated and scene radiance can be recovered. Preferred embodiments determine TM by measuring the distance from the camera to each scene point. To measure the distance from the camera to each scene point, the distance do between the closest scene point and the camera must be estimated as well. Via the maximum difference between the estimated and the observed intensities cin the input image, the estimated can be calculated by:

[0074] If the BL accounts for a large

portion of the observed intensities for the closest scene point, the maximum difference would be small, and would be large indicating that the distance

from the camera to the closest object in the scene is long.

[0075] The final scene depth can be determined by combining Equations (16) and

(17) as follows:

[0076]

[0077] where D _∞ is a scaling constant for transforming the relative distance to the actual distance.

[0078] With df, the TM for the red channel can be calculated as:

[0079]

[0080] where for Ocean Type-I water as a distorting medium.

Approximately 98% of the world's open oceans and coastal waters fall into this category so values of β ^τ close to this value represent a preferred embodiment for underwater image restoration. With the red channel TM calculated, the values for the green and blue channels can then be calculated according to relationships between the channels, such as through the residual energy ratios of the color channels, as described by Chiang et al in the context of a wavelength compensation and image dehazing method. J. Y. Chiang and Y.-C. Chen, "Underwater image enhancement by wavelength compensation and dehazing," IEEE Trans. Image Process, vol. 21, pp. 1756- 1769, (Apr. 2012). Chiang et al. chose ratios manually, and predetermined ratios can also be selected in preferred embodiments. Another option is automatic calculation. A 2015 publication provides relations among the attenuation coefficients of different color channels based on inherent optical properties of water that are derived from the BL. X. Zhao, J. Tao, and Q. Song. "Deriving inherent optical properties from background color and underwater image enhancement," Ocean Eng., vol. 94, pp. 163-172, (Jan. 2015). The relations are determined as:

[0082] where λ° , c e {r, g, g} represent the wavelengths of the red, green, and blue channels, and m = —0.00113, and i = 1.62517. The TMs for the green and blue lights are then calculated by

[0083]

[0084] where t ^r is taken from Equation (19) in this preferred embodiment. Testing has also shown the preferred embodiments perform well compared to prior systems and techniques, as indicated by qualitative and quantitative measurement results. Quantitative and qualitative measurement results of the preferred embodiments are reported in Peng and Cosman, "Underwater Image Restoration based on Image Blurriness and Light Absorption," IEEE TRANSACTIONS XXX, VOL. X, NO. X, XXX (2016) and in Peng, Zhao and Cosman, "Single Underwater Image Enhancement using Depth Estimation based on Blurriness," in Proc. IEEE Int. Conf. on Imag. Process. (ICIP), pp. 4952-4956, (Sep. 2015). Experiments discussed above restored images according to Equation (8) in that paper. Specifically,

Then, stretch to a proper range where rO is

set to [0.3, 0.5] and rl is 0.9, empirically, denoted as t(x). Then, use to calculate the enhanced image J(x).

[0085] While specific embodiments of the present invention have been shown and described, it should be understood that other modifications, substitutions and alternatives are apparent to one of ordinary skill in the art. Such modifications, substitutions and alternatives can be made without departing from the spirit and scope of the invention, which should be determined from the appended claims.

[0086] Various features of the invention are set forth in the appended claims.