RELATED APPLICATIONS

[0001] This application claims priority to U.S. Patent Application No. 61/993,469, filed May 15, 2014. titled "Method and System for the Estimation of Different Types of Noise In Iroage and Video Signals", the entire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

[0002] The present invention relates generally to image and video noise analysis and specifically to a method and system for estimating different types of noise in image and video signals.

BACKGROUND

[0001] Noise measurement is an essential component of many image and video processing techniques (e.g., noise reduction, compression, and object segmentation), as adapting their parameters to the existing noise level can significantly improve their accuracy. Noise is added to the images or video from different sources [References 1-3] such as CCD sensor (fixed pattern noise, dark current noise, shot noise, and amplifier noise), post-filtering (processed noise), and compression (quantization noise).

[0002] Noise is signal-dependent due to physical properties of sensors and frequency- dependent due to post-capture filtering or Bayer interpolation in digital cameras. Thus, image and video noise is classified into: additive white Gaussian noise (AWGN) that is both frequency and signal independent, Poissonian-Gaussian noise (PGN) that is frequency independent but signal-dependent, i.e., AWGN for a certain intensity, and processed Poissonian-Gaussian noise (PPN) that is both frequency and signal dependent, e.g., non-white Gaussian for a particular intensity.

[0003] Many noise estimation approaches assume the noise is Gaussian, which is not accurate in practical video applications, where video noise is signal-dependent Techniques that estimate signal-dependent noise, on the other hand, do not handle Gaussian noise.

Furthermore, noise estimation approaches rely on the assumption that high frequency components of the noise exist, which makes them fail in real-world non-white (processed) noise. This is even more problematic in approaches using small patches (e.g., 5 x 5 pixels) [References 4-9] because the probability to find a small patch with a variance much less than the noise power is higher than in large patch.

BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Embodiments of the invention or inventions are described, by way of example only, with reference to the appended drawings wherein:

[0005] FIG. 1 is an example embodiment of a computing system and modules for an imaging pipeline.

[0006] FIGs 2(a) and 2(b) are examples of images captured with the same camera in a raw mode and in a processed mode respectively. FIGs 2(c) and 2(d) show the average of noise frequency magnitudes of 35 different images taken by 7 cameras in a raw mode and in a processed mode, respectively.

[0007] FIG. 3(a) and 3(b) respectively show example noise level function (NLF) approximations for two sample images and their corresponding NLF in RGB channels. FIG. 3(c) show a piecewise linear modeling of NLF.

[0008] FIG. 4 is an intra-frame block diagram of the estimator operating spatially within one image or video frame.

[0009] FIG. 5 is an inter-frame and intra-frame block diagram of the estimator operating spatio-temporal in a video signal.

[0010] FIG. 6 is an example image showing different intensity classes of target patches and the corresponding connectivity.

[0011] FIG. 7 is an example image showing selected weighted clusters in different intensity classes.

[0012] FIG. & is an example graph showing low-to-high frequency power ratios of homogeneous regions in raw and processed images taken by 7 different cameras.

[0013] FIG. 9(a) is an example graph showing a relation between the filter strength and low-to-high average frequency power ration. FIG. 9(b) is an example graph showing linear approximation using the low-to-high ration.

[0014] FIG. 10 is an example graph of an NLF approximation.

[0015] FIG. 11 is a set of 14 test images for an additive white Gaussian noise (AWGN) test. [0016] FIGs. 12(a) and (b) are example images used in homogeneity selection under AWGN.

[0017] FIG. 13 is an example graph showing stability of the proposed method in video signal under AWGN with and without temporal weights.

[0018] FIG. 14 shows examples of 7 real-world test images.

[0019] FIGs. 15(a) and 15(b) are examples of homogeneity selection for real Poissonian- Gaussian noise (PGN).

[00201 FIGs. 16(a) - 16(c) are a set noise removal examples using BM3D- FIG. 16(a) are original images. FIG. 1 (b) shows images processed using noise estimated according to [Reference 7]. FIG. 16(c) shows images processed using noise estimated according to IVHC.

[0021] FIG- 17 is an example graph showing MetricQ of real noise removal using different noise estimators for In-to-tree sequence.

[0022] FIG. 18 is an example graph showing processed synthetic noise in a video in peak signal-to-noise ratio (PS R).

[0023] FIGs 19(a) to 19(d) are a set noise removal examples using BM3D.

[0024] FIGs 20(a)-20(d) are example graphs of estimated NLFs with respect to SRxlOOIL Intotree, Salpha77, and Sintel.

[0025] FIG. 21 is a table showing example results for averages of absolute errors using test images in FIG. II.

[0026] FIG. 22 is a table of MetricQ comparison of PGN removal.

[0027] FIG. 23 is a table of real-world processed noised removal results according to average MetricQ using BM3D.

[0028] FIG.24 is a table of root mean square error (RMSE) values and maximum values of error of LF in noise images.

[0029] FIG. 25 is a table of the average of elapsed time to process the test images.

DETAILED DESCRIPTION

[0030] It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the example embodiments described herein. However, it will be understood by those of ordinary skill in the art that the example embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the example embodiments described herein. Also, the description is not to be considered as limiting the scope of the example embodiments described herein.

[0031] A method and a system are provided for the estimation of different types of noise in images and video signals using preferably, intensity-variance homogeneity classification as will be described herein.

[0032] Fig. 1 is an example embodiment of a computing system 101 with components for a CCD (charge-coupled device) camera pipeline. The computing system 101 includes a processor 102, memory 103 for storing images and executable instructions, and an image processing module 104.

[0033] The computing system 101 may also include a camera device 106, or may be in data communication with a CCD or camera device 100. In an example embodiment, the computing system also includes, though not necessarily, a communication device 107, a user interlace module 108, and a user input device 110.

[0034] Throughout this sensing pipeline as best seen by module 104, noise is added to the image from different sources, including but not limited to a CCD sensor, creating noises such as fixed pattern noise, dark current noise, shot noise, and amplifier noise, post filtering (processed non-white noise), and compression (quantization noise), which render a digital image 206. Referring to Fig. 1, r w sensor data is collected and passes through lens correction 201. The lens corrected data then undergoes Bayer interpolation 202, white balancing 203, post filtering 204 and finally compression 205 before being rendered as a digital image 206.

[0035] In a non-limiting example embodiment, the computing system may be a consumer electronic device, such as a camera device. In other words, the electronic device may include a physical body to house the components. Alternatively, the computing system is a computing device that is provided with image or video feed, or both.

[0036] It will be appreciated that any module or component exemplified herein that executes instractions or operations may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for escarnple, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage of informatiori, such as computer readable instructions, data structures, program modules, or other data, except transitory propagating signals per se. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the computing system 101, or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions or operations that may be stored or otherwise held by such computer readable media.

[00371 The proposed systems and methods are configured to perform one or more of the following functions:

• operate on a still image or a video signal;

• operate on gray-scale as well as color image or video;

• estimate the noise variance of AWGN, PGN, and PPN automatically;

• estimate the noise level function (NLF), e.g., the relation between the noise variance and the intensities of the input noisy signal;

• temporally stabilize the current estimate using estimates from previous frames;

• differentiate noise from image structure by relating the input noisy signal and its down- sampled version;

• adapt the patch size for intensity classification using both the input noisy signal and its down-sampled version;

• rank noise representative regions (clusters) based on intra-image (spatial) features including intensity, spatial relation (connectivity and neighborhood dependency), low- high frequency relation, size, and margins; • rank noise representative regions based on inter-image (temporal) features including temporal difference between patch signal in neighboring frames and difference between current estimate and estimates from previous frames;

• rank noise representative regions based on camera and capture settings, if they are available as metadata; and

• rank noise representative regions based on manual user input in offline applications such as post production.

[0038] These features extend beyond [Reference 10], as the proposed systems and methods additionally a) estimate both the noise variance and the NLF; b) estimate both processed and unprocessed noise; and c) broadens the solution by adding many new features such as using temporal data. As a result, the performance significantly improved compared to [Reference 10].

[0039] 1. NOISE MODELING

[0040] 1.1 White noise

[0041] The input noisy video frame (or still image) I can be modeled as, I = n _d + n _g + n _q , where I _org represents the noise-free image, n _d represents white signal-dependent noise, n _g represents white signal-independent noise, and n _q represents quantization and amplification noise. Wrth modern camera technology n ₉ can be ignored since it is very small compared to rt = n _d + n _g . n _d and« _e are assumed zero-mean random variables with variance (I) and σ , respectively. (For simplicity of notation, the symbol I is herein used to refer to either a whole image or to an intensity of that image; this will be clear from the context.) The NLF of the image intensity I can be assumed,

[0042] The computing system defines = max (cr ² © ) as the peak of «7^(1) . When a video application, e.g. _} motion detection, requires a single noise variance, the best descriptive value is the maximum level, since a boundary can be effectively designated to discriminate between signal and noise. In (15)., the computing system estimates σ _ρ ² as the peak of the level function of the observed video noise, which can be AWGN, PGN, or PPN. Under PGN, the peak variance is al which becomes er* as estimated in (15); under PPN, the peak variance is estimated from σ* using (2). [0043] 1.2 Processed noise

[0044} Processing technologies such as Beyer pattern interpolation, noise removal, bit- rate reduction, and resolution enlargement, are being increasingly embedded in digital cameras. For example, spatial filtering is used to decrease the bit-rate. Accurate data about in-camera processing is not available, in many cameras, however, processing can be bypassed manually, which allows to explore statistical properties of noise before and after processing. Experiments show that the low-power high frequency components of the noise (compared to noise power) are eliminated. As a result, low f equency and impulse shaped noise remains. Fig. 2 shows parts of two images taken under the same condition in raw and processed image mode. This figure also shows the frequency spectrum of noise in both modes. The noise was studied using homogeneous image regions that were manually selected from 35 images taken by 7 different cameras (e.g. Canon EOS 6D ₅ Fujifilm xlOO, Nikon D700, Olympus E-5, Panasonic LX7, Samsung NX200, Sony RX100). As can be seen, filtering changes the frequency spectrum of the noise and makes it processed (e.g.

frequency dependent). In many video processing applications, estimation of the noise level before the in-caraera filtering is desirable for accurate processing. It is herein recognized that such estimation is challenging since some of noise frequency components are removed and calculation of the pre-processing (original) noise level by its current power (e.g., variance of homogeneous patches) is no longer accurate,

[0045) When PGN becomes processed, the resulting noisy image can be modeled as I = lors ^{+ n} p ^fth ⁿ _P as the PPN and peak variance σ _ρ ^ζ . The before in-camera processing image I is modeled as l~I„ + "^with n _Y as the distortion noise and peak variance c^ .

The method thus differentiates here between PGN n _a , PPN n _p , and distortion noise η _γ , where

n _B - n _p + n _T . Let 1≤ γ < be the degree (power) of processing on . The method estimates,

(2) [0046] γ = 1 means the observed noise is PGN; means Iwas not heavily processed, as shown in Fig. 9. Heavily processed means the nature of PGN was heavily changed resulting in large compared to since the mean absolute

difference of Iand I _p is large.

[0047J 1.3 Noise level function

[0048} A better adaptation of video processing applications to noise can be achieved by considering the NLF instead of a single value. It is herein recognized however, that there is no guarantee that pure noise (signal-free) pixels are available for all intensities, and thus NLF estimation is challenging. The NLF strongly depends on camera and capture settings

[Reference 11] as illustrated in Fig. 3.

[0049] Assume the computing system divides the intensity range of the input noisy image I into M sub-intensity classes. A piecewise linear function;, see Fig. 3(c), can approximate the NLF in intensity class / as follow,

[0050] where define the intensity class boundaries, represents a point of and is its corresponding intensity, is, for example, the median of in (3) represents the slope of a line approximating the

NLF in the intensity class 1 as illustrated in Fig. 3. If Mis appropriately selected (not too many nor too few classes), α ₁ will not exceed _max ≥ max . The computing system uses to locate patches that fit into linear model of the NLF. Equation (3) states that given o

computing system can reject non-homogeneous patches that their variances are greater than this can thus be used to target homogeneous patches, as shown below.

[0051] 2. State-of-the-art

[0052] A GN estimation techniques can be categorized into filter-based, transform- based, edge-based, and patch-based methods. Filter-based techniques [Reference 12], [Reference 13] first smooth the image using a spatial filter and then estimate iJie noise firom the difference between the noisy and smoothed images. In such methods, spatial filters are designed based on parameters that represent the image noise. Transform (wavelet or DCT) based methods [References 14-20] extract the noise from the diagonal band coefficients. [Reference 19] proposed a statistical approach to analyze the DCT filtered image and suggested that the change in kurtosis values results from the input noise. They proposed a model using this effect to estimate the noise level in real-world images. It is herein recognized that although the global processing makes transform-based methods robust, their edge-noise differentiation lead to inaccuracy in low noise levels or high structured images.

[0053] [Reference 19] aims to solve this problem by applying a block-based transform. [Reference 20] uses £*lf-similarity of image blocks, where similar blocks are represented in 3D form via a 3D DCT transform. The noise variance is estimated from high-frequency components assuming image structure is concentrated in low frequencies. Edge-based methods [Reference 11, Reference 21, Reference 22] select homogeneous segments via edge- detection. In patch-based methods [References 6-9], noise estimation relies on identifying pure noise patches (usually blocks) and averaging the patch variances.

[0054] Overall local methods that deal with subsets of images (i.e. homogeneous segments or patches) are more accurate, since they exclude image structures more efficiently. [Reference 6] utilizes local and global data to increase robustness. In [Reference 7], a threshold adaptive Sobel edge detection selects the target patches, then averages of the convolutions over the selected blocks to provide accurate estimation of noise variance. Based on principal component analysis [Reference 8] first finds the smallest eigenvalue of the image block covariance matrix and then estimates the noise variance. Gradient covariance matrix is used in [Reference 9] to select "weak" textured patches through an iterative process to estimate the noise variance.

[0055] It is herein recognized that patch size is critical for patch-based methods. A smaller patch is better for low level of the noise, while, larger patch makes the estimation more accurate in higher noise level. For all patch sizes, estimation is error prone under processed noise however by taking more low frequency components into account, larger patches are less erroneous. By adapting the patch size in these estimators to image resolution, it as more likely to find noisy (signal-free) patches, which consequently increases the performance. Logically finding image subsets with lower energy under AWGN conditions leads to accurate results. However, under PGN conditions underestimation normally occurs. Under AWGN, [References 7-9] outperform others, however, it is herein recognized that noise underestimation in PGN makes them impractical for real-world applications.

[0056] PGN estimation methods express the noise as a function of image brightness. The main focuses of related work is to first simplify the variance-intensity function and second to estimate the function parameters using many candidates as fitting points. In [Reference 4], [Reference 23], the NLF is defined as a linear function σ ² (/) al + b and the goal is to estimate the constants a and b. Wavelet domain [Reference 4] and DCT [Reference 23] analysis are used to localize the smooth regions. Based on the variance of selected regions, each point of curve is considered to perform the maximum likelihood fitting. [Reference 24] estimates noise variation parameters using maximum likelihood estimator. It is herein recognized that this iterative procedure brings up the initial value selection and convergence problems. The same idea is applied in [Reference 11] by using a piecewise smooth image model.

[0057] After image segmentation, the estimated variance of each segment is considered as an overestimate of the noise level. Then the lower envelope variance samples versus mean of each segment is computed and based on that, the noise level function by a curve fitting is calculated. In [Reference 25], particle filters are used as a structure analyzer to detect homogeneous blocks, which are grouped to estimate noise levels for various image intensities with confidences. Then, the noise level function is estimated from the incomplete and noisy estimated samples by solving its sparse representation under a trained basis. The curve fitting using many variance-intensity pairs, requires enormous computations, which is not practical for many application especially when the curve estimation is needed to be presented as a single value. As a special case of PGN with zero dependency, AWGN cases are not examined in these NLF estimation methods, In [Reference 26], a variance stabilization transform (VST) converts the properties of the noise into AWGN. Instead of processing the Gaussianized image and inverting back to Poisson model, a Poisson denoising method is applied to avoid an inverted VST.

[0058] PPN is not yet an active research and few estimation methods exist. In [Reference 27], first, candidate patches are selected using their gradient energy. Then, the 3D Fourier analysis of current frame and other motion-compensated frames is used to estimate the amplitude of noise. A wider assumption is in [Reference 28] by considering both frequency and signal dependency, I n this method, the similarity between patches and neighborhood is the criterion to differentiate the noise and image structure. Using the exhaustive search, candidate patches are selected and noise is estimated in each DCT coefficient

[0059] 3. Proposed systems and methods

[0060} The proposed systems and methods are based on the classification of intensity- variances of signal patches (blocks) in order to find homogeneous regions that best represent the noise. It is assumes that noise variance is linear, with limited slope, to the intensity in a class. To find homogeneous regions, the method works on the down-sampled input image and divides it into patches. Each patch is assigned to an intensity class, whereas outlier patches are rejected. Clusters of connected patches in each class are formed and some weights are assigned to them. Then, the most homogeneous cluster is selected and the mean variance of patches of this cluster is considered as the noise variance peak of the input noisy signal. To account for processed noise, an adjustment procedure is proposed based on the ratio of low to high frequency energies. To account for noise variations along video signals, a temporal stabilization of the estimated noise is proposed. The block diagram in Fig.4 shows how the proposed method estimates the noise within one image or video f ame without temporal considerations. Fig. 5 shows how the method is stabilized using temporal processing in video. The proposed noise estimation based on intensity-variance homogeneity classification (IVHC) can be summarized as in Algorithm 1. In the remainder of this section, a discussion of the following is included: building homogeneous patches; classifying patches; building clusters of connected patches and estimating the noise peak variance; estimating parameters of processed noise; approximating the NLF; temporally stabilizing the estimate; computing intra-frame and inter-frame weights; adapting to camera settings; and showing how to adapt the method to user input in offline applications.

[0062] 3.1 Homogeneity guided patches [0063] Homogeneous patches are image blocks i i size W * W

where is the down-sampled version of the input noisy image at the spatial location (x,y), mod( )is the modulus after division, and r is the image height (number of rows). After decomposing the image into non-overlapped patches, the noise of each patch can be described as where is the observed patch corrupted by independent and identically-distributed zero-mean Gaussian noise and is the original non-noisy image patch. The variance of a patch represents the level of homogeneity

[0064] A small expresses high patch homogeneity. Under PGN conditions, noise is i.i.d for each intensity level. If an image is classified into classes of patches with same intensity level, the homogeneity model can be applied to each class. Assuming M

intensity classes, represents the patches of the 1th intensity class,

[0065]

defining lower and upper bounds of class intensity.

[0066] 3.2 Adaptive patch classification

[0067] Images contain statistically more low frequencies than high frequencies. But small image patches show more high frequencies than low frequencies. Thus small patches have the advantage of better signal-noise differentiation. Large image patches,, on the other side, are less likely to fall in the local minima, especially when noise is processed. To benefit from both, the computing systems uses image downscaling with rate R with a coarse averaging as the anti-aliasing filter,

[0068] where J and Ϊ are the observed and down-sampled images. This gives small patches in Γ and large patches in I. Furthermore, the processed noise converges to white in the downscaled image. Other desirable effects of downscaling are: 1) noise estimation parameters can be fixed for a lowest possible resolution of the images (note that R varies depending on the input image resolution) and 2) since the down-scaled image contains more low frequencies, the signal to noise ratio is higher. Assuming £ represents the set of patches in ; the computing system binary classifies the patches of the fth intensity class in T into , where ¾ are the target patches as in,

[0069] It uses the homogeneity values and a threshold value to binary classify . Assuming the maximum value of the slopes of the We define

(9)

[0070] where β = 1 and amax = 3. To calculate H _med (i) _> the computing system first divides into three sub-classes, then finds the minimum in each sub-class and finally finds the median of the three values. When class contains overexposed or underexposed patches, becomes very small. Therefore, the offset β is considered to include noisy patches. Fig. 6 shows sample target patches and their connectivity with = 4. Spatial information from horizontal and vertical connectivity can be used to form patch clusters as explained next.

[0071] 3.3 Cluster selection and peak variance estimation

[0072J Due to complexity of noise and image structure, the variance based classification (8) by itself does not describe the noise in the image. In addition to statistical analysis, the computing system uses a spatial analysis to extract a more reliable noise descriptor. The computing system uses connectivity of patches in both horizontal and vertical directions to form clusters of similar patches. Next, for each cluster of connected patches in the down- sampled image the computing system first finds the corresponding connected patches B, (with size of from the cluster in the input noisy image

I and then eliminate the outliers of cluster based on their mean and variance. Finally, the computing system assesses each cluster (after outlier removal) based on the intra- and inter-frame weights represents the kth cluster of connected patches in the class

before outlier removal.

[0073J 3.3.1 Outlier removal [0074] The removal of outliers in each cluster is based on Euclidean distance of both the mean and the variance. For each cluster the patch with higher probability of homogeneity is defined as the reference patch and patches out of certain Euclidean distance are removed. Assuming Φ (I, k) represents the kth cluster of connected patches in the class I before outlier removal, the computing system defines the reference value of variance and mean of each cluster as, where is the patch with the minimum variance in and its variance and mean are considered references. By defining two intervals using two thresholds,

the cluster after outlier removal is,

(11 where an are the variance and the mean thresholds that are directly

proportional to as.

Where

To avoid including image structure in the clusters, the similarity of the patches is considered and in (12) we replace with defined as,

[0075] 3.3.2 Cluster ranking (0076] For each outlier-reduced connected cluster (7, k) the computing system first computes the weights wj (/, k) and then selects the final homogeneous cluster Φ as in,

(14)

Then the computing system defines the peak noise level σ _ρ ^ζ in the input image as the average of the patch variances in Φ the cluster ranked highest, e.g., best represents random noise,

(15)

where N{<J>} is the number of patches in the cluster Φ. The value σ _ρ ² is considered as the peak variance because the computing system gives higher weights to cluster with higher variances. Estimates of {0 < c¾ (/, k) < 1} are proposed in the below, where it considers noise in both low and high frequencies, size of the cluster, patch variances, intensity and variance margins, maximum noise level, clipping factors, temporal error, and previous estimates. Fig. 7 shows selected weighted clusters in different intensity classes.

[0077] 3.4 Processed noise estimation

[0078] It is herein recognized that the assumption that the noise is frequency-independent in each homogeneous cluster is incorrect in processed images. In such situations, the variance of selected cluster (15) does not represent the true level of the noise in the unprocessed noisy image because some frequency components of the noise have been removed. In many applications such as enhancement, the level of the unprocessed (original) noise is required. To estimate this original noise, the relation between low and high frequency components is necessary to trace the deviation from whiteness because the computing system assumes that the degree of noise removal in high frequency and low frequency is different. Let E(L ) represent the variance of low-pass filtered pixels of Φ (1, k) and E(Hf) represent the median of the power of high-pass filtered pixels of Φ A). The computing system estimates their relation as follows, [0079] where * is convolution, is a 3 x 3 moving average filter, and the

high-pass filter with a 3 kernel of zero elements except the center is one. With the given low- pass filter 3.7. The ratio E _r increases with spatial filtering occurs. The computing system selects as the median energy because high-frequency noise after filtering has an impulse shape and is divided into high and low levels. In many cameras, the filtering process is optional, allowing for study of the effect of this filtering on processed noise. Fig. 8 shows the low-to-high ratio of homogeneous regions in different raw and processed images. The more noise deviates from whiteness, the higher E _r becomes.

[0080] To approximate the processing degree γ of (2), the effect of applying anisotropic diffusion [Reference 29] and bilateral filters [Reference 30] on synthetic AWGN is considered. Fig. 9 shows the relation between I and how Er relates to γ . It is herein therefore proposed to use linear approximation of a function of as in,

[0081] The computing system temporally stabilizes γ using the procedure discussed in section 3.6. As can be seen in Fig. 9(b) at the approximation becomes less accurate.

[0082] 3.5 Noise level function approximation

(0083] The computing system estimates the NLF based on the peak noise variance of the selected cluster defined in (15) and employs other outlier-removed clusters to approximate the NLF. First, the computing system sets all the initial NLF curve ¾ (.) to cr , which means the noise level is identical in all intensities (Gaussian). Then, the computing system updates the Ω (.) based on N{ Φ (/, k)} the size (i.e., number of patches) and on σ ² (I, k) the average of the variances of cluster Φ (/, k). The computing system assigns a weight

(confidence) A (/, k) to cr ¹ (7, k) : the larger N{ (I, £)} is, the better σ ² (I, k) represents the noise at intensity μ (I, k), meaning the closer λ ft k) should be to 1. The point-wise NLF _! (.) is then,

i¾ _e divisor constant 5 is considered according to 3σ rule by

considering that a cluster with 15 (or more) patches is completely reliable i.e., A (/, k) - 1. By applying a regression analysis, e.g., curve fitting, the continuous NLF can be approximated from as illustrated in Fig. 10 using polyfit of Matlab. In case of AWGN,

When PGN gets processed the NLF points are reduced

by factor γ but the normalized NLF shape is not altered. s

in (2) under PGN of each cluster, the proposed method can estimate the NLF whether the noise is processed or -white.

[0084] 3.6 Temporal stabilization of estimates

[0085] In many video applications, instability of noise level is intolerable, unless the temporal coherence between frame is very small e.g., a scene change. Let represent the similarity between the current I _t and previous frame determines how the

statistical properties of new observation (i.e. _; image) are related to previous observations. Consider a process (such as median) to filter out outliers from the set of current and previous estimates , the accurate estimate should

where, is the stabilized final noise variance for frame

time /. he stabilization process in (1 ) can be perform

and ¾

[0086] 3.7 Infra-frame weighting

[0087} 3.7,1 Noise in low frequencies

[0088] Image signal is more concentrated in low frequencies, however noise is equally distributed- Down-sampled versus input images can be exploited to analyze noise in the low- frequency components. The variance of finite Gaussian samples follows a scaled chi- squared distribution. But here the computing system utilizes an approximation benefiting the normalized Euclidean distance,

where exp(.) symbolizes the exponential function, ² and cr ² (7, k) are the average of variances of the input and down-sampled patches in the cluster after outlier removal Φ (I, k). The positive constant C _\ (e.g., 0.4) varies depending on the R and the W. Low values of o _t (l, k) account for image structure, which the signal is concentrated in low frequencies.

[G089J 3.7-2 Noise in high frequencies

[0090] The dependency of neighboring pixels is another criterion to extract image structure. The median absolute deviation (MAD) in the horizontal, vertical and diagonal directions expresses this dependency,

0 < m,n < R - W - 2 (21) where T _t - is the MAD of For a block of Gaussian samples, with the block size 10 < i? · W 25, ffgi— 1.1T _e . The computing system profits from this property to extract the likelihood function of neighborhood dependency. Assuming for each Φ (/, k\ τ (/, k) is the average of Tiof the blocks in the Φ (l _r it). Under AWGN, the following likelihood function is defined,

(22)

where Cj = 0.2. Low values of ω _ζ (J, k) mean a strong neighboring dependency, which is a hint of image structure. In case of white noise, the computing system analyzes the MAD versus variance to estimate if the patch contains structure. Thus, in final estimation step, the computing system uses 1.1 r ² (/, k) instead of o ^z (I, k) for patches with structure.

(00911 3.7.3 Size of the cluster

[0092] The target patches are more concentrated in homogeneous regions and the size of the homogeneous region should be large enough to precisely represent the noise statistics. Therefore, larger cluster has a higher probability of presenting the homogeneous regions. However, a linear relationship between cluster size and the corresponding weight is not advantageous, since once it is past a certain size, sufficient noise information can be obtained. The following is proposed for with respect to the weight for the size of the cluster,

image, respectively.

[0093] 3-7.4 Variance of means and variance of variances

[0094] In a homogeneous cluster with relatively large number of pixels in each patch, the normalized value of the variance of variances v l, fc)and variance of means It) of

[0095] 3.7.5 Intensity margins

[0096] Excluding the intensity extremes from the estimation procedure can be problematic when the signal margins are informative. For instance, the elimination of dark intensities in an underexposed image leads to the removal of the majority of data and, consequently, inaccurate estimation. It therefore herein proposed to use negative weights to margins,

[0097] 3.7.6 Variance margins

[0098] There are cases where underexposed or overexposed image parts with verylow variances are not observed in the intensity margins. On the other hand, extremely high variances signify image structure. For consumer electronic related applications, the PS usually is not below a certain value (e.g., 22dB). Thus, similar to intensity margins, variance margins also affect the homogeneity characterization. It is therefore proposed to use the following weight, .

[0099] 3.7.7 Maximum noise level

[00100 Under PGN, the maximum noise level distinguishes the signal and noise boundary. Hence, the maximum noise level and the corresponding intensity can be used to estimate the NLF, As a result, the Φ(l,k) with the maximum level of the noise should be ranked higher. However, some consideration should be taken into account in order to exclude clusters containing image structures for this weighting procedure. The basic assumption that noise variance slope is limited helps to restrict the maximum level of noise in each intensity class. So,

[00101] 3.7.8 Clipping factor

[00102] Due to bit-depth limitations, the intensity values of the input images are clipped in low and high margins. It is proposed to use a weight according to 3σ bound,

(00103] 3.8 Inter-frame weighting

[00104J Utilizing only spatial data in video signals may lead to estimation uncertainty, especially in processed noise, where the relation between low and high frequency components deviates from AWGN, which in turn makes structure and noise differentiation more challenging- Another issue to consider in video is robust estimation over time especially in joint video noise estimation and enhancement applications.

[001051 3.8.1 Temporal error weighting

[00106] Assume is /th patch in the noisy frame at time and is corresponding patch in the the adjacent noisy frame at time where Based on which adjacent frame (previous or following) has less temporal error for whole frame Assuming the noise level does not change through time the matching (or temporal consistency) factor can be defined as, (

[OQ107J where is the kth connected cluster of class

Since the homogeneity detection is applied on the input noisy image, there is no guarantee that the temporal is also homogeneous. Therefore, high temporal error of

few patches should not significantly affect For this, the computing system analyzes each patch error and aggregates all matching degrees. This is more reliable than assessing the aggregated variances.

(00108] 3.8.2 Previous estimates weighting

[00109] In video applications, noise estimation should be stable through time and coarse noise level jumps are only acceptable when mere is a scene (or lighting) change. Therefore, the cluster with the variance closer to previous observation is more likely to be the target cluster. Assuming is the estimated noise for the previous frame, the following is defined to add temporal robustness,

where measures scene change estimated at patch level.

Assuming the temporally matched patches have the mean error less than the ratio of temporally matched patches to the whole patches defines the Note that (32) guides the estimator to find the most similar homogeneous region in [00110] 3.9 Camera settings adaptation

[00111] For a specific digital camera, the type and level of the noise can be desirably modeled using camera parameters such as ISO, shutter speed, aperture, and flash on of However, creating a model for each camera requires an excessive data processing. Also such meta-data can be lost for example, due to format conversion and image transferring, Thus, the computing system cannot only rely on the camera or capturing properties to estimate the noise; however, these properties, if available, can support the selection of homogeneous regions and thereby increase estimation robustness. It is assumed the camera settings give probable range of noise level. Patch selection threshold H _th (I) in (9) can be modified according to this range. The computing system can also use variance margin weights in (27) to reject out of range values.

[00112] 3.10 User input adaptation

[00113] In some video applications such as post-production, users require manual intervention to adjust the noise level for their specific needs. Assuming user knowledge about the noise level can define the valid noise range, the variance margin used in (27) can be used to reject the out of range clusters.

[00114] 4. Experimental results

[00115] The down-sampling rate R is a function of image resolution. For example, R - 2 for low resolution (less than 720p) and R = 3 for higher resolutions. As a result, noise estimation parameters become resolution independent. In an example embodiment, the down-sampled patch size is set to 5. The number of classes was set to M=4. This is because a too high number M causes the classes to be too small and their statistics invalid. AH constant parameters used in the proposed weights are given and explained directly after their respective equations. The same set of values was used in all the results described herein.

[00116] The proposed homogeneous cluster selection can be performed either on one channel of a color space or on each channel separately. Normally the Y channel is less manipulated in capturing process and therefore noise property assumptions in it are more realistic. Observation confirms that adapting the estimation to Y channel leads to better video denoising. Therefore, the estimated target cluster is used in the Y as a guide to select corresponding patches in chroma. Utilizing these patches, the computing system calculates the properties of chroma noise, i.e.. γ and according to (15) and (17)- Due to space constraint, simulation results here are given for the Y channel.

[00117] Target patches in (8) can be recalculated in a second iteration by adapting the Hmin D ^to 0 ^" ! (estimated in first iteration). A finer estimation can be performed by limiting the bound meaning smaller value for <x _max - The rest of the method is the same as in the first iteration. The complexity of a second iteration is very minor and much less than the first one since patch statistics are already computed. However, tests show that a second iteration improves the estimation results slightly, not justifying iterative estimation.

[0011$] Next, the performance of the proposed estimation of the NLF, AWGN, PGN, and, PPN has been evaluated separately.

100119] 4.1 Additive white Gaussian noise (AWGN)

[00120] Six state-of-the-art approaches [References 5-9], [Reference 19] are selected and their performance is evaluated on 14 test images as in Fig.l 1. Noisy images were generated by adding a zero-mean AWGN to the ground-truth, with 4 levels of standard deviation, from 4 to 16 with the step of 4 and the computing system ran 10 Monte- Carlo experiments for each noise level. Table I (see Fig. 21) demonstrates mean of absolute errors of related and proposed method which outperforms. The average variance of the error for our method compared to related methods is similar and is not given here. Method [Reference 8] and [Reference 9] give the closest results. Fig.12 also shows examples of selected homogeneous clusters.

[00121] The proposed method in video signals was also tested and Fig. 13 shows average result of noise estimation with and without using temporal data for the first 100 frames of two sequences. Collaboration of inter-frame weighting (31), (32) and temporal stabilization (19) improves the estimation. In this figure, a comparison to [9] is shown as closest related work from Table I of Fig 21.

[00122] 4.2 Poissonian-Gaussian noise (PGN)

[00123] To evaluate the performance of the proposed estimation of PGN, six state-of-the- art approaches [References 5-9, [Reference 19] were tested on seven real-world test image. See FigJ4. In particular, irttotree from SVTHD Test Set, tears from Mango Blender and five other real-world noisy images were taken in raw mode, where noise is visibly signal- dependent. To objectively evaluate the PNG estimator without a reference frame, the computing system combined the denoising method BM3D [Reference 31] with noise levels provided from the proposed method and related estimators. The output performance is verified through the no-reference quality index MetricQ [Reference 32], Table H (see Fig. 22) compares MetricQ of denoised images with a higher value indicating better quality. The proposed method yields higher quality than related methods, where [Reference 6] and

[Reference 1 ] achieve closest results. WHC avoids underestimation by selecting the cluster with higher variance. Fig.l 5 shows examples of selected homogeneous clusters and Fig.16 shows visual comparison of noisy and noise-reduced image parts. As can be seen, by using IVHC noise is better removed.

[00124] The proposed PGN estimator described herein is also evaluated to denoise video signals using BM3D. Fig. 17 confirms the better quality of our method compared to closest related methods (from Table II) for 150 frames of the intotree sequence.

[00125] 43 Processed Poissonian-Gaussian noise (PPN)

[00126] If the observed noise is PPN, downscaling has the effect of converging it to white. This in turn leads to better patch selection under processed noise. Moreover, since the proposed method uses a large patch size, it leads to include more low f equencies and more realistic estimation. Fig. 18 shows better performance of the proposed method with λ adjustment in (2), and compared to the related method [Reference 9] (which we selected since it is closest to our method under σ— 8 in Table I). To evaluate the proposed method under real-world processed noise, 6 images were chosen (4 from iPhone 5 and 2 from iPhone 6) and BM3D [Reference 31] was applied using noise levels provided by [Reference 8, Reference 9], and proposed WHC. Table III (see Fig.23) and Fig. 1 show that objectively and subjectively noise is better removed based on IVHC.

[00127] 4.4 Noise level function

[00128] The proposed NLF estimation was applied on images with synthetic and real PGN. The ground-truth for real PGN images has been extracted manually (i.e., subjectively extracted homogeneous regions). Two state-of-the-art methods [Reference 11] and

[Reference 4] are selected for comparison. Fig. 20 shows NLF results and Table TV (see Fig. 24) shows the root mean squared error (RMSE) and the maximum error comparison.

Proposed WHC has a better performance of finding the noise level peak especially when the level is greater in higher intensities (e.g., Intotree signal).

[00129] 4.5 Adaptation to camera settings and to user input [00130] The more image information is provided, the more reliable estimation can be performed. Capturing properties if available as a meta-data can be useful for guiding the cluster selection procedure. To test this, 10 highly-tejctured images taken by a mobile camera were selected (Samsung SS) in the burst mode without motion. First, the ground-truth peak of the noise was manually identified by analyzing the homogeneous patches and temporal difference of burst mode captured images. Second, the proposed noise estimator was applied using only Intra-frame weights and the estimated PSNR when compared the ground truth show an average estimation error of 1.2 dB. In the last step, both the patch selection threshold HthC ⁱⁿ (9) and variance margin weight ω ₇ (1, fc) in (27) were adapted to the meta-data brightness value and ISO. This led to more reliable estimation with average error of 0.34dB in PSNR.

[00131] Performance of image and video processing methods improves if expertise of their users can be integrated. The proposed method easily allows for such integration. For example, if the user of an offline application can define possible noise range, the proposed variance margin (27) can be used to reject the out of range clusters.

[00132] 5. Conclnsion

[00133) Noise estimation methods assume visual noise is either white Gaussian or white signal-dependent. The proposed systems and methods bridge the gap between the relatively well studied white Gaussian noise and the more complicated signal-dependent and processed non-white noises. In one aspect of the systems and methods, a noise estimation method is provided that widens the assumptions using vector of weights, which are designed based on statistical property of noise and homogeneous regions in the images. Based on selected homogeneous regions in the different intensity classes, noise level function and processing degree is approximated. It was shown that this visual noise estimation method robustly handles different type of visual noise: white Gaussian, white Poissonian-Gaussian, and processed (non-white) that are visible in real-world video signals. The simulation results showed better performance of the proposed method both in accuracy and speed.

[00134] 6. References

[00135] The details of the references mentioned above, and shown in square brackets, are listed below. It is appreciated that these references are hereby incorporated by reference.

[00136] [Reference 1] R. Szeliski, Computer vision; algorithms and applications, Springer, 2010. [00137] [Reference 2] Y. Tsin, V. Ramesh, and T. Kanade, "Statistical calibration of CCD imaging process," in Computer Vision ICCV ₇ IEEE Int. Conf. on. IEEE, 2001, vol. 1, pp. 4S0-487.

[00138] [Reference 3] G.E. Healey and R. Kondepudy, "Radiometric CCD camera calibration and noise estimation," Pattern Analysis and Machine Intelligence, IEEE Trans, on, vol. 16, no. 3, pp.267-276, Mar 1994.

[00139] [Reference 4] A. Foi, M. Trimeche, V. atkovnik, and - Egiazarian, "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw data," Image

Processing, IEEE Trans, on, vol. 17, no. 10, pp. 1737-1754, 2008.

[00140] [Reference 5] M. Ghazal and A. Amer, "Homogeneity localization using particle filters with application to noise estimation," Image Processing, IEEE Trans, on, vol. 20, no. 7, pp. 1788-1796, 2011.

[00141] [Reference 6] J. Tian and Li Chen, "Image noise estimation using a variation- adaptive evolutionary approach," Signal Processing Letters, IEEE, vol. 19, no. 7, pp. 395- 398, 2012.

[00142J [Reference 7] Sh.-M. Yang and Sh.-Ch. Tai, "Fast and reliable image-noise estimation using a hybrid approach/ ⁵ Journal of Electronic Imaging, vol. 19, no. 3, pp.

033007-033007, 2010.

[00143] [Reference 8] S. Pyatykh, J. Hesser, and Lei Zheng, "Image noise level estimation by principal component analysis," Image Processing, IEEE Trans, on, vol.22, no. 2, pp. 687- 699, 2013.

[00144] [Reference 9] X. Liu, M. Tanaka, and M. Okutomi, "Noise level estimation using weak textured patches of a single noisy image,** in Image Processing (ICIP), IEEE Int Conf. on, 2012, pp. 665-66&.

[00145] [Reference 10] M. Rakhshanfar and A. Amer, "Homogeneity classification for signal dependent noise estimation in images," in Image Processing (ICIP), IEEE Int. Conf. on, Oct 2014, pp.4271-4275.

[00146] [Reference I I] Ce Liu, R. Szeliski, S.B. Kang, C.L. Zitnick, and W.T. Freeman, "Automatic estimation and removal of noise from a single image," Pattern Analysis and Machine Intelligence, IEEE Trans, on, vol. 30, no. 2, pp. 299-314, 2008. [00147] [Reference 12] T.-A. Nguyen and M.-Ch. Hong, Tiltering-based noise estimation for denoising the image degraded by Gaussian noise," in Advances in Image and Video Technology, pp. 157-167. Springer, 2012.

{0014»] [Reference 13] D.-H. Shin, R.-H. Park, S. Yang, and J.-H. Jung, "Block-based noise estimation using adaptive Gaussian filtering," Consumer Electronics, IEEE Trans, on, vol. 51, no. 1, pp.218-226, 2005.

[00149] [Reference 14] D.L. Donoho and J. . Johnstone, "Ideal spatial adaptation by wavelet shrinkage," Biometrika, vol. 81, no. 3, pp. 425-455, 1994.

[00150] (Reference 15] EJ. Balster, Y.F. Zheng, and RX. Ewing, "Combined spatial and temporal domain wavelet shrinkage algorithm for video denoising," Circuits and Systems for Video Technology, IEEE Trans, on, vol. 16, no.2, pp. 220-230, 2006.

[00151] [Reference 16] J. Yang, Y. Wang, W. Xu, and Q. Dai, "Image and video denoising using adaptive dual-tree discrete wavelet packets," Circuits and Systems for Video Technology, IEEE Trans, on, vol. 19, no. 5, pp.642-655, 2009.

[00152] [Reference 17] M. Hashemi and S. Beheshti, "Adaptive noise variance estimation in Bayes-Slu-ink,'' Signal Processing Letters, IEEE, vol. 17, no. 1, pp. 12-15, 2010,

[00153] [Reference 18] H.H. Khalil, R.OX Rahmat, and ΨΛ. Mah oud, "Chapter 15: Estimation of noise in gray-scale and colored images using median absolute deviation (MAD)," in Geometric Modeling and Imaging GMAI, 3rd Int. Conf. on, July 2008, pp. 92- 97.

[00154] [Reference 19] D. Zoran and Y. Weiss, "Scale invariance and noise in natural images," in Computer Vision, IEEE 12th Int. Conf. on, Sept 2009, pp. 2209-2216.

[00155] [Reference 20] A. Danielyan and A. Foi, "Noise variance estimation in nonlocal transform domain," in Local and Non-Local Approximation in Image Processing LNLA, Int. Workshop on. IEEE, 2009, pp. 41-45.

[00156] [Reference 21] Sh.-Ch. Tai and Sb.-M. Yang, "A fast method for image noise estimation using Laplacian operator and adaptive edge detection," in Communications, Control and Signal Processing ISCCSP, 3rd Int. Symposium on, 2008, pp. 1077-1081.

[00157] [Reference 22] P. Fu; Q. Sun; Z. Ji; Q. Chen, "A new method for noise estimation in single-band remote sensing images," Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th International Conference on , vol., no., pp.1664,1668, 29-31 May 2012. [00158 J [Reference 23] A. Foi, "Practical denoising of clipped or overexposed noisy images*" in EUSIPCO, 16th European Signal Processing Conf, 2008. pp. 1-5.

[00159] [Reference 24] A Jezierska, C. Chaux, J.-C. Pesquet, H. Talbot, and G. Engler, "An EM approach for time-variant Poisson-Gaussian model parameter estimation," Signal Processing, IEEE Trans, on, vol 62, no. 1, pp. 17-30, Jan 2014,

(00160] [Reference 25] J, Yang, Zh. Wu, and Ch. Hon, "Estimation of signal-dependent sensor noise via sparse representation of noise level functions," in Image Processing (ICIP), 19th IEEE Int. Conf. on, Sept 2012, pp. 673-676.

[00161] [Reference 26] X. Jin, Zh, Xu, and . Hirakawa. "Noise parameter estimation for Poisson corrupted images using variance stabilization transforms," Image Processing, IEEE Trans, on, vol. 23, no. 3, pp. 1329-1339, March 2014.

[00162] [Reference 27] A. Kofcaram, D. Kelly, H. Denman, and A. Crawford, 'Measuring noise correlation for improved video denoising," in Image Processing (ICIP), 1 th IEEE Int. Conf. on, Sept 2012, pp. 1201-1204.

[00163] [Reference 28] M. Colom, M. Lebrun, A. Buades, and J.M. Morel "A non- parametric approach for the estimation of intensity-frequency dependent noise," in Image Processmg (ICIP), 21th IEEE Int Conf. on, Oct 2014.

[00164] [Reference 29] P. Perona and J. Malik, "Scale-space and edge detection using anisotropic diffusion," Pattern Analysis and Machine Intelligence, IEEE Trans, on, vol. 12, no. 7, pp.629-639, 1990.

[00165] [Reference 30] C. Tomasi and R. Manduchi, "Bilateral filtering for gray and color images," in Computer Vision, Sixth Int. Conf. on, Jan 1998, pp. 839-846.

[00166] [Reference 31] K_. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3-Dtransfonn-domain collaborative filtering," Image Processing, IEEE Trans, on, vol. 16, no. 8, pp.2080-2095, 2007.

[00167] [Reference 32] X. Zhu and p. Milanfar, "Automatic parameter selection for denoising algorithms using a no-reference measure of image content,'' Image Processing, IEEE Trans, on, vol 19, no. 12, pp. 3116-3132, 2010.

[00168] It will be appreciated that the features of the systems and methods for estimating different types of image and video noise and its level function are described herein with respect to example embodiments. However, these features may be combined with different features and different embodiments of these systems and methods, although these combinations are not explicitly stated.

[00169] While the basic principles of these inventions have been described and illustrated herein it will be appreciated by those skilled in the art that variations in the disclosed arrangements, both as to their features and details and the organization of such features and details, may be made without departing from the spirit and scope thereof. Accordingly, the embodiments described and illustrated should be considered only as illustrative of the principles of the inventions, and not construed in a limiting sense.