[0001] The present disclosure relates in general to image signal processing. Specifically, the present disclosure relates to apparatus and methods for compensating bias introduced by image signal processors (ISPs) or ISP components while effectively reducing noise. More specifically, bias-compensating noise reduction systems and methods are provided for generating high-fidelity images and videos by reducing bias introduced from various ISPs or ISP components including spatial noise reduction filters, HDR interpolation units, and demosaicking units while effectively reducing noise.

[0002] Noise reduction has become an important aspect of image and video capturing systems as cameras and sensors with size of pixel continue to reduce while the availability of digital processing power continues to improve. In general, video noise reduction can be broadly divided into spatial noise filters and temporal noise filters. Spatial filters are known to use neighboring pixels in each video frame to produce each output pixel. Temporal filters are known to use consecutive pixels between frames to produce each output pixel. Spatial and temporal noise reduction filters can be used at the same time to produce better results.

[0003] Spatial noise reduction can be effective for still images, but most existing spatial noise reduction filters result in some form of bias such as smoothing textures and fine structure, or artifacts such as ringing or blockiness, in the final results. When applied to a video, spatial noise reduction can give rise to visible residual temporal variations between frames, which is not visible from a single frame.

[0004] Temporal noise reduction in one of its most common forms comprises averaging or in other ways combining pixels in stationary parts of the input frames. When a temporal filter converges slowly, i.e., few frames are available for combining pixels, the resulting images or videos would produce noise trails. And ghosting artifacts may occur where changing parts of the input frames are incorrectly classified as stationary.

[0005] When a spatial noise filter is used together with a temporal noise filter, they are often referred to jointly as 3-D noise reduction filters or spatiotemporal noise reduction filters. Existing forms of spatiotemporal filters, however, present some similar problems from either its component spatial noise filter or its component temporal noise filter. For example, where spatial filtering is applied first, such a spatiotemporal filter results in certain bias including the smoothing of texture and details. Where temporal filter is applied first, though, motion detection or estimation is not as effective due to noise. The convergence of recursive temporal filter can be slow as well. Where switching spatial and temporal filtering is implemented, on the other hand, motion detection remains not as effective due to noise and convergence of recursive temporal filter remains slow. In such switching spatial and temporal filtering systems, moreover, bias from spatial filtering would persist in non- stationary regions where temporal filter is not effective.

[0006] As is clear in existing systems, therefore, a classical ISP component such as a spatial noise filter may introduce bias in resulting images and videos, and thereby undercut the overall fidelity of a camera or video communication system powered by ISPs. Another example where an ISP introduces bias in the resulting images and videos is a demosaicking unit or debayer unit, which may cause smoothing, zippering artifacts, or false colors in the output frames. Additional examples include HDR (high dynamic range) interpolation on interleaved long and short exposure pixels, which introduces bias of its own.

[0007] There is, consequently, a need for improved methods and systems to reduce or compensate bias introduced by ISPs in camera and video communication systems and to generate high-fidelity images and videos. SUMMARY OF THE VARIOUS EMBODIMENTS

[0008] It is therefore an object of this disclosure to provide methods and systems for reducing bias introduced by ISPs while effectively reducing noise, thereby generating high- fidelity images and videos.

[0009] Particularly, in accordance with this disclosure, there is provided, in one embodiment, a spatiotemporal noise reduction system for compensating bias from image signal processing of raw video signals. The system comprises an image signal processor which comprises a spatial noise reduction filter adapted to output each pixel based on neighboring pixels in each raw video frame; a signal change detector adapted to receive the output of the image signal processor and a previous output frame thereby detecting any signal changes in each pixel; a signal combiner adapted to receive more than two input frames and therefrom generate a combination output frame. The more than two input frames comprise a raw signal frame, an output from the image signal processor, and a previous output frame.

[0010] In another embodiment, the signal combiner is adapted to output a linear combination of the more than two input frames. In yet another embodiment, the signal combiner is adapted to output a convex combination of the more than two input frames.

[0011] In a further embodiment, the system further comprises a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on the detection of any signal changes received from the signal change detector. A convex combination weight is calculated for each input of the convex combination based on the current confidence indicator.

[0012] In another embodiment, the previous confidence indicator is assigned zero, for a first output frame from the raw video signals. [0013] According to yet another embodiment, the system further comprises a motion compensator adapted to reduce motion based on the output from the image signal processor and the previous output frame. The more than two input frames of the signal combiner comprise the raw signal frame, the output from said image signal processor, and an output from the motion compensator.

[0014] In a further embodiment, the raw video signals comprise exposure mosaic having spatially interleaved long- and short exposure time pixels. The image signal processor further comprises a spatial HDR interpolation unit and a dcmosaicking unit. The more than two input frames comprise a raw signal frame, an output from the spatial HDR interpolation unit, an output from the demosaicking unit, and a previous output frame.

[0015] In another embodiment, the raw video signals comprise color mosaic having spatially interleaved different color pixel. The image signal processor further comprises a demosaicking unit. The more than two input frames comprises a raw signal frame, an output from the demosaicking unit, and a previous output frame.

[0016] According to another embodiment, the raw video signals comprise demosaicked frames.

[0017] In yet another embodiment, the raw video signals comprise frames that have not been processed by a spatial noise reduction filter and have not been demosaicked, and the image signal processor further comprises a demosaicking unit. The more than two input frames comprise a raw signal frame, an output from the demosaicking unit, and a previous output frame.

[0018] In a further embodiment, the signal combiner is adapted to output a linear combination of the more man two input frames.

[0019] In another embodiment, the signal combiner is adapted to output a convex combination of the more than two input frames. [0020] In yet another embodiment, the system further comprises a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on the detection of any signal changes received from said signal change detector. A convex combination weight is calculated for each input of the convex combination based on the current confidence indicator.

[0021] In a further embodiment, the system further comprises a motion compensator adapted to reduce motion based on the output from the image signal processor and the previous output frame. The more man two input frames of the signal combiner comprise the raw signal frame, the output from the image signal processor, and an output from the motion compensator.

[0022] In accordance with this disclosure, there is provided, in one embodiment, a method for compensating bias from image signal processing of raw video signals. The method comprises: generating a linear combination of more than two input frames; and outputting the linear combination frame. The more than two input frames comprise a raw signal frame, an output from the image signal processing, and a previous output frame

[0023] In another embodiment, the linear combination is a convex combination of more than two input frames.

[0024] In yet another embodiment, the method further comprises generating a signal change detection classifier for each block of pixels based on an output of the image signal processor and a previous output frame; updating a confidence indicator for a current output frame based on the signal change detection classifier and a previous confidence indicator for the previous output frame; and calculating a weight for each input frame of the convex combination based on the updated confidence indicator for the current output frame.

[0025] In a further embodiment, the calculating a weight for each input frame of the convex combination further comprises providing a decreasing function for the ratio between the weight for the output of the image signal processing and the weight for the unfiltered raw input frame based on the confidence indicator of the current output frame.

[0026] According to another embodiment, the decreasing function is a monotone decreasing function.

[0027] In yet another embodiment, the calculating a weight for each input frame of the convex combination further comprises providing an increasing function for the weight for the previous output frame based on the confidence indicator of the current output frame.

[0028] In a further embodiment, the confidence indicator is a numerical number having a range between zero and one.

[0029] According to another embodiment, the method further comprises reducing motion based on the output from the image signal processing and the previous output frame thereby generating a motion-compensated output. The convex combination is a convex combination of the raw signal frame, the output from the image signal processing, and the motion-compensated output.

[0030] According to yet another embodiment, the raw video signals are selected from the group consisting of (i) spatially interleaved long- and short exposure time pixels, (ii) color mosaic having spatially interleaved different color pixels, (iii) demosaicked frames, and (iv) non-spatially filtered and non-demosaicked frames. The image signal processing is selected from the group consisting of (i) spatial HDR interpolation, demosaicking, and spatial noise reduction filtering, (ii) demosaicking and spatial noise reduction filtering, and (iii) spatial noise reduction filtering.

[0031] In a further embodiment, the more than two input frames comprise a raw signal frame, a previous output frame, and an output from the group consisting of the spatial HDR interpolation, the demosaicking, and the spatial noise reduction filtering. [0032] In another embodiment, the method further comprises reducing motion based on the output from the image signal processing and the previous output frame thereby generating a motion-compensated output. The convex combination is a convex combination of the raw signal frame, the motion-compensated output, and an output from the group consisting of the spatial HDR interpolation, the demosaicking, and the spatial noise reduction filtering.

[0033] According to yet another embodiment, the method further comprises generating a signal change detection classifier based on an output of said image signal processing and a previous output frame; determining a current confidence indicator for each pixel based on the signal change detection classifier and confidence indicators of neighboring pixels; and calculating a weight for each input frame for the convex combination based on the current confidence indicator.

BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 depicts a bias-compensating noise reduction system according to one embodiment of this disclosure.

[0035] Figure 2 depicts a bias-compensating noise reduction system according to another embodiment.

[0036] Figure 3 depicts a bias-compensating spatiotemporal noise reduction system according to another embodiment.

[0037] Figure 4 depicts a bias-compensating spatiotemporal and debayer (demosaicking) noise reduction system according to another embodiment.

[0038] Figure S depicts a confidence updater of the bias-compensating noise reduction system according to one embodiment. [0039] Figure 6 shows the convex combination weight of each input of the convex combination calculated by a weight calculator of the bias-compensating noise reduction system according to one embodiment.

DETAILED DESCRIPTION OF THE VARIOUS EMBODIMENTS System and Methodology Overview

[0040] The methods and systems according to the various embodiments of this disclosure employ a weighted combination of multiple input frames or blocks of pixels as part of a recursive temporal noise filter, to reduce bias from an ISP such as a spatial noise filter, a demosaicking unit, or HDR interpolation unit, thereby providing improved noise reduction solutions. The bias-compensating and noise reduction system in various embodiments of this disclosure are designed to reduce biases introduced at an ISP stage of exposure mosaic interpolation, color demosaick interpolation, or spatial noise reduction filtering.

[0041] In one embodiment, referring to Figure 3, for example, a spatial noise reduction filter is the ISP that is combined with a low complexity temporal noise filter. The resulting spatiotemporal filter reduces bias from the spatial noise filter and at the same time reduces the disadvantages of temporal filtering. According to some embodiments, the spatial noise reduction filter is provided in hardware or firmware, and a temporal noise reduction filter is customized and implemented in software. A customized temporal filter of this disclosure may be combined with an existing spatial filter in certain embodiments, forming an extension to the existing equipment thereby providing an improved noise reduction solution.

[0042] In other embodiments where raw input data include spatially interleaved long- and short exposure time pixels or color mosaic having spatially interleaved different color pixels, a demosaicking unit or debayer unit is the ISP that is combined with a temporal noise filter. The resulting bias-compensating filter reduces the bias or artifacts from demosaicking and achieves higher fidelity in image or video output frames. In further embodiments, further ISPs including spatial noise filters are combined in the system along with the demosaicking unit and the recursive temporal filter. See, e.g., Figure 4. The resulting bias-compensating filter reduces the bias or artifacts from the spatial noise filter and the debayer unit and achieves improved fidelity in the output frames.

Signal Combiner and Confidence Indicator

[0043] Referring to Figure 1, a signal combiner according to one embodiment is a central part of a recursive temporal filter of the present disclosure. The signal combiner provides a linear combination in one embodiment of at least two input frames, including the unfiltered input frame, the ISP-filtered input frame, and the previous output frame. In another embodiment, the signal combiner provides a convex combination of at least two input frames, including the unfiltered input frame, the ISP-filtered input frame, and the previous output frame. The convex combination according to various embodiments are made per block of pixels or per pixels. A confidence indicator is associated with each convex combination per pixel or per block of pixels. The convex combination based on the updated confidence indicator determines the new output frame.

[0044] According to one embodiment, a confidence indicator is assigned for every input frame or block of pixels in each input frame. The confidence indicator in various embodiments generally represents the extent to which the previous output frame is a good representation of the current input frame. It is defined as a numerical number with a predetermined range, such as from 0 to 1 according to one embodiment. The confidence indicator is stored together with the pixel values in a frame buffer. It is calculated by a confidence updater based on a confidence update function. [0045] Referring to Figure 5, the confidence update function calculates the confidence indicator of the current frame or current block of pixels (C(n)) based on the confidence indicator of the previous output frame (C(n-l)) and a signal change classifier ("true" or "false") derived for each block of pixels.

[0046] In one embodiment, the updated confidence indicator is defined to be the previous confidence indicator for the previous frame or block of pixels plus a positive increment for pixels or blocks of pixels classified as stationary. And for pixels or blocks of pixels classified as changing, the updated confidence indicator is defined to be the lowest possible value for the system, such as 0 in some embodiments. The increment according to certain embodiments depends on the previous confidence indicator but is constrained such that the updated confidence indicator stays within a predetermined range, such as between 0 and 1 according to one embodiment. See, e.g., Figure S.

[0047] Referring to Figure 1 , the signal change classifier based on which the confidence indicator is calculated, is determined by a signal change detector connected to an image signal processor (ISP) stage. The ISP stage according to a particular embodiment comprises a spatial noise reduction filter, and the signal change detector detects location changes in relevant blocks of pixels based on the output of the spatial filter and data from the previous output frame.

Convex Combination Weights

[0048] Referring to Figure 1, the weight calculator takes input form the confidence updater in calculating convex combination weights for each input frames or block of pixels.

According to one embodiment, the unfiltered raw input is weighted more than the ISP- filtered input or spatially filtered input (where the ISP is a spatial noise reduction filter) as the confidence indicator increases. Accordingly, the bias of the spatial filter is reduced as more frames are averaged in the process. When the confidence indicator for the previous output frame is low, on the other hand, the spatially filtered input is weighted more, thereby reducing the noise variance of the output.

[0049] Therefore, in certain embodiments, the ratio between the weight for the ISP-filtered input frame and the weight for the unfiltered raw input frame is a monotone decreasing function of the confidence indicator. Further, the weight for the previous output frame is an increasing function of the confidence indicator. This weight for the previous output frame is zero or near zero when the confidence indicator is zero, and it is close to one when the confidence indicator is one according to certain embodiments.

[0050] Referring to Figure 6, convex combination weights according to one embodiment are derived by the following formula based on the confidence indicator: gamma = c, beta = where c refers to the confidence indicator, alpha refers to

the weight for the unfiltered raw input, beta refers to the weight of the ISP-filtered input or the spatially filtered input where the ISP is a spatial filter, and gamma refers to the weight for the previous output frame. In this embodiment, the sum of alpha, beta, and gamma is 1, and the confidence indicator has a value between 0 and 1. According to the way alpha and beta vary as a function of the confidence indicator for the previous frame in this embodiment, therefore, beta (weight for the ISP or spatial filtered frame) is high in the beginning of the processing such that convergence to a true pixel value occurs faster, while alpha (weight for the unfiltered raw input) increases gradually to reduce the bias from the ISP or the spatial filter where the ISP is a spatial filter.

[0051] In alternative embodiments, piecewise linear approximations may be adopted to derive the weights for each input frames or block of pixels in the combination output frame. The combination output frame is based on a linear combination or convex combination of more than two input frames in alternative embodiments. Signal Change Detector

[0052] As discussed above, the confidence updater updates the confidence indicator for each block of pixel or each input frame based on a change detection classifier generated by the signal change detector in certain embodiments. See, e.g., Figures 1, 2, and S. The signal change detector takes as its inputs the ISP-filtered signal (or the spatial filtered signal where the ISP is a spatial filter) together with the previous output frame, and outputs a binary change detection classifier ("true" or "false") per pixel or block of pixel indicating the status as "stationary" or "changing." For example, in a basic embodiment, the difference between the two inputs is evaluated, and a threshold is determined for the classification of

"changing." A difference between the two inputs above the threshold leads to a "true" reading in the signal change classifier indicating "changing," otherwise a "false" reading in the signal change classifier indicating "stationary."

[0053] According to other embodiments, the signal change detector takes as inputs the spatial neighborhood of the pixels in the previous and the current input frame to improve the classification of signal change for the system. In further embodiments, the signal change classifier adopts background and foreground estimation algorithms to classify pixels as stationary or changing.

[0054] The bias-compensating noise reduction system according to various embodiments therefore provides the flexibility where the inputs to the signal change detector for the classification of change on the one hand, are decoupled from the inputs to the signal combiner for the convex combination forming the new output frame on the other hand. The inputs for determining the signal detection classifier are selected in various embodiments to maximize classification performance, while the inputs to the convex combination are selected to effectively reduce the visual degradations and biases of the ISPs such as spatial filters and debayer units employed by the system. Motion Compensator

[0055] Referring to Figure 2, the bias-compensating noise reduction system in another embodiment further comprises a motion compensator. The motion compensator employs motion estimation techniques to modify the previous output frame, and compensates for motion between the previous output frame and current input frame. As motion is estimated and compensated for, the bias-compensating noise reduction system according to this embodiment enables better opportunities for combining frames in the temporal dimension. For the convex combination, the motion compensated video frame is taken as an input frame in lieu of the previous output frame according to this embodiment.

[0056] Figures 3 provides another example of a bias-compensating spatiotemporal noise reduction system in which motion detection and compensation is implemented. In this embodiment, the convex combination is based on the output from the motion compensator, the spatial-filtered input frame, and the unfiltered input frame.

[0057] Figure 4 provides a further example of a bias-compensating spatiotemporal and debayer noise reduction system in which motion detection and compensation is implemented. In this embodiment, the convex combination is based on the output from the motion compensator, the spatial-filtered input frame, the debayered frame, and the raw bayer input frame.

ISP Stage and Corresponding Bias Reduction

[0058] As discussed above, the systems and methods of this disclosure reduce biases introduced by ISPs or ISP components during ISP-filtering of the raw input data, also referred to as an ISP stage. The raw input data to the system are of a variety of types in various embodiments. The ISP stage in various embodiments adopts and operates one or more ISPs of different utilities. In one embodiment, the ISP stage comprises exposure mosaic interpolation. In another embodiment, the ISP stage comprises color demosaick interpolation. In a further embodiment, the ISP stage comprises spatial noise reduction filtering.

[0059] For example, in certain embodiments block mosaic of long and short exposures, or interleaved lines with different exposures are generated by existing sensors. HDR sensing is adopted at the ISP stage in one embodiment. As spatial interleaving and spatial reconstruction are involved in HDR interpolation, which results in certain bias intrinsically, according to one embodiment an enhanced reconstruction with better qualities is provided via a convex combination based on weighted input frames including the raw interleaved exposure pixels, the biased spatial reconstruction, and the previous output frame.

[0060] In this embodiment therefore, the raw data is exposure mosaic, the ISP filtering stage is spatial HDR interpolation, demosaicking, and spatial noise filtering. In another embodiment, the raw data is color mosiac, the ISP filtering stages is demosaicking and spatial noise filtering. In a further embodiment, the raw data is demosaicked frame, and the ISP filtering stage is spatial noise filtering. The convex combination in various embodiments is made by combining weighted input frames including the raw data, the ISP filtered data, and the temporal filtered data (the previous output frame or the motion- compensated output).

[0061] The descriptions of the various embodiments provided in this disclosure, including the various figures and examples, are to exemplify and not to limit the invention and the various embodiments thereof.