TECHNICAL FIELD

The present disclosure relates generally to the field of processing image data, and, more particularly, to pre-processing of image data before it is provided to an application. To this end, a device and a method are disclosed that determine a Transfer Function (TF), which may be applied to the image data before it is provided to the application. The application may be a Computer Vision (CV) application, a human vision application, or the like.

BACKGROUND

Generally, a pre-processing procedure may be performed on image data, for example, before the image data is provided to an application like a computer vision algorithm. A conventional pre-processing method may include applying a tone mapping curve to the image data, or a conventional transfer function that is intended for visualization of images. Furthermore, a standard gamma curve is an example of a conventional transfer function for a low dynamic range sensor and standardized in BT-709.

Moreover, an adaptive gamma curve is also suggested, which is an extension of the standard gamma curve. The standard gamma curve has a gamma value of 2.4, and the adaptive gamma curve determines a corresponding gamma value based on the image data. Furthermore, a Perceptual Quantizer (PQ) is an example of a conventional transfer function for a high dynamic range display and standardized in ST-2084.

However, certain adverse issues arise from the conventional procedure. As an example, the conventional transfer functions such as the standard gamma curve, the adaptive gamma curve, and the PQ, are intended for visualization, and thus, may not be optimal for a computer vision (CV) application. For example, by using a transfer function that is not optimal for a computer vision algorithm of such a CV application, the CV application may perform worse in extreme conditions such as a low illumination environment, a high noise environment, and a high dynamic range in the scene. Moreover, a conventional tone mapping curve derived for a single CV application may not be generalized to other CV applications.

SUMMARY

In view of the above-mentioned problems and disadvantages, embodiments of the present disclosure aim to provide improved devices and methods for pre-processing image data. An objective is, in particular, to determine a TF which may be optimized for being used for a CV application, a human application, or the like.

Another objective is to provide a TF that may be adapted to different image sensors. The goal with such a TF is specifically to enhance a performance of a computer vision algorithm or a CV application that operate on input image data received from a high dynamic range image sensor. Thus, the TF may be designed such that a robust signal is provided to the CV application.

These and other objective are achieved by the embodiments of the disclosure as described in the enclosed independent claims. Advantageous implementations of the embodiments of the disclosure are further defined in the dependent claims.

A first aspect of the present disclosure provides a device for pre-processing image data. The device is configured to obtain, by means of a channel, image data of an image sensor, wherein the image data comprises an intensity value for each of a plurality of image sensor pixels associated with the channel. The device is further configured to determine, in a logarithmic domain, a histogram of the image data based on a noise profile of the image sensor, the histogram comprising a plurality of bins, wherein each bin from the plurality of bins is associated with a set of intensity values of the image data. The device is further configured to determine a Transfer Function (TF) based on the histogram of the image data.

The device may be, or may be incorporated in, a computer vision digital camera, a digital video recorder, a mobile phone, a smartphone, a play station, an augmented reality device, a virtual reality device, a personal computer, a desktop computer, a laptop and a notebook computer, a tablet or a pad device, an image signal processing module of an imaging device, a part of an optical flow generation for an ISP of an imaging device, a camera module of a high dynamic range imaging device, etc.

The image sensor may be any conventional image sensor, for example, it may be a Charge- Coupled Device (CCD) image sensor, a Complementary Metal-Oxide Semiconductor (CMOS), etc.

Moreover, the device of the first aspect may convert the obtained image data to the logarithmic domain. The logarithmic domain may be any known logarithmic domain. Furthermore, the device may compute, in the logarithmic domain, the initial histogram based on the intensity values of the image data. Moreover, the device may obtain the noise profile of the image sensor, for example, based on at least one of: a type of the image sensor, the intensity values of the image data, an amplification factor of the image sensor, an exposure time of the obtained image data, an Analog-to-Digital Converter (ADC) noise parameter of the image sensor, a readout noise parameter of the image sensor.

In some implementations, the device of the first aspect may further determine at least one of a Signal-to-Noise Ratio (SNR)-weighting function for the image sensor based on the noise profile of the image sensor, an amplification factor of the image sensor, and a threshold-SNR. Moreover, the device may further weigh the initial histogram based on the SNR-weighting function, and may further obtain the histogram of the image data. The histogram of the image data may thus be a weighted histogram, that is, a histogram that is weighted based on the determined SNR-weighting function of the image sensor. Hence, in some implementations, the histogram may be adapted to the noise profile of the image sensor, the type of the image sensor, or the like.

Moreover, the device of the first aspect is configured to determine the TF based on the histogram of the image data. For example, in some implementations, the device may be configured to determine a Histogram Equalization (HE) model for the image sensor. Furthermore, the device may be configured to determine the TF based on the HE model using the histogram of the image data.

In some implementations, the device is further configured to determine the TF based on the histogram of the image data and the noise profile of the image sensor. For example, the determined TF may further be adapted to the noise profile of each type of camera sensor, i.e., the device may determine the TF, e.g. once for each type of the camera’s image sensor.

The TF may be a noise adaptive transfer function, i.e., the TF may be adapted based on the noise profile of the image sensor. Further, the TF may be adapted based on the image data. Hence, in some implementations, it is possible that the TF is adapted to the image data and the noise profile of the image sensor.

The device of the first aspect may be configured to further provide the TF. Thereby, the TF may be a noise adaptive TF, which may be used for encoding color and luma values. For example, the TF may be a function that maps intensity values to encoded values, i.e. each linear intensity values may be mapped into an integer value which may be a suitable input for a computer vision application. The TF may improve a performance of computer vision method. The device may also be configured to pre-process the image data based on the TF, and to provide the processed image data to the CV application.

Hereinafter, the terms “transform” and “transfer function” are used interchangeably, without limiting the present disclosure.

The device may comprise a circuitry. The circuitry may comprise hardware and software. The hardware may comprise analog or digital circuitry, or both analog and digital circuitry. In some implementations, the circuitry comprises one or more processors and a non volatile memory connected to the one or more processors. The non-volatile memory may carry executable program code which, when executed by the one or more processors, causes the device to perform the operations or methods described herein.

In an implementation form of the first aspect, the TF comprises a plurality of parts, wherein each part of the TF is associated with a determined bin of the histogram of the image data.

For example, the device may be configured to determine the histogram of the image data comprising a plurality of bins. Each bin from the plurality of bins of the histogram may be determined based on a respective set of intensity values of the image data. Moreover, the device may be configured to use a respective bin of the histogram to determine a respective part of the TF. The TF may comprise the plurality of parts in which each part of the TF is determined using a respective bin of the histogram. In other words, each part of the TF may be determined for a different set of the intensity values and the number of parts of the TF may be equal with the number of bins of the histogram of the image data.

In a further implementation form of the first aspect, the device is further configured to determine the histogram of the image data by converting the image data to the logarithmic domain, and computing, in the logarithmic domain, an initial histogram based on the intensity values of the image data.

For example, the device may be configured to obtain, in the logarithmic domain, a plurality of sets of intensity values. Moreover, the device may be configured to obtain a bin of the initial histogram of the image data using a set of intensity values. Further, the device may be configured to obtain the initial histogram comprising a plurality of bins such that each respective bin is obtained for a respective set of intensity values.

In a further implementation form of the first aspect, the device is further configured to determine a Signal-to-Noise Ratio (SNR)-weighting function for the image sensor based on the noise profile of the image sensor, an amplification factor of the image sensor, and a threshold-SNR, and weighting the initial histogram of the image data based on the SNR- weighting function to obtain the histogram of the image data.

The SNR-weighting function may be a function that is specifically determined for the image sensor or for each type of the image sensor in a device. For instance, the SNR- weighting function of the image sensor may be determined based on the threshold-SNR, the SNR of the pixels of the image sensor, etc. Further, the device may be configured to weigh the initial histogram by the SNR-weighting function for obtaining the histogram of the image data.

In a further implementation form of the first aspect, the device is further configured to determine a Histogram Equalization (HE) model for the image sensor based on the histogram of the image data, and determine the TF based further on the HE model using the histogram of the image data. For example, the device may be configured to determine the HE model for the image sensor based on the histogram of the image sensor which is weighted by the SNR-weighting function of the image sensor. The HE model may be used for determining the TF.

In a further implementation form of the first aspect, the HE model is based on a Contrast- Limited HE (CLHE) model, and wherein the device is further configured to limit a contrast information of at least one bin of the histogram of the image data using the CLHE model.

In a further implementation form of the first aspect, the device is further configured to obtain the noise profile of the image sensor based on one or more of:

- the intensity values of the image data, an amplification factor of the image sensor, a Analog-to-Digital Converter (ADC) noise parameter of the image sensor, a readout noise parameter of the image sensor, a noise fitting parameter, an exposure time of the obtained image data, and a type of the image sensor.

In a further implementation form of the first aspect, the device is further configured to determine a SNR value for a pixel of the image sensor based on the intensity value of the pixel, the image data, and the noise profile of the image sensor.

In a further implementation form of the first aspect, the device is further configured to determine a threshold-SNR for the image sensor based on at least one of a type of the computer vision application, an amplification factor of the image sensor, and the noise profile of the image sensor.

In a further implementation form of the first aspect, the device is further configured to determine the SNR-weighting function based further on the SNR value of the pixel and the threshold-SNR.

In a further implementation form of the first aspect, the SNR-weighting function is based on a binary noise threshold function, and wherein the SNR-weighting function has a value of 1, when the SNR value of the pixel is greater than the SNR threshold of the image sensor, or the SNR-weighting function has a value of 0, when the SNR value of the pixel is smaller than the SNR threshold of the image sensor.

In a further implementation form of the first aspect, the SNR-weighting function is determined based on an identity function and the SNR value of the pixel, or the SNR- weighting function has a predefined value.

In a further implementation form of the first aspect, the device is further configured to apply, in the logarithmic domain, each part of the TF to the set of the intensity values associated with the part, in order to obtain transformed image data.

In a further implementation form of the first aspect, the device is further configured to apply an inverse logarithmic TF to the transferred image data for obtaining image data or video data in a determined format according to the computer vision application.

In a further implementation form of the first aspect, the device is further configured to be implemented in any one of: an Image Signal Processing (ISP) module of an imaging device, an imaging device for a computer vision application of an autonomous driving system, a part of an optical flow generation for an ISP of an imaging device, a camera module of a High Dynamic Range (HDR) imaging device.

A second aspect of the disclosure provides a method for pre-processing image data. The method comprising obtaining, by means of a channel, image data of an image sensor, wherein the image data comprises an intensity value for each of a plurality of image sensor pixels associated with the channel, determining, in a logarithmic domain, a histogram of the image data based on a noise profile of the image sensor, the histogram comprising a plurality of bins, wherein each bin from the plurality of bins is associated with a set of intensity values of the image data, and determining a TF based on the histogram of the image data. In an implementation form of the second aspect, the TF comprises a plurality of parts, wherein each part of the TF is associated with a determined bin of the histogram of the image data.

In a further implementation form of the second aspect, the method further comprises determining the histogram of the image data by converting the image data to the logarithmic domain, and computing, in the logarithmic domain, an initial histogram based on the intensity values of the image data.

In a further implementation form of the second aspect, the method further comprises determining a SNR-weighting function for the image sensor based on the noise profile of the image sensor, an amplification factor of the image sensor, and a threshold-SNR, and weighting the initial histogram of the image data based on the SNR-weighting function to obtain the histogram of the image data.

In a further implementation form of the second aspect, the method further comprises determining a HE model for the image sensor based on the histogram of the image data, and determining the TF based further on the HE model using the histogram of the image data.

The device may adapt the TF to the noise characteristic of the image sensor and the image data. The noise characteristics of the image sensors may be measured and modeled. The TF may be a TF that is adapted to the noise, which depends on a type of the image sensor and a gain setting.

Additionally, the device may use the HE or one of its variants, e.g., a contrast-limited histogram equalization (CLHE), for computing the histogram of the image data. Further, the device may determine the TF based on the histogram of image data. The image data with different contents (different intensity values) may lead to different histograms, and thus the transfer function is content-adaptive TF. Moreover, the noise profile of the image sensor may also be considered for computing the histogram of the image data, and thus, the histogram may be a noise-adaptive histogram. In a further implementation form of the second aspect, the HE model is based on a CLHE model, and wherein the method further comprises limiting a contrast information of at least one bin of the histogram of the image data using the CLHE model.

In a further implementation form of the second aspect, the method further comprises obtaining the noise profile of the image sensor based on one or more of:

- the intensity values of the image data, an amplification factor of the image sensor, an ADC noise parameter of the image sensor, a readout noise parameter of the image sensor, a noise fitting parameter, an exposure time of the obtained image data, and a type of the image sensor. In a further implementation form of the second aspect, the method further comprises determining a SNR value for a pixel of the image sensor based on the intensity value of the pixel, the image data, and the noise profile of the image sensor.

In a further implementation form of the second aspect, the method further comprises determining a threshold-SNR for the image sensor based on at least one of a type of the computer vision application, an amplification factor of the image sensor, and the noise profile of the image sensor.

In a further implementation form of the second aspect, the method further comprises determining the SNR-weighting function based further on the SNR value of the pixel and the threshold-SNR.

In a further implementation form of the second aspect, the SNR-weighting function is based on a binary noise threshold function, and wherein the SNR-weighting function has a value of 1 , when the SNR value of the pixel is greater than the SNR threshold of the image sensor, or the SNR-weighting function has a value of 0, when the SNR value of the pixel is smaller than the SNR threshold of the image sensor. In a further implementation form of the second aspect, the SNR-weighting function is determined based on an identity function and the SNR value of the pixel, or the SNR- weighting function has a predefined value.

In a further implementation form of the first aspect, the method further comprises applying, in the logarithmic domain, each part of the TF to the set of the intensity values associated with the part, in order to obtain transformed image data.

In a further implementation form of the second aspect, the method further comprises applying an inverse logarithmic TF to the transferred image data for obtaining image data or video data in a determined format according to the computer vision application.

In a further implementation form of the second aspect, the method may be for any one of: an ISP module of an imaging device, an imaging device for a computer vision application of an autonomous driving system, a part of an optical flow generation for an ISP of an imaging device, a camera module of a HDR imaging device.

The method of the second aspect achieves the advantages and effects described for the device of the first aspect.

A third aspect of the present disclosure provides a computer program comprising a program code for performing the method according to the second aspect or any of its implementation forms.

A fourth aspect of the present disclosure provides a non-transitory storage medium storing executable program code which, when executed by a processor, causes the method according to the second aspect or any of its implementation forms to be performed.

It has to be noted that the devices, elements, units and means described in the present application could be implemented in software or hardware elements or any kind of combination thereof. The steps which are performed by the various entities described in the present application, as well as the functionalities described to be performed by the various entities, are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

BRIEF DESCRIPTION OF DRAWINGS

The above described aspects and implementation forms will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which FIG. 1 depicts a schematic view of a device for pre-processing image data, according to an embodiment of the disclosure;

FIG. 2 depicts a schematic view of the device providing a TF that is used in an image signal processing pipeline of a camera, according to an embodiment of the disclosure;

FIG. 3 shows a diagram illustrating exemplarily noise profiles of image sensor for three different image sensors; FIG. 4 shows a diagram illustrating exemplarily noise profiles of image sensor for three different gains of one image sensor;

FIG. 5A shows a diagram illustrating an initial histogram of the image data determined based on the image data;

FIG. 5B shows a diagram illustrating a histogram of the image data obtained based on the initial histogram of FIG. 5 A and a SNR- weighting function; FIG. 5C shows a diagram illustrating a histogram of the image data obtained based on the initial histogram of FIG. 5 A and a SNR- weighting function;

FIG. 6 shows a diagram illustrating exemplary TF determined based on histograms of image data;

FIG. 7 is a schematic view of the device comprising a TF used for generating optical flows, according to an embodiment of the disclosure; FIG. 8 is a schematic view of the device comprising a TF used on the sensor side and in a camera module, according to an embodiment of the disclosure; and

FIG. 9 depicts a flowchart of a method for pre-processing image data, according to an embodiment of the disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 depicts a schematic view of a device 100 for pre-processing image data 111, according to an embodiment of the disclosure.

The device 100 may be, or may be incorporated in, a computer vision digital camera, a digital video recorder, a mobile phone, a smartphone, an image signal processing module of an imaging device, a part of an optical flow generation for an ISP of an imaging device, a camera module of a high dynamic range imaging device, etc.

The device 100 is configured to obtain, by means of a channel 101, image data 111 of an image sensor 110. The image data 111 comprises an intensity value for each of a plurality of image sensor pixels associated with the channel 101. As can be deduced from FIG. 1, the channel 101 may be adapted to interconnect the image sensor 110 and the device 100.

The device 100 is further configured to determine, in a logarithmic domain, a histogram 120 of the image data 111 based on a noise profile 130 of the image sensor 110. The histogram 120 comprises a plurality of bins, wherein each bin from the plurality of bins is associated with a set of intensity values of the image data 111. The histogram may be determined based further on the image data 111.

For example, the device 100 may convert the image data 110 to the logarithmic domain, and may further obtain a plurality of sets on intensity values in the logarithmic domain. The device may use each set of intensity values for determining one of the bins of the histogram.

In particular, the device 100 may obtain an initial histogram comprising a plurality of bins based the plurality of sets of intensity values. Each respective bin may be determined for a respective set of intensity values.

Moreover, the device 100 may obtain the noise profile 130 of the image sensor 110 (as indicated by the dashed arrow in FIG. 1). In particular, the noise profile 130 of the image sensorllO may be determined based on (or may include) the photon noise, the read-out noise and the ADC noise, a threshold SNR, an SNR of a pixel of the image sensor, etc. The noise profile of the image sensor may indicate how much noise is produced by the image sensor, in the pixels, at different image sensor settings. In some embodiments, the device 100 may determine a SNR- weighting function for the image sensor 110. The SNR- weighting function of the image sensor may be a function that is related to image pixel luminance, a gain parameter, an ISO setting, or the like. The parameters of the SNR- weighting function may be different for each image sensor.

Furthermore, the device 100 may be configured to determine the histogram 120 of the image data 111, e.g., by weighting the initial histogram based on the SNR- weighting function determined for the image sensor 110. Hence, the device 100 may be configured to obtain the histogram which may be adapted based on the noise profile of the image sensor 110, an amplification factor (i.e., again parameter) of the image sensor 110, a camera parameter, or the like.

For example, the device 100 may be configured to use a HE model or a CLHE model to determine the TF 140 based on the histogram 130 (i.e., a noise-adaptive histogram) of the image data 110. In some embodiments, the device 100 may further be configured to apply, in the logarithmic domain, each part of the TF 140 to its respective set of the intensity values associated with that part of the TF 140, in order to obtain transformed image data. The transformed image data may be used as an input for a computer vision application 150. For example, the TF 140 may be used for encoding color and luma values that may improves the performance of computer vision application 150. In particular, the TF 140 may improve a performance of computer vision application 150 in a high noise scenario such as a dark scene, a high camera gain or ISO settings, etc.

The device 100 may comprise processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the device 100 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field- programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.

FIG. 2 depicts a schematic view of the device 110 according to an embodiment of this disclosure which builds on the embodiment in FIG. 1. Same elements in FIG. 1 and FIG. 2 are labelled with the same reference signs. The device 110 of FIG. 2 is configured to provide a TF 140 that is used in an image signal processing pipeline 210 of a camera, according to an embodiment of the disclosure.

The device 100 of FIG. 2 is exemplary based on a CV digital camera. Moreover, the TF 140 may be used (incorporated) in the image signal processing modules of the device 100 (camera).

The image signal processing (ISP) herein does not refer to a general concept of processing the image, but the specific systems in cameras, which convert sensor raw data, typical Bayer-pattem but possibly also other formats, to standard RGB or YUV format, which are appropriate for image and video storage in typical formats (for example, known conventional formats such as Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), etc.

The camera ISP 210 may be configured to use the transfer function 140 as it is depicted in FIG. 2. The raw data may come from the image sensor 110 (i.e., in FIG. 2 the image sensor 110 and the ADC are in one unit) and may go into the camera ISP unit 210. The image sensor 110 and ADC may provide the image data 111 in a certain bit-depth, which typically ranges from 10 bit to 24 bit. Without loss of generality, it may be assumed that the image data 111 is provided in 16 bit.

The camera ISP 210 comprises in this example a fixed pattern noise suppression unit 211. The fixed pattern noise suppression unit 211 is configured to remove the noise in the image data 111 based on some prior knowledge of the noise patterns of the image sensor. The fixed pattern noise suppression unit 211 may be only a small fraction of the sensor noise, and may be independent from the image pixel intensity and camera gain or ISO setting. After the fixed pattern noise suppression unit 211, the image data 111 may still indicate a significant amount of noise. In this embodiment, the device 100 is further configured to consider and model the noise of the image sensor 110 after the fixed pattern noise suppression unit 211.

The camera ISP 210 further comprises a demosaicing unit 212. The image data 111 provided by the image sensor 110 may comprise a single color component, either R or G or B, and the demosaicing unit 212 may be configured to interpolate the missing color components for every pixel. Thus, it may end up with a full RGB image and every pixel comprises the all of three color components in RGB.

The Camera ISP unit 210 further comprises the pre-computed sensor noise model 120, a logarithmic unit 213, the histogram 130, and the TF 140. This is an exemplary (minimal) ISP setup, and there may be more processing steps before and after the TF 105.

The output of the camera ISP 210, image or video either in RGB format or YUV format, may be sent to the CV application unit 150. The CV application unit 150 may be a computer vision algorithm, with which computers gains high-level understanding from digital images or videos, e.g., for face detection, face recognition, object detection, optical flow, object detection, etc.

The device 100 may be configured to obtain the noise profile 120 of the image sensor 110.

Different computer vision methods may be robust to the noise (e.g., photon noise, read-out noise and ADC noise, etc.) introduced by the image sensor 110. The device 100 may be configured to consider the introduced noises when determining the transfer function 140.

It is assumed that the image sensor 110 of the camera can collect F electron per second. Moreover, if the gain 214 of the image sensor 110 is g, the image sensor’s readout for the color channel of c (ce {R,G,B}) over the exposure time t may be estimated, as follows:

Y _c = min {k _c ( tg + n), Y _max }, where the coefficient k _c may account for any digital gain and the white balance correction applied after the charge has been converted into digital units. The symbol n is a normally distributed random variable that models the noises (for example, the photon noise, the read out noise and the ADC noise).

Furthermore, the variance of the noises may be modelled, as follows:

For example, a model (e.g., a noise profile of the image sensor) for the camera noise characteristics may be determined, and the required parameters for the determined model may be estimated. A list of the symbols used for determining the noise model are presented in table I.

Table I: List of used symbols for noise model

The device 100 may further be configured to fit the parameters of the noise model 120 to the image data 111 captured by the image sensor 110 of the camera under different light intensities, exposure times, gains or ISO settings 214.

The device 100 may be further configured to determine the SNR using the following equation: In particular, the noise may be different for R, G and B color component. However, for the ease of description, in the following a same model is used for all different color components, and the same model is used for the luma component in YCbCr or YUV color space, too, without limiting the present disclosure to this specific model or using an identical model for different components.

Considering the above assumption, the device 100 may be further configured to determine the SNR equation, as follows: where, Y is an intensity value of a pixel of the image sensor 110. In some embodiments, the device 100 may also be configured to use application specific parameters 215 for determining the histogram of the image data 120.

FIG. 3 and FIG. 4 illustrate two examples of noise models 130 determined by the device 100, for different image sensors 110.

Diagram 300 of FIG. 3 shows an example of noise profiles 130 including: a first SNR function 301 versus pixel luminance for a first image sensor 110, a second SNR function 302 versus pixel luminance for a second image sensor 110, and a third SNR function 303 versus pixel luminance for a third image sensor 110.

The SNR functions 301, 302, 303 are a function of pixel intensity, and SNR functions 301, 302, 303 are different for different image sensors 110.

In addition, for the same image sensor 110 (e.g., the first sensor), the first SNR function 301 is different for different gains.

Diagram 400 of FIG. 4 shows an example of noise profiles 130 including a first SNR function 401 versus pixel luminance for a first gain of a first image sensor 110, a second SNR function 402 versus pixel luminance for a second gain of the image sensor 110, and a third SNR function 403 versus pixel luminance for a third gain of the first image sensor 110. Herein, gain 1 is equivalent to ISO 100, gain 8 ISO 800, gain 64 ISO 6400, and so on. In practice, SNR for different gains of a sensor are either precomputed or computed dynamically as the gain changes.

The camera ISP of the device 100 of FIG. 2 further comprises the logarithmic unit 213. For example, the image data 111 having intensity values for each pixel which may be in the linear domain. Moreover, the logarithmic unit 213 may be configured to convert the image data 111 from the linear domain to a logarithmic domain. Although a logarithmic domain of any base may be used, in the following, for the ease of the description a logarithmic domain of base 2 is used, as follows: y = log ₂ (Y + eps ) where, eps is a small value added to Y to avoid the invalid log of zero.

The device 100 may further be configure to use the intensity values of the pixels in the logarithmic domain, (i.e., y) to determine the histogram 120 of the image data which may be the noise-adaptive histogram.

The device 100 may be configured to determine an initial histogram of the image data 111. The initial histogram of the image data may comprise a plurality of bins. Each bin of the initial histogram may be determined for a set of intensity values.

For instance, the device 100 may be configured to determine bin edges of the initial histogram, as follows: where, i = 0, . N, l _min is a minimum intensity value of a pixel in the logarithmic domain, and I _ma x is ^a maximum intensity value of the pixel in the logarithmic domain, and N is the number of bins of the initial histogram of image data. Generally, N may be to large enough and it is recommended that N ³ 2 l _max , without limiting the present disclosure.

Diagram 500A of FIG. 5 A shows an initial histogram of the image data 110. The initial histogram of the image data 110 comprises a plurality of bins. Three bins from the plurality of bins are indicated with reference numbers 501 A, 502A, and 503A. Inthey-axis a number of pixels of each bin is divided by total number of pixels in the image sensor.

The device 100 may further be configured to determine the histogram of image data 120 based on the noise profile of the image sensor 130. For example, the device 100 may be configured to weigh the initial histogram of image data by the SNR-weighting function of the image sensor 110.

Moreover, the device 100 may be configured to obtain the histogram 120 which may be a noise-adaptive histogram that is computed by using a SNR related function as weight, according to the following equation: where, h _t is the value of the ith histogram bin, i = 0, . N — 1, y represents all pixel values that fall in the ith histogram bin, and /(y) is a SNR- weighting function. Note that if /(y) = 1, it falls back to the basic histogram.

Furthermore, /(y) is a SNR-weighting function. In the following, two examples of the SNR-weighting function are described in more detailed. In some embodiments, the device 100 may be configured to determine the SNR-weighting function such that the SNR-weighting function is based on a binary noise threshold function. For example, the SNR-weighting function has a value of 1, when the SNR value of the pixel is greater than the SNR threshold of the image sensor, or the SNR-weighting function has a value of 0, when the SNR value of the pixel is smaller than the SNR threshold of the image sensor.

For example, the device 100 may be configured to set a SNR threshold such that the SNR- weighting function equals 0 below the SNR threshold or the SNR-weighting function equal to 1 above the SNR level, as follows: where, S _thr is an application-specific SNR threshold value or it may be preset by user of a camera. Diagram 500B of FIG. 5B shows a histogram 120 of the image data 111 obtained based on the initial histogram of FIG. 5 A and a SNR-weighting function, according to this example. In the y-axis a number of pixels of each bin is divided by total number of pixels in the image sensor, and further only pixels having SNR values above the SNR-threshold are considered. In the diagram 500B of FIG. 5B, the histogram 120 of the image data 110 comprises a plurality of bins. Three bins from the plurality of bins are indicated with reference numbers 501B, 502B, and 503B. Here, it is assumed that if the SNR value of a pixel is above a certain threshold, the noise is assumed to be mild, and the useful signal predominates over the noise, and thus this pixel is taken in the histogram and assigned a weight of 1. Similarly, it is assumed that if the SNR value of a pixel is under a certain threshold, the noise is assumed to be too strong, and overwhelm the useful signal. Further, the intensity value of the pixel may not be considered for determining the histogram of the image data. Thus, a weight of 0 is assigned to the pixel.

In some embodiments, the device 100 may be configured to determine the SNR- weighting function such that the SNR-weighting function is determined based on an identity function and the SNR value of the pixel, or the SNR-weighting function has a predefined value.

For example, the device 100 may be configured to set a S _min which is a non-negative minimum SNR value. Further, the SNR-weighting function equals the S _min for all pixels that their SNR values are below this non-negative minimum SNR value. In addition, the SNR-weighting function equals the SNR values of pixels, for all pixels that their SNR values are above this non-negative minimum SNR value, as follows: where, S _min is a non-negative threshold value set by users and is recommended to be 0.5. Diagram 500C of FIG. 5C shows a histogram 120 of the image data 111 obtained based on the initial histogram of FIG. 5 A and a SNR-weighting function, according to this example.

In the diagram 500C of FIG. 5C, the histogram 120 of the image data 110 comprises a plurality of bins. Three bins from the plurality of bins are indicated with reference numbers 501 C, 502C, and 503C. Here, it is assumed that, the SNR value of the pixel can be used as the weight. However, for a case that the SNR is negative, and a negative weight would be invalid for computing histogram, and thus, a non-negative S _min is set to obtain a weight that is larger than 0.

Moreover, for the pixels with larger SNR, i.e., having a lower noise, a higher weight is assigned. Further, for the pixels with lower SNR, i.e., having a higher noise, a lower weight is assigned. Hence, the device 100 is configured to obtain the SNR-weighting function for the image sensor based the SNR values of the pixels of the image sensor. Moreover, the device 100 is configured to determine the histogram based on the SNR-weighting function (e.g., the noise characteristics of the image sensors), and therefore, the histogram is a noise-adaptive histogram.

The device 100 may further be configured to determine the TF 140 based on the histogram 120 of the image data 111.

Reference is now made to FIG. 6 which is a diagram illustrating determined TF 120. As discussed, the device 100 may be configured to use the HE model or one of its variants, e.g., the CLHE model to determine the TF 140.

Diagram 600 of FIG. 6 shows three examples of TF determined by the device 100.

The TF 120 indicated with a reference number 601 in diagram 600 of FIG. 6 is a TF that is determined based on the CLHE model using the initial histogram illustrated in diagram 500A of FIG. 5A. No noise is considered for the TF 120 indicated with the reference number 601 and is shown for ease of comparison.

The TF 120 indicated with a reference number 603 in diagram 600 of FIG. 6 is a TF that is determined based on the CLHE model using the histogram illustrated in diagram 500B of FIG. 5B. The TF 120 indicated with a reference number 602 in diagram 600 of FIG. 6 is a TF that is determined based on the CLHE model using the histogram illustrated in diagram 500C of FIG. 5C. The input are pixel values in the logarithmic domain while the output is a number between 0 and 1. The minimum input is mapped to 0 and the maximum input is mapped to 1.

As it can be derived from diagram 600 of FIG. 6, the TF values between -5 to 5 basic curve (solid line 601) is higher, which means that the noise is higher. In bright regions noise is small, the curve depends less on noise.

The device 100 may be configured to further apply the TF 120 to image data, for obtaining the transformed image data. The transformed image data are provided to the CV 150.

FIG. 7 is a schematic view of the device 100 comprising a TF 140 being used for generating optical flows on raw sensor data (obtained image data of the image sensor), according to an embodiment of the disclosure.

The device 100 may be configured to determine the TF 120 that may help generating optical flows 701. For example, the determined TF 140 is used to generate more accurate optical flow 701 to assist the camera’s ISP 210.

In this embodiment, the transfer function 120 is applied at an early stage of the device 100, i.e., directly applied to the raw data of the image sensor 110. The optical flow 701 is generated using the output of the transfer function 140. The generated optical flow is forwarded to the camera’s ISP 210. Note that, not all camera’s ISP 210 depends on the transfer function 140, but some of the camera ISP may benefit from optical flow 701. The output of the camera ISP 210 is sent to the CV application unit 150.

Moreover, one function of the transfer function 140 is a pre-processing step of the raw data for optical flow 701. In this manner, the accuracy of the output of the optical flow unit 701 is expected to be improved. Higher accuracy of optical flow may lead to better quality of camera ISP output images. Finally, a higher accuracy of CV applications is expected. Note that the transfer function 140 may also be used in the camera ISP, for example, similar to the embodiment of FIG. 2, and for the parameter setting, the gain is the same and the SNR threshold may be different.

FIG. 8 is a schematic view of the device 100 comprising a TF 140 used on the sensor side 110 and in a camera module 820, according to an embodiment of the disclosure; and

In the embodiment of FIG. 8, the device 100 determines the TF 1405. Moreover, the transfer function 140 is used on the side of the image sensor 110 and in the camera’s module 820.

The transfer function 140 may be used to solve the issue of limited bandwidth between the camera module 210 and the camera’s ISP 820. The standard connection interface is the Mobile Industry Processor Interface (MIPI) / the Camera Serial Interface (CSI). Although there has been a variant supporting higher bit depth, for legacy or cost reason, the 12 bit bandwidth often exists in industry. The HDR camera sensor data, however, are often 20 bits or 24 bits, and there is a trend for even higher bit depth. Therefore, the HDR image data are quantized to 12 bit in this scenario. Direct compression of the image sensor data, however, results in poor performance. And the transfer function 105 may be used before quantization.

The camera ISP pipeline 820 comprises the inverse of the transfer function 821, an inverse logarithmic unit 822, and the other ISP components 823.

The adaptive transfer function 140 is applied to the sensor raw data, e.g., 16 bits. Then, the data are quantized or compressed to low bit depth, e.g., 12 bit, in order to fit in the mipi/csi standard. Here, the image data are quantized, e.g., compressed after the TF 140 and before the image data and transfer function being submitted to the camera ISP 820. At this stage, apart from the image data, the transfer function also needs to be transmitted to the ISP.

Typically, the TF 140 may be sub sampled and the sample values, often 128 values or 256 values, may be transmitted. At the beginning of the ISP pipeline, the inverse of the transfer function 821 is applied. Next, inverse of the logarithmic domain is applied in the inverse logarithmic domain unit 822. Thereby, the signal is converted from logarithmic domain back to linear domain. These data may be further processed by the ISP pipeline and the output is used in CV applications 150.

FIG. 9 shows a method 900 according to an embodiment of the disclosure for pre processing image data 111. The method 900 may be carried out by the device 100, as it is described above.

The method 900 comprises a step 901 of obtaining, by means of a channel 101, image data 111 of an image sensor 110, wherein the image data 111 comprises an intensity value for each of a plurality of image sensor pixels associated with the channel 101.

The method 900 further comprises a step 902 of determining, in a logarithmic domain, a histogram 120 of the image data 111 based on a noise profile 130 of the image sensor 110, the histogram 120 comprising a plurality of bins, wherein each bin from the plurality of bins is associated with a set of intensity values of the image data 111.

The method 900 further comprises a step 903 of determining a TF 140 based on the histogram 120 of the image data 111.

The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed disclosure, from the studies of the drawings, this disclosure, and the independent claims. In the claims as well as in the description the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.