ILLUMINATION COMPENSATION AND DEPTH LOOK-UP

TABLE TRANSMISSION IN VIDEO CODING

TECHNICAL FIELD [0001] The invention relates generally to Three-Dimensional (3D) video processing. In particular, the present invention relates to methods for illumination compensation (IC) and depth look-up table (DLT) in 3D video coding.

BACKGROUND

[0002] 3D video coding is developed for encoding/decoding videos of multiple views simultaneously captured by several cameras. Figure 1 shows the possible prediction structure used in the common test conditions for 3D video coding. The video pictures and depth maps corresponding to a particular camera position are indicated by a view identifier (V0, VI and V2 in Figure 1). All video pictures and depth maps that belong to the same camera position are associated with the same viewld. The view identifiers are used for specifying the coding order inside the access units and detecting missing views in error-prone environments. Inside an access unit, the video picture and, when present, the associated depth map with viewld equal to 0 are coded first, followed by the video picture and depth map with viewld equal to 1 , etc. The view with viewld equal to 0 (V0 in Figure 1) is also referred to as the base view or the independent view and is coded independently of the other views and the depth data using a conventional HEVC video coder.

[0003] Illumination compensation (IC)

[0004] Illumination compensation (IC) is a technique to reduce the differences caused by the different light fields of two views captured by different cameras at different locations.

[0005] In HTM, a linear IC model [1] is used to compensate the illumination discrepancy between different views. Parameters in IC model are estimated for each PU using available nearest reconstructed neighbouring pixels and therefore are not needed to be transmitted to the decoder. Applying IC or not could be decided at two different level (slice and CU level). One slice flag is coded to indicate whether IC is enabled for the CUs within the current slice. When this slice flag is off, no further information is transmitted below the slice level for IC. When this slice flag is on, an IC flag is coded to indicate whether the IC is enabled at CU level in which inter-view prediction is used. More detail of this CU-level flag is illustrated in the following section. The linear IC model used in inter-view prediction is shown in Eq.

(1):

Here, PUc is the current PU, (i, j) is the coordinate of pixels in PUc, (dvx, dvy) is the disparity vector of PUc. p(i, j) is the prediction of PUc, r is the PU's reference picture from neighboring view, a and b are parameters of the linear IC model.

[0006] To estimate parameter a and b for a PU, two set of pixels as shown in Fig.2 are used: 1. available reconstructed neighboring pixels in left column and above row of current CU (the CU that contains current PU), 2. Corresponding neighboring pixels of current CU's reference block. Reference block of current CU is found by using current PU's disparity vector. Similarly, an adaptive luminance compensation tool [2] for inter-view video encoding is also applied in ATM. Currently, this adaptive luminance compensation is only applied to P slices. An MB-level flag is transmitted for a skip MB, P16xl6, P16x8, P8xl6 and P8x8 MB to turn on/off the adaptive luminance compensation.

[0007] Signaling of IC [0008] Whether illumination compensation is used is signalled on coding unit. In Skip / Merge mode, ic flag is conditionally sent depend on merge idx and the slice segment header flag slice ic disable merge zero idx flag. If ic flag is not sent on merge mode, ic flag is inferred to 0. Detailed condition is shown in Table l .In 3D- HEVC test model, HTM-7.0, there exists a process to derive a disparity vector predictor, known as NBDV (Neighboring Block Disparity Vector). The disparity vector derived from NBDV is then used to fetch a depth block in the depth image of the reference view. The fetched depth block would have the same size of the current prediction unit (PU), and it will then be used to do backward warping for the current PU.

[0009] When merge idx is equal to 0, the temporal inter-view motion predictor candidate is typically used. Therefore, inter-view prediction is not used very often. To avoid the overhead of signalling the ic flag, in this case, illumination compensation is not available. It is utilized by setting slice ic disable merge zero idx flag 1. In some pictures where the inter-view prediction is mostly used, the above assumption does not hold. So the merge idx base skipping of the ic flag is only applied under the condition that (POC % IntraPeriod) is not 0. This POC based condition is not decided by decoder but by encoder, which sends a slice header flag slice ic disable merge zero idx flag. This allows encoders can control the condition depends on coding structure or sequences. In addition, for one inter-coded PU, illumination compensation is always disabled when ARP is applied. Therefore, when ARP weighting factor for one inter-coded PU is unequal to 0, the signalling of ic flag is skipped and it is set to 0.

[0010] Encoding algorithm of IC

[0011] In current HTM encoder, to decide whether the IC is enabled for current slice/picture, it has to do a statistic over the pixels of current picture and the pixels of the inter-view reference picture which introduces a latency of one slice processing.

[0012] To be specific, in current HTM encoder, it will first check if there is any inter-view reference picture in the reference picture list of current slice/picture. If no inter-view reference picture in the reference list, the IC will be turned off for coding current slice/picture. If at least one inter-view reference picture exists in the reference list, it will derive two histograms of pixel values over the current picture and over the inter-view reference picture. After the two histograms are derived, a summation of the absolute difference between each entry of the two histograms is calculated. If the summation of difference value is larger than a predefined threshold, IC is enabled for coding current slice/picture. Otherwise, IC is disabled for coding current slice.

[0013] Depth lookup table (DLT)

[0014] Depth lookup table (DLT) is adopted into 3D-HEVC. Since there are often only several values appearing in depth component, DLT signals those valid values from the encoder to the decoder. When a CU is coded in intra simplified depth coding

(SDC) mode or depth map modeling (DMM) mode, DLT is used to map the valid depth value to a DLT index which is much easier to compress. Fig.3 demonstrates an example of DLT approach. DLT is signaled in picture parameter set (PPS). And it is left to be an encoder issue how to get the DLT when encoding.

[0015] In the current design in HTM, as many as 24 or more depth pictures for a view are analyzed first before the encoding process start. All the values appearing in these analyzing pictures are put into the DLT for this view. This approach imposes a high encoding latency and cannot adapt well to some situations such as scene change.

SUMMARY

[0016] In the current design (HTM- 10.0), the latency of determining the slice- level enabling or disabling of IC is the process time of one slice/picture, which is too long to be used in the low delay applications.

In this invention, we propose a scheme of encoding algorithm to determine the enabling or disabling of IC for coding a picture/slice with low delay processing time. We also propose a scheme of encoding algorithm to get the DLT table for coding a picture/slice with low delay processing time.

[0017] Other aspects and features of the invention will become apparent to those with ordinary skill in the art upon review of the following descriptions of specific embodiments.

BRIEF DESCRIPTION OF DRAWINGS

The invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:

[0018] Fig. 1 is a the possible prediction structure used in the common test conditions for 3D video coding.

[0019] Fig. 2 are diagrams illustrating the pixels used to estimate the IC parameters. [0020] Fig. 3 is a diagram illustrating an example of DLT. There are five valid values appears in depth samples Therefore, there are five entries in DLT in the order: 50, 108, 110, 112 and 200.

[0021] Fig. 4 is a diagram illustrating DLT for the current picture is got by making statistics on a previously reconstructed depth picture.

[0022] Fig. 5 is a diagram illustrating DLT for the current picture depends on the statistics of two previously reconstructed depth pictures.

[0023] Fig. 6 is a diagram illustrating only one row in each two is involved in the statistics. [0024] Fig. 7 is a diagram illustrating only one sample in each four is involved in the statistics.

DETAILED DESCRIPTION

[0025] The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

[0026] Encoder decision of illumination compensation (IC)

[0027] The basic concept of the invention is to decide whether IC is enabled or disabled for current picture/slice according to the information of previously reconstructed pictures (decoded pictures) or the high level coding information of the current picture/slice without accessing the pixel values of the current picture/slice to attain low latency IC encoding. Several methods are illustrated as below:

[0028] IC encoding method 1 :

Encoder simply decides whether IC is enabled or disabled for current picture/slice depending on the statistics of the selected reference pictures/slices. The selected reference pictures/slices are previously reconstructed pictures/slices. If no selected reference pictures/slices could be found, the slice level IC enabling flag is set to a default value (enable or disable) for coding the current picture/slice.

The methods used to do the statistic could be any one or combination of the following methods:

Statistic method 1 (calculate the enabling ratio of CU level IC flag): If the IC enabling ratio of the selected reference picture is larger than a predefined threshold,

IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. Statistic method 2 (calculate the average pixel difference of the selected reference picture and its inter-view reference picture): If average pixel value between the selected reference picture and its inter-view reference picture is larger than a predefined threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. The selected reference pictures could be one of the following pictures: Selected reference picture/slice 1:

The selected reference picture/slice is the latest reconstructed picture/slice. Selected reference picture/slice 2:

The selected reference picture/slice is the latest reconstructed picture/slice with temporal ID smaller than the temporal ID of current picture/slice. Selected reference picture/slice 2:

The selected reference picture/slice is the latest reconstructed picture/slice with temporal ID smaller than or equal to the temporal ID of current picture/slice. Selected reference picture/slice 3: The selected reference picture/slice is the latest reconstructed picture/slice with QP smaller than the QP of current picture/slice. Selected reference picture/slice 4:

The selected reference picture/slice is the latest reconstructed picture/slice with QP smaller than or equal to the QP of current picture/slice.

[0029] IC encoding method 2: Encoder simply decides whether IC is enabled or disabled for current picture/slice depending on the high level coding information of the current pictures/slices.

The high level coding information of the current picture/slice could be any one or combination of the items listed as below.

High level coding information 1: If no inter-view reference picture exists in the reference picture list of current picture/slice, IC is set as disabled for coding current picture/slice; otherwise, IC is set as enabled for coding current picture/slice.

High level coding information 1:

If no temporal reference picture exists in the reference picture list of current picture/slice and only inter-view reference picture in the list, IC is set as enabled for coding current picture/slice. High level coding information 2:

If smallest POC distance between the current picture/slice and the temporal reference pictures is larger than a threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. If no temporal reference picture exists, the smallest POC distance is inferred as a default value larger than the defined threshold. High level coding information 3:

If current picture/slice is an anchor picture/slice, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. High level coding information 4: If the QP of the current picture/slice is larger than a threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice.

High level coding information 5: IC is set as enabled for the pictures/slice with temporal id equal to 0. High level coding information 6:

IC is set as enabled periodically for the pictures/slice. The period could be N picture order count (POC) where N is a positive integer. High level coding information 7: IC is set as enabled according to the slice TYPE. For example, IC is only enabled for B slices.

[0030] IC encoding method 3 :

Encoder simply decides whether Single Sample Mode is enabled or disabled for current picture/slice depending on the statistics of the selected reference pictures/slices and the high level coding information of the current pictures/slices. In this method, all the methods described in [0021] and [0022] could be used jointly. [0031] Some examples utilized the invented methods

Below we will elaborate several examples utilizing the invented methods to decided the enabling of IC for coding a slice/picture.

[0032] In one example, if no inter-view reference picture exists in the reference picture list of current picture/slice, IC is set as disabled for coding current picture/slice; otherwise, it firstly select the latest reconstructed picture/slice with temporal ID smaller than the temporal ID of current picture/slice as the selected reference picture.

Then it checks whether the IC enabling ratio is larger than a threshold in the selected reference picture; if the IC enabling ratio is larger than a threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. If no selected reference picture could be found, IC is set as enabled for coding the current picture/slice. [0033] In another example, if no inter-view reference picture exists in the reference picture list of current picture/slice, IC is set as disabled for coding current picture/slice; otherwise, it firstly select the latest reconstructed picture/slice with QP smaller than the QP of current picture/slice as the selected reference picture. Then it checks whether the IC enabling ratio is larger than a threshold in the selected reference picture; if the IC enabling ratio is larger than a threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. If no selected reference picture could be found, IC is set as enabled for coding the current picture/slice.

[0034] In another example, if the smallest POC distance between the current picture/slice and the temporal reference pictures is larger than a threshold, IC is set as enabled for coding current picture/slice; otherwise, IC is set as disabled for coding current picture/slice. If no temporal reference picture exists, the smallest POC distance is inferred as a default value larger than the defined threshold.

[0035] Encoder method for depth lookup table (DLT)

[0036] Method 1

[0037] Encoder simply gets the DLT and determines whether to use DLT for the current picture/slice depending on the statistics of the selected reference pictures/slices. The selected reference pictures/slices are previously reconstructed pictures/slices. If no selected reference pictures/slices could be found, the encoder can

• turn off DLT for the current picture/slice; or

• get the DLT for current picture/slice depending on the statistics of the current picture/slice; or

· utilize a default DLT;

• get the DLT for current picture/slice depending on the statistics of a previously reconstructed picture/slice which is not the reference pictures/slices for the current picture.

[0038] Method 2 [0039] Encoder simply gets the DLT and determines whether to use DLT for the current picture/slice depending on the statistics of a previously reconstructed picture/slice no matter it is a reference picture/slice for the current picture/slice or not. [0040] Method 3

[0041] Encoder simply gets the DLT and determines whether to use DLT for the current picture/slice depending on the statistics of two or more selected reference pictures/slices. The selected reference pictures/slices are previously reconstructed pictures/slices. If no selected reference pictures/slices could be found, the encoder can

• turn off DLT for the current picture/slice; or

• get the DLT for current picture/slice depending on the statistics of the current picture/slice; or

· utilize a default DLT; or

• get the DLT for current picture/slice depending on the statistics of one, two or more previously reconstructed picture/slices which are not the reference pictures/slices for the current picture.

[0042] Method 4 [0043] Encoder simply gets the DLT for current picture/slice depending on the statistics of two or more previously reconstructed picture/slices no matter they are reference pictures/slices for the current picture/slice or not.

[0044] Method 5

[0045] The previously reconstructed picture used to get the DLT for the current picture/slice in method 1 - method 4 can be in the same view of the current picture, or it can be in different view of the current picture.

[0046] Method 6

[0047] The DLT obtained in method 1 - method 4 can also be used by one or more pictures/slices following the current picture in coding order or displaying order.

[0048] Method 7

[0049] In, method 1 - method 4, values appearing in previously reconstructed picture/slices for statistics are put into DLT for the current view, while values not appearing in statistics previously reconstructed picture/slices are not put into DLT for the current view.

[0050] Method 8

[0051] In method 7, only parts of previously reconstructed picture/slices are used for statistics. Values appearing in the specific parts in previously reconstructed picture/slices for statistics are put into DLT for the current view, while values not appearing in the specific parts in statistics previously reconstructed picture/slices are not put into DLT for the current view.

[0052] Method 9 [0053] In method 1- method8, the statistics should be done on the original pictures/slices, instead of the reconstructed ones.

[0054] Method 10

[0055] In methodl- method8, DLT is turned off for the current picture/slice or for more pictures/slices following the current picture if the number of appearing values is greater than a number M. M is greater than 0 and lower than MaxDepth Value, which is the maximum value for depth samples. For example, M can be MaxDepth Value/2 or (MaxDepthValue+ 1 )/2.

[0056] Some examples utilized the invented methods

[0057] Examples for method 1 -method 4 [0058] As depicted in Fig. 4, DLT for the current picture is got by making statistics on a previously reconstructed picture.

[0059] Example for Method 4

[0060] As depicted in Fig. 5, DLT for the current picture depends on the statistics of two previously reconstructed pictures.

[0061] Example for Method 8

[0062] As depicted in Fig. 6, only one row in each two is involved in the statistics.

[0063] As depicted in Fig. 7, only one sample in each four is involved in the statistics.