Login| Sign Up| Help| Contact|

Patent Searching and Data


Title:
DEVICE AND METHOD FOR A MULTI-CAMERA SYSTEM
Document Type and Number:
WIPO Patent Application WO/2022/167096
Kind Code:
A1
Abstract:
The present disclosure relates to a multi-camera system. The disclosure proposes a processing entity for a multi-camera system comprising a first and second cameras, wherein the first camera refers to a first viewing area, the second camera refers to a second viewing area that is at least partially overlapped with the first viewing area. The processing entity is configured to: match a first image segment referring to the first camera and a second image segment referring to the second camera, wherein the first and second image segments refer to at least partially the same viewing area segment; determine a transformation specification based on a matching area of the first and second image segments, wherein the transformation specification is based on a characteristic of the first camera; and apply the transformation specification to the second viewing area of the second camera.

Inventors:
GULDOGAN ESIN (SE)
ILMONIEMI MARTTI (SE)
Application Number:
PCT/EP2021/052935
Publication Date:
August 11, 2022
Filing Date:
February 08, 2021
Export Citation:
Click for automatic bibliography generation   Help
Assignee:
HUAWEI TECH CO LTD (CN)
GULDOGAN ESIN (SE)
International Classes:
G06T5/50
Foreign References:
US20130250123A12013-09-26
Other References:
TRINIDAD MARC COMINO ET AL: "Multi-View Image Fusion", 2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, 27 October 2019 (2019-10-27), pages 4100 - 4109, XP033723579, DOI: 10.1109/ICCV.2019.00420
GUO PEIYAO ET AL: "Low-Light Color Imaging via Dual Camera Acquisition", COMPUTER VISION - ACCV 2020, 30 November 2020 (2020-11-30), Cham, pages 150 - 167, XP055851497, ISBN: 978-3-030-69531-6, Retrieved from the Internet [retrieved on 20211014], DOI: 10.1007/978-3-030-69532-3_10
XIAOYUN YUAN ET AL: "Multiscale gigapixel video: A cross resolution image matching and warping approach", 2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL PHOTOGRAPHY (ICCP), May 2017 (2017-05-01), pages 1 - 9, XP055394291, ISBN: 978-1-5090-5745-0, DOI: 10.1109/ICCPHOT.2017.7951481
DING CHUNQIU ET AL: "Multi-Camera Color Correction via Hybrid Histogram Matching", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS, US, vol. 31, no. 9, 17 November 2020 (2020-11-17), pages 3327 - 3337, XP011876573, ISSN: 1051-8215, [retrieved on 20210901], DOI: 10.1109/TCSVT.2020.3038484
"Recent Advances in Image and Video Coding", 23 November 2016, INTECH, ISBN: 978-953-51-2776-5, article QUEVEDO EDUARDO ET AL: "Approach to Super-Resolution Through the Concept of Multicamera Imaging", XP055851502, DOI: 10.5772/65442
TAN YANG ET AL: "CrossNet++: Cross-scale Large-parallax Warping for Reference-based Super-resolution", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 23 May 2020 (2020-05-23), pages 1 - 1, XP055851031, Retrieved from the Internet [retrieved on 20211013], DOI: 10.1109/TPAMI.2020.2997007
Attorney, Agent or Firm:
KREUZ, Georg (DE)
Download PDF:
Claims:
Claims

1. A processing entity (100) for a multi-camera system (10), wherein the multicamera system (10) comprises a first camera (11) and a second camera (12), wherein the first camera (11) refers to a first viewing area, the second camera (12) refers to a second viewing area that is at least partially overlapped with the first viewing area, and wherein the processing entity (100) is configured to: match a first image segment referring to the first camera (11) and a second image segment referring to the second camera (12), wherein the first and second image segments refer to at least partially the same viewing area segment; determine a transformation specification (101) based on a matching area (102) of the first and second image segments, wherein the transformation specification (101) is based on a characteristic of the first camera (11), wherein the characteristic of the first camera (11) refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera (12) refers; and apply the transformation specification (101) to the second viewing area of the second camera (12).

2. The processing entity (100) according to claim 1, configured to: determine the characteristic of the first camera (11) and the same characteristic of the second camera (12) based on at least the matching area (102) of the first and second image segments.

3. The processing entity (100) according to claim 1 or 2, wherein the characteristic comprises at least one of color information, a super-resolution function, and a point spread function.

4. The processing entity (100) according to claim 3, wherein the characteristic comprises the color information, and the transformation specification (101) comprises a color transformation specification for transforming the color information of the second camera (12) to the color information of the first camera (11), wherein the processing entity (100) is further configured to: determine the color information of the first camera (11) and the color information of the second camera (12) based on at least the matching area of the first and second image segments; calculate the color transformation specification based on the color information of the first and second cameras (10, 12); and apply the color transformation specification to the second viewing area of the second camera (12).

5. The processing entity (100) according to claim 4, wherein the color transformation specification comprises at least one of a color transformation matrix and a 3D lookup table.

6. The processing entity (100) according to claim 3, wherein the characteristic comprises the super-resolution function or the point spread function, the transformation specification (101) comprises a kernel of the super-resolution function or the point spread function for transforming the super-resolution function or the point spread function of the second camera (12) to the super-resolution function or the point spread function of the first camera (11), wherein the processing entity (100) is further configured to: determine the super-resolution function or the point spread function of the first camera (11) and the super-resolution function or the point spread function of the second camera (12) based on at least the matching area of the first and second image segments; estimate the kernel of the super-resolution function or the point spread function based on the super-resolution functions or the point spread functions of the first and second cameras (11, 12); and apply the kernel to the second viewing area of the second camera (12).

7. The processing entity (100) according to claim 6, wherein the first camera (11) is a tele camera, and the second camera (12) is a main camera.

8. The processing entity (100) according to one of the claims 1 to 7, wherein the matching area comprises at least partially a region-of-interest.

9. A terminal (1) comprising a multi-camera system (10) and a processing entity (100) according to one of the claims 1 to 8.

10. A method (500) performed by a processing entity (100) for a multi-camera system (10), wherein the multi-camera system (10) comprises a first camera (11) and a second camera (12), wherein the method comprises: matching (501) a first image segment referring to the first camera (11) and a second image segment referring to the second camera (12), wherein the first and second image segments refer to at least partially the same viewing area segment; determining (502) a transformation specification (101) based on a matching area (102) of the first and second image segments, wherein the transformation specification (101) is based on a characteristic of the first camera (11), wherein the characteristic of the first camera (11) refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera (12) refers; and applying (503) the transformation specification (101) to the second viewing area of the second camera (12).

11. The method according to claim 10, comprising: determining the characteristic of the first camera (11) and the same characteristic of the second camera (12) based on at least the matching area of the first and second image segments.

12. The method according to claim 10 or 11, wherein the characteristic comprises at least one of color information, a super-resolution function, and a point spread function.

13. The method according to claim 12, wherein the characteristic comprises the color information, the transformation specification (101) comprises a color transformation specification for transforming the color information of the second camera (12) to the color information of the first camera (11), wherein the method further comprising: determining the color information of the first camera (11) and the color information of the second camera (12) based on at least the matching area of the first and second image segments; calculating the color transformation specification based on the color information of the first and second camera (12)s; and

18 applying the color transformation specification to the second viewing area of the second camera (12).

14. The processing entity (100) according to claim 13, wherein the color transformation specification comprises at least one of a color transformation matrix and a 3D lookup table.

15. The method according to claim 12, wherein the characteristic comprises the super-resolution function or the point spread function, the transformation specification (101) comprises a kernel of the super-resolution function or the point spread function for transforming the super-resolution function or the point spread function of the second camera (12) to the super-resolution function or the point spread function of the first camera (11), wherein the method further comprising: determining the super-resolution function or the point spread function of the first camera (11) and the super-resolution function or the point spread function of the second camera (12) based on at least the matching area of the first and second image segments; estimating the kernel of the super-resolution function or the point spread function of the first camera (11) based on the super-resolution functions or the point spread functions of the first and second camera (12)s; and applying the kernel to the second viewing area of the second camera (12).

16. The method according to claim 15, wherein the first camera (11) is a tele camera, and the second camera (12) is a main camera.

17. The method according to one of the claims 10 to 16, wherein the matching area comprises at least partially a region-of-interest.

18. A computer program product comprising a program code for carrying out, when implemented on a processor, the method according to one of the claims 10 to 17.

19

Description:
DEVICE AND METHOD FOR A MULTI-CAMERA SYSTEM

TECHNICAL FIELD

The present disclosure relates generally to a multi-camera system. Particularly, the present disclosure relates with utilizing a feature from one camera for another camera of the multicamera system, in order to improve a final image performance. The disclosure proposes, to this end, a device and a method for transferring or sharing characteristics among different cameras in the multi-camera system. The device and the method may thus beneficially be used for improving a final-image quality of the multi-camera system.

BACKGROUND

Recent mobile devices may be equipped with multiple lenses and sensors, which have varied limits and light capturing abilities. For instance, cameras with larger lens apertures and larger pixels can capture more photons per pixel, and thus may show less prominent photon shot noise.

A current multi-camera system that comprises multiple cameras does not share any characteristics among different cameras, however, characteristics of different cameras can be merged to create different domain data, such as depth-maps etc.

SUMMARY

In view of the above-mentioned limitations, embodiments of the present disclosure aim to provide a method for a multi-camera system, wherein multiple cameras may share their characteristics. An objective is to enable learning of a preferable characteristic from one camera and applying it to another camera of the multi-camera system. One aim of the disclosure is thus to improve a final-image quality of the multi-camera system.

The objective is achieved by embodiments as provided in the enclosed independent claims. Advantageous implementations of the embodiments are further defined in the dependent claims. A first aspect of the disclosure provides a processing entity for a multi-camera system, wherein the multi-camera system comprises a first camera and a second camera, wherein the first camera refers to a first viewing area, the second camera refers to a second viewing area that is at least partially overlapped with the first viewing area, and wherein the processing entity is configured to: match a first image segment referring to the first camera and a second image segment referring to the second camera, wherein the first and second image segments refer to at least partially the same viewing area segment; determine a transformation specification based on a matching area of the first and second image segments, wherein the transformation specification is based on a characteristic of the first camera, wherein the characteristic of the first camera refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera refers; and apply the transformation specification to the second viewing area of the second camera.

The processing entity of the first aspect is accordingly designed for transferring a preferable characteristic (closer to the target image performance) from one camera to another camera of the multi-camera system. This may be achieved by learning the characteristics from paired patches (i.e., the overlapping matching area), and then determining the transformation specification accordingly. In particular, it may be assumed that one camera, i.e., the first camera, has a superior feature over the other one, i.e., the second camera. The superior feature refers to a feature that can cause a preferable image performance of the camera. The processing entity of the first aspect thus allows applying the learned feature to the whole image of the second camera, and thus improving a final-image quality.

In an implementation form of the first aspect, the processing entity is configured to determine the characteristic of the first camera and the same characteristic of the second camera based on at least the matching area of the first and second image segments.

Notably, characteristics from matching patches can be easily learned or determined. This may also be applicable to the whole entire image. Moreover, there could be some cases where characteristics from matching patches can be learned from the entire image of one camera, and then can be applied to the entire image of another camera. In an implementation form of the first aspect, the characteristic comprises at least one of color information, a super-resolution (SR) function, and a point spread function (PSF).

The transferable characteristics discussed in this disclosure could be color, a SR function, or PSF. Typically, SR is the task of generating a high-resolution (HR) image from a given low-resolution (LR) image. The PSF typically describes the response of an imaging system to a point source or point object.

In an implementation form of the first aspect, the characteristic comprises the color information, and the transformation specification comprises a color transformation specification for transforming the color information of the second camera to the color information of the first camera, wherein the processing entity is further configured to: determine the color information of the first camera and the color information of the second camera based on at least the matching area of the first and second image segments; calculate the color transformation specification based on the color information of the first and second cameras; and apply the color transformation specification to the second viewing area of the second camera.

Generally, there may be different color sensors in a multi-camera system. For example, one camera could have a RYYB color sensor (i.e., one red, two yellow, and one blue), and another camera could have a RGB color sensor (i.e., red, green, and blue). For instance, the first camera may comprise a RGB color sensor, while the second camera may comprise a RYYB color sensor. Typically, RYYB sensors are small in size and have low- performance in color in certain illuminations. However, some of the color cannot be distinguished in RYYB sensors. For those cases, the RGB sensor has superior quality over the RYYB sensor.

In an implementation form of the first aspect, the color transformation specification comprises at least one of a color transformation matrix and a 3D lookup table.

In particular, a color transformation matrix can be used on transformation of the representation of a color from one color space to another. For instance, by multiplying the second viewing area of the second camera with the color transformation matrix, the color correction is transferred to the second camera with RYYB color sensor. 3D lookup tables can also be used to map one color space to another.

In an implementation form of the first aspect, the characteristic comprises the SR function or the point spread function, the transformation specification comprises a kernel of the SR function or the point spread function for transforming the SR function or the point spread function of the second camera to the SR function or the point spread function of the first camera, wherein the processing entity is further configured to: determine the SR function or the point spread function of the first camera and the SR function or the point spread function of the second camera based on at least the matching area of the first and second image segments; estimate the kernel of the SR function or the point spread function based on the SR functions or the point spread functions of the first and second cameras; and apply the kernel to the second viewing area of the second camera.

The basic model of SR assumes that a LR image is the result of down-scaling a HR image, by a scaling factor using some kernel. Thus, in order to recover the HR image from a given LR image, the kernel can be estimated and then used to recover the HR image.

In an implementation form of the first aspect, the first camera is a tele camera, and the second camera is a main camera.

Notably, a tele camera may have much better details than a main camera of the multicamera system.

In an implementation form of the first aspect, the matching area comprises at least partially a region-of-interest (ROI).

Notably, ROI may refer to for example an important object or objects.

A second aspect of the disclosure provides a terminal comprising a multi-camera system and a processing entity according to the first aspect and any of implementation forms of the first aspect. A third aspect of the disclosure provides a method performed by a processing entity for a multi-camera system, wherein the multi-camera system comprises a first camera and a second camera, wherein the method comprises: matching a first image segment referring to the first camera and a second image segment referring to the second camera, wherein the first and second image segments refer to at least partially the same viewing area segment; determining a transformation specification based on a matching area of the first and second image segments, wherein the transformation specification is based on a characteristic of the first camera, wherein the characteristic of the first camera refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera refers; and applying the transformation specification to the second viewing area of the second camera.

In an implementation form of the third aspect, the method further comprises determining the characteristic of the first camera and the same characteristic of the second camera based on at least the matching area of the first and second image segments.

In an implementation form of the third aspect, the characteristic comprises at least one of color information, a SR function, and a point spread function.

In an implementation form of the third aspect, the characteristic comprises the color information, the transformation specification comprises a color transformation specification for transforming the color information of the second camera to the color information of the first camera, wherein the method further comprises: determining the color information of the first camera and the color information of the second camera based on at least the matching area of the first and second image segments; calculating the color transformation specification based on the color information of the first and second cameras; and applying the color transformation specification to the second viewing area of the second camera.

In an implementation form of the third aspect, the color transformation specification comprises at least one of a color transformation matrix and a 3D lookup table.

In an implementation form of the third aspect, the characteristic comprises the SR function or the point spread function, the transformation specification comprises a kernel of the SR function or the point spread function for transforming the SR function or the point spread function of the second camera to the SR function or the point spread function of the first camera, wherein the method further comprises: determining the SR function or the point spread function of the first camera and the SR function or the point spread function of the second camera based on at least the matching area of the first and second image segments; estimating the kernel of the SR function or the point spread function of the first camera based on the SR functions or the point spread functions of the first and second cameras; and applying the kernel to the second viewing area of the second camera.

In an implementation form of the third aspect, the first camera is a tele camera, and the second camera is a main camera.

In an implementation form of the third aspect, the matching area comprises at least partially a ROI.

A fourth aspect of the disclosure provides a computer program product comprising a program code for carrying out, when implemented on a processor, the method according to the third aspect and any implementation forms of the third aspect.

It has to be noted that all devices, elements, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

BRIEF DESCRIPTION OF DRAWINGS The above described aspects and implementation forms of the present disclosure will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which

FIG. 1 shows a processing entity according to an embodiment of the disclosure.

FIG. 2 shows a procedure of transferring characteristics among different cameras according to an embodiment of the disclosure.

FIG. 3 shows an example showing transferring a SR kernel from the first camera to the second camera according to an embodiment of the disclosure.

FIG. 4 shows a terminal according to an embodiment of the disclosure.

FIG. 5 shows a method according to an embodiment of the disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

Illustrative embodiments of the disclosure are described with reference to the figures. Although this description provides a detailed example of possible embodiments and implementations, it should be noted that the details are intended to be exemplary and in no way limit the scope of the application.

Moreover, an embodiment/example may refer to other embodiments/examples. For example, any description including but not limited to terminology, element, process, explanation and/or technical advantage mentioned in one embodiment/example is applicative to the other embodiments/examples.

FIG. 1 shows a processing entity 100 for a multi-camera system 10, according to an embodiment of the disclosure. The processing entity 100 may comprise processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the processing entity 100 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. The processing entity 100 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software. For instance, the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the various operations of the processing entity 100 to be performed. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the processing entity 100 to perform, conduct or initiate the operations or methods described herein.

In particular, the multi-camera system 10 comprises a first camera 11 and a second camera 12, wherein the first camera 11 refers to a first viewing area, and the second camera 12 refers to a second viewing area that is at least partially overlapped with the first viewing area. For instance, the first camera 11 may a tele camera, and the second camera may be a main camera. Generally speaking, the tele camera or telephoto camera, is capable of taking detailed photographs of distant objects, typically by means of a telephoto lens. The main camera may refer to a high quality RGB sensor based camera with a shorter focal length. For example, a main camera may have a focal length of 23mm at an aperture of f/ 1.9. A telephoto camera may have a longer focal length, for instance, of 240mm at an aperture of f/4.4. The main camera may produce a final image for the multi-camera system. Notably, the tele camera view and the main camera view may overlap with each other. The second camera may also be a wide-angle camera. The multi-camera system may also comprise more than two cameras, e.g., a telephoto camera, a main camera, and a wide-angle camera, and any two cameras of these more than two cameras may be the first camera and second camera described in this disclosure.

The processing entity 100 is configured to match a first image segment referring to the first camera 11 and a second image segment referring to the second camera 12. The first and second image segments refer to at least partially the same viewing area segment. The processing entity 100 is further configured to determine a transformation specification 101 based on a matching area 102 of the first and second image segments. In particular, the transformation specification 101 is based on a characteristic of the first camera 11, wherein the characteristic of the first camera 11 refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera 12 refers. Then, the processing entity 100 is configured to apply the transformation specification 101 to the second viewing area of the second camera 12.

Embodiments of this disclosure propose methods to transfer a preferable characteristic from one camera to another camera in the same multi-camera system, thereby to improve a final-image quality. In this disclosure, it is assumed that there is one camera in the multicamera system 10, i.e., the first camera 11, which has a superior feature or characteristic over other camera(s), e.g., the second camera 12. The superior feature refers to a feature or characteristic that can cause a preferable image performance of the camera.

According to an embodiment of the disclosure, the processing entity 100 may be configured to determine the characteristic of the first camera 11 and the same characteristic of the second camera 12 based on at least the matching area 102 of the first and second image segments.

Notably, characteristics from matching patches (areas) can be easily learned or determined. Moreover, there could be some cases where characteristics from matching patches can be learned from the entire image of one camera and further applied to the other entire image of another camera.

The transferable characteristics discussed in this disclosure may comprise at least one of color information, SR function, or PSF. Transferring any of these characteristics from one camera to another camera may be implemented using steps as shown in FIG. 2, which include step 1: patch matching, step 2: learning characteristics from paired patches, and step 3: applying learned characteristics to whole image. Details of these steps will be discussed in the following specific embodiments.

In an embodiment of the disclosure, the characteristic comprises the color information. Accordingly, the transformation specification 101 in this embodiment comprises a color transformation specification for transforming the color information of the second camera 12 to the color information of the first camera 11.

As previously discussed, there may be different color sensors in the multi-camera system 10. For instance, the first camera 11 may comprise a RGB color sensor, while the second camera 12 may comprise a RYYB color sensor. Typically, RYYB sensors setups are smaller in size than a RGB sensor setups, since there are no color filters required on top of the RYYB sensor (pixels), but the RYYB sensor setups have low-performance in color in certain illuminations. It may be worth mentioning that some of the color cannot be distinguished in RYYB sensors. For those cases, RGB sensors has superior quality over RYYB sensors. This disclosure thus proposes to get the correct color information from RGB sensors, learn such transformation from the paired patches, and apply it to RYYB camera for better color details.

In particular, the processing entity 100 may be further configured to determine the color information of the first camera 11 and the color information of the second camera 12 based on at least the matching area 102 of the first and second image segments. Further, the processing entity 100 may be configured to calculate the color transformation specification based on the color information of the first and second cameras 10, 12; and apply the color transformation specification to the second viewing area of the second camera 12.

In particular, a color transformation matrix can be used as the transformation specification 101. Possibly, A 3D lookup table can also be used to bring more color details to the second viewing area of the second camera 12.

Transferring the color information from one camera to another camera may be implemented in the following steps. In this embodiment, it is assumed that the first camera 11 is a tele camera with a RGB color sensor, and the second camera 12 is a main camera with RYYB color sensor.

Step 1 : Patch matching

With camera intrinsic and extrinsic parameters, an image patch from a view of the main camera corresponding with a view of the tele camera can be extracted. After a roughly patch extraction, two images can be aligned accurately. Notably, basic image matching with local-features algorithms may be utilized here, for example, an existing image matching method may be used for image alignment.

Step 2: Color matrix calculation:

A 3x3 color transformation matrix may be calculated from paired RGB patch, which is from the view of the tele camera, and apply it globally to the RYYB sensor of the main camera. 3D lookup table can be also calculated from RGB patch, for more detailed color transformation.

Step 3: Color matrix transformation

After finding the correct color matrix, the color transformation matrix (or a 3D LUT), can be used to multiply with the whole image of the main camera, in order to transfer the color correction.

In an embodiment of the disclosure, the characteristic may comprise the SR function or the PSF. Accordingly, the transformation specification 101 may comprise a kernel of the superresolution function or the point spread function for transforming the SR function or the PSF of the second camera 12 to the SR function or the PSF of the first camera 11. In particular, the processing entity 100 may be further configured to determine the SR function or the PSF of the first camera 11 and the SR function or the PSF of the second camera 12 based on at least the matching area 102 of the first and second image segments. Further, the processing entity 100 may be configured to estimate the kernel of the SR function or the PSF based on the SR function or the PSF of the first and second cameras 11, 1, and then apply the kernel to the second viewing area of the second camera 12.

As previously discussed, SR is the task of generating a HR image from a given LR image. Taking SR application as a use-case scenario, transferring the SR kernel from one camera to another camera may be implemented in the following steps. In this embodiment, it is assumed that the first camera 11 is a tele camera, and the second camera 12 is a main camera. As shown in FIG. 3, the tele camera is capable of taking detailed photographs of distant objects, thus the view of the tele camera (the right photo of FIG. 3) has more detailed information.

Step 1 : Patch matching With camera intrinsic and extrinsic parameters, an image patch from a view of the mam camera corresponding with a view of the tele camera can be extracted, as shown in the middle photo of FIG. 3. After a roughly patch extraction, two images can be aligned accurately. Notably, basic image matching with local-features algorithms may be utilized here, for example, an existing image matching method may be used for image alignment.

Step 2: Kernel Estimation

The basic model of SR assumes that the LR input image (ILR) is the result of down-scaling a HR image (IHR) by a scaling factor s using some kernel “k s ” (i.e., the "SR kernel"), namely:

ILR = (IHR * k s ) J,s (1)

The goal is to recover IHR using a given ILR. Thus, if the kernel ks can be estimated, the IHR can be recovered. In embodiments of this disclosure, state-of-art kernel estimation methods can be utilized for estimating the “k s ” kernel.

Step 3: Non-blind image SR

After finding the correct kernel, it can be applied to a global image of the main camera (i.e., the left photo of FIG. 4). The results can outperform blind SR methods.

For instance, a learning-based single image SR (SISR) method may be used for applying the SR kernel to the global images. SISR methods are continuously showing superior effectiveness and efficiency over traditional model-based methods, largely due to the end- to-end training.

In particular, an end-to-end trainable unfolding network which leverages both learningbased methods and model-based methods. Specifically, by unfolding the MAP inference via a half-quadratic splitting algorithm, a fixed number of iterations consisting of alternately solving a data subproblem and a prior subproblem can be obtained. The two subproblems then can be solved with neural modules, resulting in an end-to-end trainable, iterative network. As a result, the proposed network inherits the flexibility of model-based methods to super-resolve blurry, noisy images for different scale factors via a single model, while maintaining the advantages of learning-based methods. It may be worth mentioning that, although the above-described embodiment are refer to transferring of SR function, it also applies to the user case of transferring PSF. The PSF is a well-known term in optics, and describes the response of an imaging system to a point source or point object. It is the spatial domain version of the optical transfer function of the imaging system. The degree of spreading (blurring) of the point object is a measure for the quality of an imaging system.

Optionally, the matching area 102 may comprise at least partially a ROI, for instance, a region including human faces may be selected.

FIG. 4 shows a terminal 1 according to an embodiment of the disclosure. The terminal 1 may comprise processing circuitry (not shown) configured to perform, conduct or initiate the various operations of the terminal 1 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as ASICs, FPGAs, DSPs, or multi-purpose processors. The terminal 1 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software. For instance, the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the various operations of the terminal 1 to be performed. In one embodiment, the processing circuitry comprises one or more processors and a non- transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the terminal 1 to perform, conduct or initiate the operations or methods described herein.

In particular, the terminal 1 may comprise a multi-camera system 10 and a processing entity 100. The processing entity 100 may the processing entity 100 as shown in FIG. 1.

FIG. 5 shows a method 500 according to an embodiment of the disclosure. In a particular embodiment of the disclosure, the method 500 is performed by a processing entity 100 as shown in FIG. 1. In particular, the processing entity 100 is for a multi-camera system 10, wherein the multi-camera system 10 comprises a first camera 11 and a second camera 12. Possibly, the multi-camera system 10 may be the multi-camera system 10 as shown in FIG. 1.

In particular, the method 500 comprises: a step 501 of matching a first image segment referring to the first camera 11 and a second image segment referring to the second camera 12. The first and second image segments refer to at least partially the same viewing area segment. Then, the method 500 further comprises a step 502 of determining a transformation specification 101 based on a matching area 102 of the first and second image segments. The transformation specification 101 is based on a characteristic of the first camera 11, wherein the characteristic of the first camera 11 refers to a first image performance that is closer to a target image performance than a second image performance to which the same characteristic of the second camera 12 refers. The method further comprises a step 503 of applying the transformation specification 101 to the second viewing area of the second camera 12.

To summarize, embodiments of this disclosure propose to leam better characteristics from one camera and apply it to the other camera, in particular, the learned details may be applied to the whole other camera view with larger full-of-view (FOV). The transferable characteristics could be color, SR and PSF functions, or other features that could help to improve the final image quality.

The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed disclosure, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Furthermore, any method according to embodiments of the disclosure may be implemented in a computer program, having code means, which when run by processing means causes the processing means to execute the steps of the method. The computer program is included in a computer readable medium of a computer program product. The computer readable medium may comprise essentially any memory, such as a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable PROM), a Flash memory, an EEPROM (Electrically Erasable PROM), or a hard disk drive.

Moreover, it is realized by the skilled person that embodiments of the processing entity 100, or the event sensing device 10, comprise the necessary communication capabilities in the form of e.g., functions, means, units, elements, etc., for performing the solution. Examples of other such means, units, elements and functions are: processors, memory, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selecting units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiver units, transmitter units, DSPs, trellis-coded modulation (TCM) encoder, TCM decoder, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together for performing the solution.

Especially, the processor(s) of the processing entity 100, or the event sensing device 10 may comprise, e.g., one or more instances of a CPU, a GPU, a NPU, a processing unit, a processing circuit, a processor, an ASIC, a microprocessor, or other processing logic that may interpret and execute instructions. The expression “processor” may thus represent a processing circuitry comprising a plurality of processing circuits, such as, e.g., any, some or all of the ones mentioned above. The processing circuitry may further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.