BACKGROUND

[0001] The present disclosure relates to the processing of digital images. Specifically, some embodiments relate to techniques for reducing mosaic effects that can occur when full-color images are captured using a Bayer filter array.

SUMMARY

[0002] A method according to some embodiments includes: obtaining a first array of image samples; obtaining a second array of image samples; generating a combined array of image samples, wherein, for at least a first sample in the combined array: a first value associated with a first color component is obtained from a corresponding position in the first array; a second value associated with a second color component is obtained from a corresponding position in the second array; and a third value associated with a third color component is obtained by interpolating a value from at least one of the first and second image samples. [0003] In some embodiments, for at least a second sample in the combined array: a first value associated with the first color component is obtained from a corresponding position in the second array; a second value associated with the second color component is obtained by interpolating a value from at least one of the first and second image samples; and a third value associated with the third color component is obtained from a corresponding position in the first array. The second sample may neighbor the first sample either vertically or horizontally.

[0004] In some embodiments, for at least a third sample in the combined array: a first value associated with the first color component is obtained from a corresponding position in the second array; a second value associated with the second color component is obtained from a corresponding position in the first array; and a third value associated with the third color component is obtained by interpolating a value from at least one of the first and second image samples. The third sample may neighbor the first sample either vertically or horizontally. [0005] In some embodiments, for at least a fourth sample in the combined array: a first value associated with the first color component is obtained from a corresponding position in the first array; a second value associated with the second color component is obtained by interpolating a value from at least one of the first and second image samples; and a third value associated with the third color component is obtained from a corresponding position in the second array. The fourth sample neighbors the first sample diagonally.

[0006] In some embodiments, the first and second arrays of image samples are obtained using a Bayer filter array. The Bayer filter arrays used to capture the first and second arrays of image samples may have substantially perpendicular orientations.

[0007] In some embodiments, the first array of images samples is obtained using a first image sensor on a user device and the second array of images samples is obtained using a second image sensor on the user device, the first and second image sensors having substantially perpendicular orientations. The first array of image samples and the second array of image samples may be captured substantially simultaneously using the first and second image sensors. The substantially simultaneous capture of the first and second array of image samples may be performed in response to an input on the user device triggering a photograph.

[0008] In some embodiments, generating the combined array of image samples may be performed in response to a determination that the user device is not in a macro mode. In some embodiments, generating the combined array of image samples may be performed in response to a determination that an autofocus distance is greater than a threshold distance.

[0009] In some embodiments, the first color component is green, the second color component is red, and the third color component is blue.

[0010] In some embodiments, the first color component is green, the second color component is blue, and the third color component is red.

[0011] A method in some embodiments includes: capturing a first input image using a first Bayer filter array and a second input image using a second Bayer filter array, where the first and Bayer filter arrays have substantially perpendicular orientations; and obtaining a combined image from the first image and the second image, wherein each pixel in the combined image includes a first color component obtained from the first image, a second color component obtained from the second image, and a third color component obtained by interpolating a value from at least one of the first and second input images.

[0012] Further embodiments include an apparatus comprising a processor configured to perform any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1 is a system diagram illustrating an example wireless transmit/receive unit (WTRU) that may be used in some embodiments.

[0014] FIG. 2 is a schematic illustration of a demosaicing process performed on an image captured using a Bayer filter.

[0015] FIG. 3 is a schematic illustration of a demosaicing process performed on an image captured using a Bayer filter, illustrating which resulting samples that are interpolated and which samples are not interpolated.

[0016] FIG. 4 is a schematic illustration of image samples captured using a Bayer filter in a landscape orientation, which may be used as an input to methods performed according to some embodiments.

[0017] FIG. 5 is a schematic illustration of image samples captured using a Bayer filter in a portrait orientation, which may be used as an input to methods performed according to some embodiments.

[0018] FIG. 6 is a schematic illustration of an example demosaicing method according to some embodiments, combining image samples as captured according to FIGs. 4 and 5.

[0019] FIG. 7 is a schematic illustration of a mobile device with two image sensors that may be used to capture a portrait-oriented image and a landscape-oriented image for processing according to some embodiments.

[0020] FIGs. 8A and 8B illustrates arrays of image samples that may be combined in a demosaicing process according to some embodiments.

[0021] FIG. 9 illustrates an array of image samples that may be generated from the image samples of FIGs. 8A and 8B.

[0022] FIG. 10 illustrates a method according to some embodiments. EXAMPLE APPARATUS FOR IMPLEMENTATION OF THE EMBODIMENTS

[0023] FIG. 1 is a system diagram illustrating an example wireless transmit receive unit (WTRU) 102. As shown in FIG. 1 B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and/or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0024] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 1 depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0025] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In an embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and/or receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0026] Although the transmit/receive element 122 is depicted in FIG. 1 as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ Ml MO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0027] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11 , for example.

[0028] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random- access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0029] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0030] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0031] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.

[0032] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0033] Although the WTRU is described in FIG. 1 as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

DETAILED DESCRIPTION

[0034] Devices, such as for example a WTRU, may have multiple cameras whose sensors may be portrait or landscape oriented. Example embodiments combine information from different camera sensors, which may have different orientations, for use in a debayering or other demosaicing process.

[0035] The pixels of cameras sensors are generally designed to capture light (photons) from a broad spectrum to give a “luminance” frame. To get color information, various different techniques may be used. In one such technique, three frames are captured with three different filters (Red/Green/Blue) at substantially the same time. With the red filter, each pixel is red, and so on for green and blue. By combining the 3 (RGB) frames, a color picture is obtained. This can be done with three sensors (each dedicated to a color) or one sensor with three filters. In other techniques, one frame is captured, and a Bayer filter array is applied on the sensor. An interpolation algorithm process (for example: nearest neighbor, bilinear, smooth hue, or variable number of gradients (VNG)) is used to obtain a color picture. FIG. 2 is a schematic illustration of an example of a demosaicing process with a Bayer matrix.

[0036] FIG. 3 is a schematic illustration of a demosaicing process with a Bayer matrix, illustrating that the color values of some pixels are interpolated. In some processes, 75% of the red and blue pixels are interpolated and 50% of the green pixels are interpolated.

[0037] Example embodiments are intended to increase the RGB output quality by decreasing the number of interpolated pixels. Some embodiments operate to make use of multiple camera orientations to increase demosaicing process quality.

[0038] In an example embodiment, a first image sensor of a device is in “landscape mode,” which provides color samples as illustrated in FIG. 4. On the same device, a second image sensor is in “portrait mode,” which provides color samples as illustrated in FIG. 5. The portrait mode sensor in this example is rotated by +90° or -90° with respect to the landscape mode sensor. FIG. 6 illustrates the merging of the sensor captures. As illustrated in FIG. 6, the merger of the sensor captures allows for a reduced level of interpolation. In this example, only 50% of the red and blue pixels are now interpolated and none of the green pixels are interpolated.

[0039] In some embodiments, calibration between the two sensors is performed during manufacturing. In some embodiments, the subjects of the images sensor captures are distant enough from the device to prevent a disparity effect between the two sensors (e.g. a disparity effect of less than one pixel). In some embodiments, the merging of sensor captures is not performed during macro mode.

[0040] Interpolation may be performed by hardware or software. Different interpolation techniques may be selected to achieve a balance between complexity, power consumption, and reactivity. Interpolation will typically be better with more “true” pixel colors.

[0041] FIG. 7 illustrates an example of a device (e.g. a WTRU) employed in some embodiments. The example device includes an image sensor with a portrait orientation and another image sensor with a landscape orientation. The image sensors can be configured to capture images at least partly of the same entity (object, scene, target, aim,...), which typically means that the image sensors are aimed in the same direction.

[0042] In some embodiments, the image captures are performed in portrait and landscape mode at the same time. One potential benefit of such an embodiment is that it is not necessary to select one orientation to take the picture. Images is captured by both sensors, and the user may subsequently select whether to keep one or both of the images. In addition, it may be easier to hold the device in a portrait mode even if it is desired to capture an image in landscape mode.

[0043] Such a device equipped with both portrait and landscape image sensors may be used to implement a demosaicing process as described herein. In an example use, images are captured substantially simultaneously by the two sensors, and a merge between landscape/portrait sensors is made before algorithm interpolation. The capture of the images may be performed in response to a user input triggering the capture of a photograph (e.g. pressing a shutter button on a camera app).

[0044] Example embodiments are not limited to the capture of still images. The methods described herein for processing of images may be performed in some embodiments on the frames of video images, for example to merge frames of video captured with a sensor having a portrait orientation and video captured with a sensor having a landscape orientation.

[0045] In some embodiments, a camera application on a WTRU, such as a smartphone or other user device, a specific color mode (“Enhanced colors” for example) may be activated, in response to which a demosaicing process such as those described herein may be implemented. In some embodiments, this mode is disabled for close subjects, such as when macro mode is in use. In some embodiments, the “Enhanced colors” mode may be automatically disabled when the autofocus is focusing on very close subjects.

[0046] In a method according to some embodiments, a first input image is using a first Bayer filter array and a second input image using a second Bayer filter array. The first and second Bayer filter arrays have at least substantially perpendicular orientations. A combined image is obtained from the first image and the second image, wherein each pixel in the combined image includes a first color component obtained from the first image, a second color component obtained from the second image, or a third color component obtained by interpolating a value from at least one of the first and second input images.

[0047] While some example embodiments operate to combine images from a sensor with a portrait orientation and a sensor with a landscape orientation, the principles described herein may also be implemented using sensors with other relative orientations or with the same orientation. In this regard, it may be noted that a Bayer filter array that is turned 90° has the same pattern of colors as a Bayer filter array that has been shifted by, for example, one pixel in an appropriate direction. The principles described herein may thus be employed with images captured using Bayer filter arrays that are appropriately rotated and/or shifted with respect to one another.

[0048] Figure 10 illustrates a method according to an embodiment of the present principles. In an example embodiment, in step S1002, a first array of image samples is obtained. FIG. 8A illustrates an example portion of a first array of image samples, such as image samples captured with an image sensor in portrait mode. Each of the image samples is associated with a color component. For example, the image sample Gi _,x -i _,y -i is an image sample in the first array that is at position (x-1 ,y-1) and is associated with a green color component. The image sample Bi _,x,y -i is an image sample in the first array that is at position (x,y-1) and is associated with a blue color component. The image sample Ri _,x -i _,y is an image sample in the first array that is at position (x-1 ,y) and is associated with a red color component.

[0049] In step S1004, a second array of image samples is also obtained. FIG. 8B illustrates an example portion of a second array of image samples, such as image samples captured with an image sensor in landscape mode. Each of the image samples in the second array is associated with a color component. For example, the image sample R2 _,i -i _j -i is an image sample in the second array that is at position (i-1 ,j-1 ) and is associated with a red color component. The image sample G2 _, U- _I is an image sample in the second array that is at position (i,j-1) and is associated with a green color component. The image sample B2 _, u is an image sample in the second array that is at position (i,j) and is associated with a blue color component.

[0050] In step S1006, the first and second arrays of image samples are merged. Merging of the first and second arrays of image samples may make use of a mapping between positions in the first array and positions in the second array. For example, a mapping may indicate that position (i,j) in the first array corresponds to position (x,y) in the second array, where i=x+a and j=y+b.

[0051] In this example method, a combined array of image samples is generated based on the first and second arrays of image samples. An example of a portion of a combined array of image samples is illustrated in FIG. 9. The image samples in the combined array are generated based on the arrays of FIGs. 8A and 8B. The samples in the combined array may be have positions indicated by, for example, coordinates (p,q). Each position (p,q) in the combined array corresponds to a sample position in the first array and a sample position in the second array. In the example of FIG. 9, the sample at position (p,q) in the combined array corresponds to position (x,y) in the first array and to position (i,j) in the second array. More generally, samples at positions (x+c,y+d), (p+c,q+d), and (i+cj+d) correspond to one another, for various positive and negative integer values of c and d. The correspondence or mapping between sample positions may be predetermined in manufacturing of an image capturing device, it may be determined during a calibration procedure, or it may be determined by comparing features of two captured images to align those images, among other examples. It may be noted that not every pixel position in one array necessarily has a corresponding pixel in another array. For example, as seen in FIG. 6, there are portions of a landscape image that do not overlay any corresponding area in a portrait image, and vice versa.

[0052] For at least one of the samples 902 in the combined array of FIG. 9, the sample has a first value G3 _{,P+i ,q+i} associated with a first color component (green, in this example), which is obtained from the value Gi _,x+i,y+i from the corresponding position in the first array. The sample 902 has a second value R3 _,P+i,q+i associated with a second color component (red, in this example) that is obtained from a corresponding position in the second array. The sample 902 has a third value B3 _,p+i,q+i associated with a third color component (blue, in this example) that is obtained by interpolating a value from at least one of the first and second image samples. In the particular example of FIG. 9, this interpolated value is obtained as the average of the two nearest samples associated with the same color component, such that B ₃ ,p ₊₁ ,q ₊₁ = (Bi , _x ,y+i+Bi , _x +2,y+i)/2. However, other interpolation techniques that use samples from the first and/or second arrays may alternatively be used.

[0053] In some embodiments, for another one of the samples, 904, in the combined array, a first value G3 _,P,q+i associated with the first color component (e.g. green) is obtained from a corresponding position in the second array, such that G3, _P , _q +i = G2,i,j+i . A second value R3, _P , _q +i associated with the second color component (e.g. red) is obtained by interpolating a value from at least one of the first and second image samples. For example, in some embodiments, R3, _P , _q +i = (R2j- i ,j+i+R2,i+i ,j+i)/2. A third value B3, , _q +i associated with the third color component (e.g. blue) is obtained from a corresponding position in the first array, such that B3 _{, ,q+i} = Bi , _x ,y+i . The sample 904 may neighbor the sample 902 either vertically or horizontally.

[0054] In some embodiments, for another one of the samples, 906, a first value G3, _P +i , _q associated with the first color component is obtained from a corresponding position in the second array, such that G3, _P +i , _q = G2,i+i j. A second value R3, _P +i , _q associated with the second color component is obtained from a corresponding position in the first array, such that R3, _P +i , _q = Ri , _x +i , _y . A third value B3, _P +i , _q associated with the third color component is obtained by interpolating a value from at least one of the first and second image samples. The sample 906 may neighbor the sample 902 either vertically or horizontally.

[0055] In some embodiments, for another one of the samples, 908, a first value G3 _,p,q associated with the first color component is obtained from a corresponding position in the first array, such that G3 _,P,q = Gi _,x,y . A second value R3 _,P,q associated with the second color component is obtained by interpolating a value from at least one of the first and second image samples. A third value B3 _,P,q associated with the third color component is obtained from a corresponding position in the second array, such that B3, _P,q = B2,ij. The sample 908 may neighbor sample 902 diagonally.

[0056] While the color components are primarily described herein as being green, red, and blue components, other components may alternatively be used. While some samples are described herein as being obtained using a Bayer filter array, the principles described herein may also be employed on arrays of samples collected using techniques other than Bayer filter arrays. It may also be noted that the methods described herein for generating a combined array of samples are not necessarily performed on the same device or devices that captured the samples in the first place. For example, raw image data captured by a first and second sensor on a user’s mobile device may later be combined in a post-processing method, which may be performed on a different device, such as the user’s home computer or a cloud-based processor.

[0057] In some embodiments, one or more of the methods described herein is performed by an apparatus that includes processor configured to perform the described methods. The processor may be configured to perform the methods using appropriate instructions stored in a computer-readable medium (e.g. a non- transitory medium).

[0058] Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one ormore processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0059] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.