FIELD OF THE INVENTION

This invention, in general, relates to the field of electronic imaging devices and, more particularly, to apparatus and methods for use in such devices for editing tone range and correcting for color errors.

BACKGROUND OF THE INVENTION

It is well understood that the perceived colors from objects depend on their spectral reflection characteristics and the spectral content of the sources which illuminate them. Objects that are neutral in color, that is, those which reflect equal amounts of an adopted set of primary colors, will appear neutral gray when illuminated by a source that is spectrally flat. For example, a neutral gray object illuminated by a spectrally flat source such as noontime sunlight, which has a uniform spectral power distribution, will appear more or less neutral to an observer. If a light source differs from neutral, it is said to have a "color cast" since its spectral content is no longer flat with wavelength, but rather, contains more of one "color" than another. It, therefore, has a dominant "cast" which makes it less "pure" than a neutral "white" light. For example, tungsten sources have relatively more red light than blue so will have a "reddish cast". Fluorescent lights, on the other hand, tend to the "greenish" part of the spectrum, and hence have "green casts".

Obviously, the color appearance of even neutral objects, whether in reflectance or transmittance, is influenced dramatically by the cast of the source illuminating them. And, where objects inherently are of one cast themselves, that cast can be distorted in color appearance to be something other than it would appear under neutral lighting.

While color cast issues are relatively easy to demonstrate to human observers since they can see what is taking place when differently reflecting objects are illuminated by sources of different casts, what takes place when a nonhuman detector does not have the same spectral response characteristics of the human visual

system is somewhat more subtle, but in principle similar if one can imagine replacing the spectral response of the receiver for that of a human observer. Like the human visual system other light detectors have preferential spectral sensitivities which also vary with wavelength. Video cameras and film scanners, for example, differ significantly from the human visual system in their response to color, yet all can "see" color but in different ways. Consequently, the response of the detector of color can introduce or modify the "color cast" of an image, and that color cast component will be seen by an observer as an additional color distortion. Thus, color cast problems are pervasive in electronic imaging systems, manifesting themselves in a variety of ways which make images have an "unreal" or "unnatural" color appearance to a human observer.

One example of color cast occurs in the electronic scanning and display of images. When scanning color photographic materials (reflection prints or transparencies) using an electronic scanner, and subsequently, displaying the scanned image via a monitor after applying a color transform of some sort, the displayed image often appears with an objectionable color cast which is bothersome because it is unnatural in appearance. Such color casts can be attributed to a variety of sources including, but not limited to, scene illumination conditions during photographic exposure, variation in the characteristics of the photographic materials themselves, ambient temperature conditions, or variations in subsequent chemical processing.

The color cast problem has been dealt with in a variety of ways depending on the context in which it appears. In labs for the processing and printing of conventional photographic films, controls for color cast removal are provided to the operator who assesses the degree of cast and manually adjusts for its removal. In many such systems today, automatic color cast removal is implemented by way of a low-resolution video camera which observes the processed negative, performs a calculation on the resulting data, and provides some form of correction to the negative-printing system. The computations typically are based on computing the

average color level in a negative which, on average, will produce a neutral gray in the print.

Consumer camcorders use a slightly different approach which deals primarily with variations in the illuminant color temperature. Most operate by attempting to determine a white object in the scene, and then infer a scene color cast from the white point cast. Capture circuitry is then recalibrated "on-the-fly" as recording continues. These methodologies are sometimes referred to as automatic gain control (AGC) and automatic white point control for correcting for differences in the illuminant' s color temperature, a measure of the color content of the illuminant.

Color copiers may employ steps similar to those above or various combinations of them.

Image processing software often provides for user adjustment of color cast by way of manual control over color content through the use of color sliders for R, G, B content. However, in the hands of an unskilled and inexperienced operator, such controls can be cumbersome and as problematical as they are beneficial.

While a number of approaches to the color cast problem have appeared in a variety of settings, it is a primary object of this invention to provide a systematic, highly automated methodology for the removal of color cast for use in electronic imaging systems.

It is another object of the present invention to provide for the systematic transformation, display, and removal of color casts from scanned hard copy images displayed on color monitors.

It is yet another object of the present invention to provide for the automated editing of the tonal range in a scanned color hard copy of an image to assure that neutrals in the image appear visibly gray.

Other objects of the invention will appear obvious and will appear hereinafter when reading the following detailed description.

SUMMARY OF THE INVENTION

The invention is related to electronic color processing of images in the form of electronic signals and, in particular, to systems which are not wholly automatic but involve some operator selection. It is applicable to electronic scanners used to scan photographic or other materials, photofmishing, electronic still cameras, video cameras, video or still picture image-editing software, or in the related arts. It may be used with systems using three or more color signals (e.g. RGB or CMYK). Although the preferred embodiment involves the use of digital signals, analog signal processing to perform these operations could also be implemented. Regardless of the signal source, all aspects of the invention involve the use of a color display which is viewed by an operator.

The inventive system operates to remove color casts or color balance errors from images. The system consists of three main parts: a set of cast-prediction operators which operate on the source image data and coarse device profile data, a preview mechanism whereby an image can be displayed to the operator on a calibrated color display, and a mechanism for selecting between candidate casts computed by the cast-prediction operators - which are applied to the images and displayed to the user for evaluation. The use of subsampled images for image preview and cast-removal computation is recommended, but not required.

In one aspect of the invention, the operator obtains a source image and presents it to the system, along with the coarse device profile for the image acquisition system and a device profile for the display monitor. Using the first two data sets, the system computes equivalent log-scene exposures (e.g. Red exposure, Green Exposure, Blue Exposure) for each pixel in the image. The cast prediction operators compute a color cast from the log-scene exposure images using knowledge derived from the image data and the known exposure balance (color balance or cast) of a neutral object in the scene — which was provided as part of the coarse profile data accompanying the image. The cast predictions from each of the operators, now represented by a log exposure triplet ( e.g., R, G, B exposure) are stored in a memory.

In another aspect of the invention, the operator selects a cast prediction for preview.

In yet another aspect of the invention, the image data is processed through the color processing system the operator normally uses to transform the input image data for display on the monitor, with modifications such that the selected color cast correction is applied. For example, if the image source is an input scanner having the property that scan data is proportional to log-scene exposure for each of the color channels, cast correction may be introduced by adding an offset to each channel before sending the data to the color processor to compute an image for rendering on the display.

In the usual operating mode, an operator repeats the color transform and preview operation for each of the cast predictions, and then selects the cast prediction having the preferred appearance. Additional cast correction could be performed in exceptional cases using conventional means, beginning with the closest prediction from the above system.

DESCRIPTION OF THE DRAWINGS

The novel features that are considered characteristic of the invention are set forth with particularity in the appended claims. The invention, itself, however, both as to its organization and method of operation, together with other objects and advantages thereof, will best be understood from the following detailed description of the embodiments when read in connection with the accompanying drawings wherein the same numerals have been used throughout the various figures to denote the same features and wherein:

Fig. 1 is a diagrammatic perspective view of an electronic imaging system in which the present invention is embodied;

Fig. 2 shows a flow diagram for an embodiment of the invention in the form of a transparency scanner and associated components which implement the cast removal methodology of the invention and demonstrates how preview and cast- removal modules of the invention interact with other system components;

Fig. 3 is a flow diagram for the cast removal module of Fig. 2, showing the interaction of the cast computation, preview, and cast selection mechanisms;

Fig. 4 is a flow diagram of the color cast prediction module of the invention; and

Fig. 5 shows a graphical user interface by which the image preview and cast selection mechanism can be operated as part of an image-editing workstation.

DETAILED DESCRIPTION

The invention in a preferred embodiment is described in connection with a color negative and transparency scanner As will be seen hereinafter, the scanner consists of a transport light source, color CCD linear array detector, a transport mechanism, a preview memory, and a color transform mechanism. Output from the scanner is digital data intended for display on a color monitor with output characteristics similar to the Apple 13 inch color monitor, manufactured by Apple Computer of Cupertino, California.

This system is designed to provide high quality digital color images from a wide variety of photographic negative and transparency materials. Further, it is designed to be used by relatively unskilled operators who may or may not have significant knowledge of color reproduction systems, and need rapid scanning of their images. The major application for the scanner is in the desktop publishing application where the scanner is attached to a personal computer used for image editing and display. As such, the scanner is controlled by and provides output data to host application software executing on a PC (e.g. Adobe Photoshop by Adobe Systems, Incorporated).

Reference is now made to Fig. 1 which shows an image-editing workstation 20 embodying a preferred form of the invention. Components of the workstation 20 comprise a computer 22 used to perform operations on image data and present a user with a graphical interface in the form of dialogs on a color monitor 24, which is preferably calibrated. Color monitor 24 is also used for image preview and display operations. A pointing device 28 and keyboard 26 facilitate operator interaction

with other components of workstation 20. A disk drive 36 can be used for input or output of image files to or from a storage disk 34 or the like. Input signals can also originate from a digital still or motion video camera 30, but preferably originate from a transparency scanner 32.

In general, the work station 20 allows an operator to place the scan media in the scanner, specify the material to be scanned (e.g., color negative)along with the scan resolution and output size. A low resolution preview scan is performed and displayed to the user after having been color processed for viewing on the calibrated monitor. The user can then perform adjustments on the scan parameters ( e.g., sharpening level, color cast removal, brightness, cropping,...) while viewing an interactive image preview showing the results of the adjustments, then initiate a final scan. A high resolution scan is then performed and the data from the CCD array is immediately processed through the color processor and sent to the host application.

Figure 2 is a block diagram showing major components of the invention as it resides in workstation 20 and is generally designated as a system 50. As seen, system 50 comprises photographic negative image sampling module 52. Module 52 includes a light source 54 and a sensor 58 in the form of a CCD array. The material being scanned, in this case a color negative, is designated at 56. Output signals are generated from CCD array 58 as the negative 56 is scanned. The output signals consist of three analog signals, one each for three primaries: red, green, and blue. The output signals are approximately linear with respect to optical transmittance of the negative being scanned, in each color channel. The output of the sampling hardware is taken as input to the digitization block 60, which consists of an adjustable analog gain stage 62 and an analog-to-digital converter 64. The analog gain stage 62 allows adjustment of the signal input to the analog-to-digital converter 64, which converts the analog signals to quantized digital signals having discrete levels. The output of the digitization stage can be directed to two modules, depending on the operational mode of the scanner. During preview mode, the output is directed to the input of a preview and cast correction module 72. Alternately, when in final scan mode, the output is directed to the input of a color processor 74.

A control line 90 and mode switch 70 direct the signal from the analog-to-digital converter 64 toward the cast preview and removal module 72 or color processor 74 based on the current selected scan mode. When scanner 32 is in operation, the outputs of the cast preview and removal module 84, 86, and 88, respectively, are directed to the light source 54, analog gain stage 62 and color processor 74, respectively. In all three cases, the signals are gain or offset signals appropriate for adjusting the corresponding subsystems to achieve the desired cast removal. When in final scan mode, data from the analog-to-digital converter 64 are processed through the color processor 74 using input from the cast preview and removal module 72 . The resultant image data can then be sent to a number of outputs 76 including host application 78 (e.g., Adobe Photoshop) running on the host computer 22, the color display monitor 80, or a storage file 82.

Fig. 3 is a block diagram showing in more detail cast preview and removal module 72. Input to module 72 is stored in digital memory 74. The contents of memory 74 can be read as input to three modules, depending on the mode of operation of the cast preview and removal module 72. In the first mode of operation, the cast predictions are determined by the cast prediction module 76, operating on the data in the preview memory buffer 74 and stored in a memory buffer 78 for later use. In a second and optional mode, an exposure shift is computed by use of a scene analysis module 92 operating on the contents of the preview buffer 74. The operation consists of creating a luminance signal from a color signal 94 and running the scene analysis processor 96. The output of processor 96 is a signal which is applied equally to all three color channels, and will not introduce or remove a color cast.

Different cast prediction methodologies are carried out in block 76 and stored in, respectively, blocks 80 or 82 or 84 or 86 in memory 78. The results of the cast prediction methodologies are selectable by a selection mechanism 90 via operator intervention and the results are combined with the signal from the scene analysis module 92 at a junction 100. The combined signal is sent to a color processor 102

and/or to a gain partitioning module 106. The gain partitioning module 106, in turn, sends a signal to the light source 54, gain stage 62 and/or color processor 74.

In the third operational mode, the contents of the preview buffer 74 is processed through the color processor 102 with input from junction 100 and the resulting image displayed on color monitor 104 for cast judgment by the operator.

Cast prediction module 76 will now be described with reference to Fig. 4. The contents of preview buffer 74 are processed through a color processor 200 to form a representation of scene exposure, which is stored in a memory buffer 202. Cast operators 206, 208, 210 and 212 take the contents of this buffer as input, and each output a cast signal which is stored in a corresponding memory buffer location 80,82,84,86, respectively, in the cast memory 78.

Fig. 5 shows a graphical user interface with which a user interacts in practicing the invention. An operator interaction dialog 120 is used in the preferred embodiment. It consists of a dialog box presented to the user on the color monitor 24 by computer 22. Region 122 is a color image display into which is mapped the contents of the preview buffer 74 after processing by color processor 102 operating in the display mode of the cast preview and removal module 72. Below the preview is a cast selection mechanism 124 consisting of four buttons, 126,128,130,132, each corresponding to a cast memory location (80,82,84,86) in the cast buffer 78. Below the selection mechanism is a button for initiating a final scan 140, which sends input to the mode switch 70. Region tool 142 allows the operator to select a region in the memory buffer 74 on which to perform the cast computations corresponding to the enclosed region in the preview image 122 using the keyboard 26 or pointing device 28.

The various color cast prediction methodologies and their use will now be described. It will be seen that these comprise no cast prediction, white point, gray world, and bright pastels. In general, the sequence of events in carrying out the invention involve: (1) Color Cast Prediction with a low resolution scan, (2) Cast Selection, (3) Image Preview, and (4) Final Scan, which will now be described.

I. Color Cast Prediction

A) Low resolution scan

Referring to Figs. 1 and 2, scanner 32 is first configured for preview scanning. The gain of the analog gain stage 62 (a linear amplifier) is set such that the minimum density (film base density) for a negative of the selected film type produces an input signal to the analog to digital converter 64 which is mapped to digit 1024 (out of 1024). Data for this adjustment is contained in a device profile for the scanner/negative combination selected by the user before initiating the scan. As CCD 58 output voltage is nearly linear with respect to incident light, the digital counts from A/D converter 64 will correspond to negative optical transmittances. The low resolution preview scan is made, and a resulting RGB 10-bit image stored in the preview buffer 74 shown in Figure 3.

B) Low resolution image data conversion

Using the characterization for the scanner/negative pair stored in the device profile, in this case a set of one dimensional LUTs relating scene log exposure to 10- bit digit for each channel: red, green, blue, the image data are converted from digital count to scene log exposure and stored in a computer memory buffer.

C) Cast prediction operators

The following operations are performed on the log exposure data to predict candidate color casts.

1) None

The first method is to perform no computation and report a color cast of zero red, zero green and zero blue exposure.

2) White Point

The white point cast prediction method assumes that the brightest object in the scene is a specular reflection of the illumination source off a neutral object. The operation is performed in two parts — isolation of the white, and determination of cast. a) Isolating the white

For every pixel location in the low-res image, compute the minimum of the three exposure values at that location minRGB = Miτήmum(red, green,blue) and compare the value of minRGB to the highest of all previous minRGB. maxMinRGB holds this value. Update it if necessary.

lϊ(minRGB > maxMinRGB)ihen maxMinRGB = minRGB b) Recording the cast

If the brightness at the location is higher than all previous pixel locations (minRGB > maxMinRGB) , save the exposure data for that pixel. whitePt_red = red; whitePt_green = green; whitePtjblue = blue; Repeat for every pixel. 3) Gray World

The gray world cast prediction method is based on the approximation that for a large number of scenes, the scene exposure in the three channels red, green, and blue when summed over the image area, will be equal. The operation is to simply determine the mean of all pixels in each of the three channels. That is:

Mean_Green = -^^ _J G _ιj

Mean Blue = — j— Σ" , ∑" 4, (nx* ny)^' ^=λ ^J=' ^IJ where Rij, Gij, Bij are the red, green, and blue exposures of the ij'th pixel, nx is the height of the selected region, and ny is the width, specified in low resolution pixels.

4) Bright Pastels

The bright pastel cast prediction method is based on the observation that a white point determined from a specular highlight (as in the White Point method above) will be incorrect due to the object reflectance not being neutral. For example,

a red Christmas-tree ball might be the brightest point found in the image, but certainly is not neutral and will cause a poor cast estimate. The bright pastel method operates by using the white point determined above, and averages together all pixels with exposure within a constant factor PASTELRANGE (e.g. 4) of the white having inter-channel color difference less than another factor (e.g. 0.15 logE) a) determine the exposure of the brightest pixel using the White Point method described above, then determine the minimum brightness threshold. minWhitePt = Mm avim(whitePt_red,whitePt_green,whitePt_bIue); minbright = minWhitePt * 1 /PASTELRANGE; b) reprocess the preview image, summing R, G, and B exposures separately for those pixels which are have brightness between the brightness threshold and the brightness of the white reference found in the image, and which have a pastel factor (Minimum(G-R,G-B) ) less than the pastellization threshold, and counting the number of pixels that meet this criteria. After the entire low-resolution image has been processed, calculate the mean R, G, and B exposures which were included in the sums.

Once the cast exposures have been determined for each of the above methods, they are converted to balance offsets. rBalance = ( GreenExp- RedExp) bBalance = ( GreenExp- BlueExp) and further modified to preserve luminance exposure level avg = ( rBalance + bBalance ) / 3; dExpRed = rBalance - avg; dExpGreen = -avg; dExpBlue = bBalance - avg; The exposure changes to be applied to implement the cast removal are then: newExpRed = OriginalExpRed + dExpRed newExpGreen = OriginalExpGreen + dExpGreen newExpBlue = OriginalExpBlue + dExpBlue

II. Cast Selection Mechanism

Once the cast prediction operation has concluded, and a set of candidate casts has been computed, the operator must select the preferred cast removal from the set of four ( three complex operations, and a null operation). A preferred method for facilitating this selection is to generate an image preview from each of the four candidates — which represents the final cast the operator will observe on the color monitor when the selected methods are applied to the final scan operation, regardless of how the cast-removal operation is actually implemented. The preferred embodiment is to provide the user a set of "buttons" to select a cast (method) and update the color monitor rapidly to reflect the change in the image.

III. Image Preview

The color monitor used for the preview step must be adjusted prior to use such that the color of a neutral object in the scene (no color cast) has the appearance the user is attempting to achieve by the reproduction. In the preferred embodiment, the operator is assumed to have calibrated the display monitor in such a way that the Apple 13 inch Color Monitor device profile adequately represents the color monitor. This calibration can be performed using readily available tools (e.g. Knoll Gamma tool supplied with Adobe Photoshop).

To preview the image, the contents of the preview buffer memory is processed through the color processor using the scanner/negative profile appropriate for the negative currently being scanned. The profile actually passed to the color processor has been modified to incorporate the cast change selected by the operator in the previous step (2 above). The resulting image is displayed on the color monitor for the operators judgment. This operation is repeated under the user control for all selections of cast (removal method).

IV. Final Scan

Having selected the appropriate cast removal, the operator initiates a final scan. The scanner is reconfigured to pass the output(s) of the A/D converter(s) directly to the color processor unit which has been configured as above to account

for the selected cast change. If the cast change is large, the change may be partitioned between the modified scanner/negative profile and the analog gain stage ahead of the A/D converted to reduce data loss. A high-resolution scan is then made, processed through the color processor, and output to the host software application.

Those skilled in the relevant arts will recognize that the invention may be implemented in yet other ways according to its teachings. For example, in conventional photography, lab negative printing equipment provides controls for color cast to a skilled operator, who manually determines and performs the cast adjustment. In many systems today, an automatic system has been implemented consisting typically of a low-resolution electronic camera (still or video) which photographs the negative, performs a calculation on the resulting data, and provides the result to the negative-printing system. Here, the present known cast correction mechanisms could be replaced by the invention.

The invention may also be implemented in consumer camcorders or the like. Consumer camcorders use a slightly different approach, as the main problem is changing illuminant color temperature. Most operate by attempting to determine a white object in the scene, and then deriving color cast from the cast of the white point. The capture circuitry is then recalibrated "on-the-fly" as photography continues. (AGC-automatic gain control, and auto white point control (corrects for difference in lighting color temperature))

Color copiers may employ variations of the techniques described above.

Image-processing software allows user adjustment of color cast — Adobe Photoshop "Variations" plug in for example. A user is given controls (R,G,B sliders for example) and can apply a specific cast change to an image.

In addition to the foregoing, other embodiments may be implemented and it is intended that such embodiments be within the scope of the invention.