BACKGROUND ART The human vision system is a very poorly understood mechanism that translates photons of various wavelengths into visual pictures that human brains can understand and respond to. The human vision system and mental system compensate for scenes under various illumination sources and provides to the viewer a"corrected"visual picture. For example, white tee shirts appear white in human vision regardless of whether the scene happened under noonday sunlight or in the last minutes of a red sunset. When digital cameras, either video motion cameras (VMC) or a digital still camera (DSC), are exposed to similar illumination environments, the resulting images are profoundly different.

Extensive research has been undertaken to predict a mathematical construct for an image called the White Point (WP). The WP is the illumination that occurred at the brightest point in the image and represents what should be considered"white"in the final image. It is assumed that every image has some white objects or highlights in it. When the brightest object, with roughly equal amounts of red, green, and blue is located, the WP operation is constructed by determining the multipliers of the red, green, and blue parts of the brightest point so that the resulting red, green, and blue values will be made equal. Once this transformation is known for the brightest point in an image, it is simultaneously applied to all the other points (which are called pixels) in the image. The WP operation typically results in a final image that looks much more realistic with respect to its color balance.

There is a significant shortcoming of the simplistic WP operation described above. It is the assumption that there are some spectrally"white"objects in the image. While this is true the majority of the time for typical"tourist"pictures, there are also numerous cases where a spectrally"white"object is not present. For example, a close-up picture of a red barn

with some blue and green metal signs attached to the barn's side. The dominant color would be red. Some digital camera systems might interpret the large amount of red as a color cast problem that typically occurs under sunset illumination conditions. The brightest part of the image would be the green signs. If the digital camera algorithm attempted to use the green area as the WP, then the resulting picture would be made very blue. The underlying problem is caused by not knowing the true nature of the illumination (light source) present at the time the image was captured.

Current technology tries to use the color content of the image to estimate the color illumination. In video motion cameras (VMCs), there has been remarkable success with this method since the videographer typically"pans"a scene to cover a large area. In this process, there is almost always some bright white object that can be identified in the multiple images.

Once this"brightest"object is imaged, the WP algorithm locks in on this area and makes a prudent estimate of the white point illumination and keeps this WP value until a"brighter" white object is discovered. This is not true for digital still cameras (DSCs) where typically only a single image is capture for a given scene.

A great deal of research is being conducted to see if the WP of an image can be deduced from just the image itself. However, examples like the barn picture described above will always cause problems. An alternative solution is to measure the scene's illumination source directly. In black and white photography, the measurement was performed with a "light meter". The meter is pointed at the light source, which would be straight up for daylight or towards a spotlight if it were focused on the object of interest. In color photography, a more sophisticated type of"light meter"called a photo spectroradiometer is used. Rather than measuring a single quantity like the black and white light meter, a photo spectroradiometer has to measure numerous points across the visual light spectrum and make a graph of the power at each wavelength that it has found. Once this graph is known, then an accurate representation of the original image can be constructed by removing the influence of the light source from the original scene. For example, an image of a white tee shirt at sunset will have a definite red cast to it. The photo spectroradiometer graph will show strong photon power in the red region of the visible spectrum. Knowing how much influence the illumination source had on the resulting image, a mathematical process is performed to remove the dominant red from the image. The final image has the white tee shirt looking truly white. In the other example of the red barn with the blue and green signs, the photo spectroradiometer graph would show normal daylight present as the illuminant. This means

that almost no color correction would be applied to the final image. So in this case the dominant red barn color would be left in the image since that is the normal color that human vision would have seen under midday circumstances. The photo spectroradiometer is the ideal instrument for taking color pictures.

The problem is that a spectroradiometer is both big and expensive. A typical unit is 10 by 6 by 4 inches in size and costs between $5000 to $50,000 in 1998. It also requires a computer to readout its graphical data and apply it to the image in question. What has long been needed is a low-cost, small, portable spectroradiometer to indicate the type of illumination present while a picture is being captured.

The high cost of spectroradiometers comes from the narrow bandwidth samples (typically ten nanometers in width) that they provide. Narrow bandwidth measurements are essential for scientific calculations, but while working on the present invention it has been determined that they are not required for illumination discrimination.

DISCLOSURE OF THE INVENTION It is an object of the present invention to provide, at reasonable cost, an optical sensor system capable of distinguishing between different illuminants, a method for the design thereof and a camera using the optical sensor.

This object of the present invention is solved by an optical sensor system as claimed in claim 1, a method for the design of an optical sensor system as claimed in claim 11 and a camera using the optical sensor as claimed in claim 17. Advantageous embodiments of the invention are claimed in the sub-claims.

In a preferred embodiment, the present invention provides an optical sensor system which uses portions of the intensity spectrums of various types of natural and artificial light and combinations thereof to determine the nature of the illuminants. The intensity spectrums are sensed by a plurality of photosensors connecting to a processing system which can discriminate characteristic areas therein. The present invention measures relative mixtures of sunlight, and artificial light, such as tungsten, fluorescent, and xenon photoflash.

In a preferred embodiment, the present invention further provides a plurality of photosensors having a plurality of bandpass filters which allow discrimination of the sources of illumination, such as natural light, artificial light, and a combination thereof.

In a preferred embodiment, the present invention further provides a plurality of photosensors having a plurality of bandpass filters selected to optimize the discrimination ability for various types of illuminants.

In a preferred embodiment, the present invention further provides a minimum of five bandpass filters measurements of the spectral power that can resolve daylight, incandescent (tungsten), and florescent light sources.

In a preferred embodiment, the present invention further provides a plurality of photosensors having a diffraction grating which allows discrimination of the sources of illumination, such as natural light, artificial light, and a combination thereof.

In a preferred embodiment, the present invention further provides a plurality of photosensors having a diffraction grating selected to optimize the discrimination ability for various types of illuminants.

In a preferred embodiment, the present invention further provides a computer program for selecting bandpass filters to optimize the discrimination ability thereof.

In a preferred embodiment, the present invention further provides a computer program for selecting dye-based bandpass filters with maximum discrimination ability for minimal cost.

In a preferred embodiment, the present invention provides a camera having an optical sensor which can respond to natural, artificial light, and a combination thereof to allow compensation for color differences in an image taken by the camera caused by the illumination.

In a preferred embodiment, the present invention further provides a camera having an optical sensor system which uses a diffraction grating and a plurality of photodiodes to determine the proportions of sunlight and artificial light to allow compensation for color differences in an image taken by the camera caused by the illumination.

In a preferred embodiment, the present invention further provides a camera having an optical sensor system in which a plurality of bandpass filters and photodiodes are used to determine the proportions of sunlight and artificial light to allow compensation for color differences in an image taken by the camera caused by the illumination.

In a preferred embodiment, the present invention further provides a digital still camera (DSC) recording an image on a matrix of photosensitive elements which are used for determining the illumination which allows for compensation of color differences caused by the illumination.

In a preferred embodiment, the present invention further provides a camera having an optical sensor system which can be used to record illumination data on film to be used during the development process thereof.

The above and additional advantages of the present invention will become apparent to those skilled in the art from a reading of the following detailed description when taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 (PRIOR ART) is an intensity spectrum of natural light; FIG. 2 (PRIOR ART) is an intensity spectrum of typical of incandescent light; FIG. 3 (PRIOR ART) is an intensity spectrum of one type of fluorescent light; FIG. 4 (PRIOR ART) is an intensity spectrum of another type of fluorescent light; FIG. 5 (PRIOR ART) is an intensity spectrum of xenon light; FIG. 6 (PRIOR ART) are the intensity spectrums of FIGs. 1 to 5 combined; FIG. 7 is a composite light spectrum according to the present invention; FIG. 8 is an intensity spectrum of four bandpass filters; FIG. 9 is a bandpass filter selection program in accordance with the present invention; FIG. 10 is an intensity spectrum of five bandpass filters selected by the bandpass filter selection program of FIG. 9; FIG. 11 is an optical sensor system of the present invention resulting from the bandpass filter selection program of FIG. 9; FIG. 12 is an alternate optical sensor system according to the present invention resulting from a modified version of the bandpass filter selection program of FIG. 9; FIG. 13 is a cross sectional view of a single lens reflex camera incorporating the present invention; and FIG. 14 is a piece of film used with the camera of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Referring now to FIG. 1 (PRIOR ART), therein is shown an intensity spectrum of natural light. The intensity spectrum plots the intensity on the ordinate axis and the wavelength on the abscissa. The visible portion of the spectrum shown is generally from 400 to 700 nanometers. The curve 10 is typical of mid-day sunlight. During the course of the day,

the general configuration of the curve representing natural light will remain the same although the intensity of different wavelengths will change based on the time of day. For example, the sunlight will be redder at the beginning and end of the day such that the intensity at the red, 700 nanometer end of the spectrum will be higher at those times.

Similarly, for a cloud-covered day with indirect sunlight, the entire intensity spectrum would be reduced.

Referring now to FIG. 2 (PRIOR ART), therein is shown an intensity spectrum for incandescent light, such as tungsten filament or halogen. The curve 20 represents light from a tungsten filament. The curve will remain generally the same except the intensity levels will be different based on the current feeding the tungsten filament.

Referring now to FIG. 3 (PRIOR ART), therein is shown an intensity spectrum for a gaseous-discharge light, such as fluorescent, mercury vapor, metal halide, sodium, or neon.

The curve 30 shows a standard warm white fluorescent light spectrum having spikes 32,34, 36, and 38 which are characteristic of the electron excitation levels of the phosphors used in fluorescent lights. It should be noted as well know to those skilled in the art that the spikes shown have been truncated to show relative intensities since the spikes are narrow and would be off scale for the intensity spectrum shown.

Referring now to FIG. 4 (PRIOR ART), therein is shown the intensity spectrum of another fluorescent light. The curve 40 is for a standard cool white fluorescent light having the characteristic spikes at 40,44,46, and 48. It should be noted that while the general curves 40 and 30 are similar, the intensities of the spikes are different, and this allows for distinguishing between these two fluorescents. It should be understood that there are a number of different types of fluorescent lights, but they all have different individual intensity spectrum characteristics.

Referring now to FIG. 5 (PRIOR ART), therein is shown the intensity spectrum of xenon light. The curve 50 represents the typical spectrum of xenon light used in photographic flash equipment.

Referring now to FIG. 6 (PRIOR ART), therein are shown an overlay of curves 10, 20,30,40, and 50 from FIG. 1 (PRIOR ART) through FIG. 5 (PRIOR ART). It should be noted that there are characteristic areas 10', 20', 30', 40', and 50'where each of the light sources is different from that of all the others and therefore it would be possible to discriminate between them when the spectral intensity curves are known. Some light sources

such as natural light can be distinguished between time of day and cloud cover by having two or more characteristic areas such as 10'and 10".

Referring now to FIG. 7, therein is shown a composite intensity spectrum with a curve 70 along with characteristic areas 10', 20', 30', 40', 50', and 10". This curve 70 is the outline composite of FIG. 6 (PRIOR ART), and it can be used in the bandpass filter selection program of the present invention as will later be described.

Referring now to FIG. 8, therein is shown the spectral intensity characteristics of a series of manually selected bandpass filters 82,84,86, and 88. The percent transmittance is shown on the ordinate axis. These filters were selected to see if it was possible to cover the entire spectrum from 400 to 700 nanometers while having different bandpasses and transmittance which would allow photosensors to detect specific characteristic bandwidths and intensities in illumination light.

Referring now to FIG. 9, therein is shown the bandpass filter selection program 90 of the present invention. The program 90 starts at block 91 with input 92 where the filter tables for all known filters and new filters are input. The advance of commercial optical band pass filters materials continues, and so new and improved filters are available on an ongoing basis.

The computer program 90 of the present invention provides a method of choosing the best bandpass filters currently available and minimizing the number of photosensors needed to produce a maximum discrimination of the various light source illuminants.

The program starts at a block 91. At a block 92, the bandpass filter tables are input. At a block 93, standard illuminant spectra are input as tables. In a block 94, all the combinations of illuminants with combinations of filter sets are iterated. In a block 95 connected to the output of the block 94, the program integrates the power of the filters through the illuminants for the combinations of filter sets. And, in a block 96, the power results are sorted to find the maximum discrimination for the most number of illuminants.

Referring now to FIG. 10, therein are shown five bandpass filters 100,102,104,106, and 108, selected using the program 10, which would respectively permit the identification of illuminants having the distinctive characteristics areas 50', 40', 10', 30', and 20'respectively.

Referring now to FIG. 11, therein is shown a bandpass filter optical sensor system 100. The optical sensor system 100 has a plurality of photosensors 110 consisting of photosensors 110A through E which are mounted on a frame 112. The photosensors 110 can be discrete photodiodes, photosensitive semiconductors, or charge-coupled devices which produce or affect the passage of current proportionally to the light energy to which it is

exposed. On top of the frame 112 is a clear, optically transparent resin 114 which has been flattened to allow the deposition of various dye-based bandpass filters 116 consisting of bandpass filters 116A through 116E.

The bandpass filters 116 are a selection structure for selecting the desired portions of the intensity spectrum of illuminants for intensity determinations by the photosensors 110. In the preferred embodiment, the bandpass filters 116 are dyes which are in the form of an ink, paint or gel which can be printed, painted, or silk-screened on the resin 114. The dye material can further be placed in several layers for increased optical density. The different bandpass filters 116 allow the combination to be able to discriminate between different portions of the spectrum of light falling on the bandpass filters 116. With the proper selection of bandpass filters, it is possible to distinguish the particular portions of the spectrum which contain particular intensities which are characteristic of various types of natural and artificial light.

The particular bandpass filter dyes selected and the number of photodiodes in the plurality of photodiodes is determined so as to integrate the signals that are derived from the plurality of photodiodes 110 with their respective bandpass filters 116 in response to various mixtures of illumination. In the present invention five photodiodes 110 A through E are shown with their accompanying bandpass filters 116 A through E.

The outputs from the plurality of photosensors or photodiodes 110 are connected by a plurality of leads 118 to a multiplexer or sample-and-hold circuitry 120. The analog signals from the sample-and-hold circuitry 120 are provided to an analog-to-digital converter 122 which provides digital signals to an application specific integrated circuit (ASIC) 124. The ASIC 124 is a specialized circuit for distinguishing from the signals which illuminants are predominant. However, it could be a modified version of a conventional integrated circuit used as a part of the camera 210 or an independent ASIC or a simple microprocessor or other processing engine.

The ASIC 124 would compare the signals from the plurality of photodiodes 110 and provide information as to how the picture taken by the camera 10 should be changed to compensate for differences caused by various mixtures of illuminants. Further, by comparing the strengths of the output signals, a determination can be made of the relative strengths or percentages of the various illuminants. In the preferred embodiment, only the relative strengths of two or possibly three illuminants is necessary to compensate a picture.

Referring now to FIG. 12, therein is shown a portion of a diffraction grating optical sensor system 150. The optical sensor system 150 includes photosensors 110A through E and

the leads 118A through E which are similar to the bandpass optical sensor system 100 components. The number and spacing of the photosensors 110 A through E would be determined using the computer program 10.

Spaced apart from the photosensors 110 is a diffraction grating 152. The diffraction grating is a selection structure for selecting the desired portions of the intensity spectrum of illuminants for intensity determinations by the photosensors 110. It takes incoming light 154 and diffracts it into its spectrum as indicated by arrows 156 and 158. This provides the intensity spectrum of FIG. 7 onto the photosensors 11 OA through E. The ASIC 124 for the diffraction grating optical sensor system 150 would then be set up to perform the discrimination function.

In operation, the present invention uses the computer program 10 to integrate the signals that are derived from the photosensors 110 with their respective bandpass filters over them in response to various mixtures of illuminants. By comparing the signal from the photosensors with others, which sample different regions of the visible spectrum, a determination of the relative strength of the various mixtures of illuminants is made.

During the initial investigations, it was determined that it would be possible to select four bandpass dye-based filters such as 82,84,86, and 88 which would cover the entire visible spectrum from 400 to 700 nanometers and have the ability to be specifically tailored to detect specific characteristics of the light spectra of various illuminants. For example, the bandpass 84 in FIG 8 would be particularly adept at identifying the spike 40'in Fig. 6 which is characteristic of the standard cool white florescent shown in FIG. 4 (PRIOR ART).

Since new optical bandpass materials continue to be developed on an ongoing basis, it is not feasible to manually test different materials and try to judge their relative cost. Further, there is a four-way trade-off between the discrimination capability of the bandpass filter material, its cost and the number of photosensors 110 required, and the combined number of illuminants which will affect the white point. The goal of the computer program 10 is to position all classes of commercially available bandpass filters on typical mixed illuminant spectra to find the combination of filters which gives the maximum signal of parts of the spectra that are characteristic of a particulate type of illuminant at minimal cost.

It has generally been determined that at least three, and optimally between five and seven photosensors 110 are required within an equivalent number of bandpass filters.

Similarly, the diffraction grating 152 will diffract incoming illumination into its spectral components with the higher intensities at the characteristic illumination spectrum

regions for the particular illuminant. Therefore, the diffraction grating 152 can be used in place of the bandpass filters 116. However, the diffraction grating optical sensor system 150 is more susceptible to placement problems ranging from the distance of the diffraction grating from the photosensors 110 and of the spacing of the photosensors 110 between each other since movement in either would displace the position of the spctra falling thereon.

Further, because of the spacing requirement, such a diffraction grating optical sensor system 150 would be larger than the equivalent band pass optical sensor system 100.

The computer program 10 is capable of determining an ideal number of bandpass filters for detecting various types of natural and artificial light illuminants using the composite light curve 70 shown in FIG. 7. The computer program 10 would select a series of bandpass filters such as bandpass filters 100,102,104,106, and 108 as shown in FIG. 10.

This would allow discrimination of the illuminant characteristic areas 50', 40', 10', 30', 10", and 20'as shown in FIG. 10 by the super position of the curve 70. Generally, it would be understood that most situations where lighting is critical would generally would be controlled by only one to three illuminates and therefore, the capability of detecting all illuminants all of the time is not necessary. It would also be understood that, depending on the circumstances, illuminants having intensities below a certain level need not be considered.

After selecting the number of bandpass filters and photosensors 110 necessary or desirable for a particular application, the optical sensor system 100 shown in FIG. 11 would be built.

It would be evident to those skilled in the art that the lowest cost photosensors 110 can be obtained when the bandpass filters can be silk-screened or printed directly on the silicon substrate in which the photosensors 110 have been manufactured by conventional semiconductor manufacturing technology. This approach would eliminate the transparent resin 114 and the frame 112, and allow the optical sensor system 100 to be only microns in size. It would also allow it to be integrated into devices which already have photosensors.

As previously explained, various intensity levels of the light impacting on the photosensors 110 A through E would generate or allow the passage of current to the sample and hold 120 where the signals would be fed to the A/D converter 122 for use by the ASIC 124 which would identify the individual illuminants and their percentages due to the spectrums and the intensities which are a function of the signals on leads 118 A through E.

In the alternate embodiment of FIG. 12, the diffraction grating optical sensor system 158 consists of the same or similar photosensor and electronic system as shown in FIG. 11.

The way that the intensity spectrum is generated is different. The diffraction grating 152 is used to take the mixed illuminant light 154 and divide it into its spectral components 156 through 158 which the diffraction grating optical sensor system 150 is capable of analyzing in the same way as for the bandpass filter optical sensor system 100 of FIG. 11.

Referring now to FIG. 13, therein is shown a single lens reflex camera 210 having a camera body 212 and an attached lens system 214. A single lens reflex camera is shown as an example where the present invention fits in well, but it would be evident that the present invention would work for all cameras. Further, the present invention could be used with different recording media such as film and magnetic recording as will later be explained.

The camera 210 has a mirror 216 which pivots on a pivot 218 to initially direct light both from an image as well as ambient light through the lens system 214 up to a prism 220 and out through an eyepiece 222.

When a picture is being taken, the mirror 216 is pivoted up so as to allow light to strike a recording medium 224. For a film camera, the recording medium is photographic film, which is designated as film recording medium 224. In a digital still camera (DSC), the recording medium 224 initially is a matrix of photosensors, such as photosensitive semiconductors, photodiodes, or charge-coupled devices (CCD). The image recorded initially by these photosensors is then electronically or magnetically recorded digitally for later playback. This medium is designated as digital recording medium 224.

The camera 210 contains a plurality of photosensors or photodiodes 110 as shown in FIG. 11 and described in connection with FIG. 11, directed at the sweet spot of the image coming through the lens system 214, which can provide illumination information through a recording mechanism 230 to the recording medium 224. The photodiodes 110 are a category of photosensors which either produce or allow the passage of current in response to light energy being applied to them. The photodiodes 110 could also be a small subset of the photosensors in the digital recording medium 224.

The camera body 212 further has an exterior photodiode 232 on top of the camera 210 pointed generally away from the image which will be in the picture. To provide even better discrimination of the illumination, the exterior photodiode 232 may be pivotable by the user to point at the illumination which provides the highest light energy.

In an alternate embodiment, a diffraction grating 234 is provided at the end of the mirror 216 to diffract entering light into its corresponding spectrum for the plurality of

photodiodes 228 like the photodiodes 11 OA to 110E. As will later be described, two types of optical sensor systems are within the scope of the present invention.

Referring now to FIG. 14, therein is shown a portion of the film recording medium 224 having sprockets 244 and a black ring 240 around the picture 242 onto which the recording mechanism 230 can provide data regarding the illumination to which the film is exposed. This illuminant information can be recorded in an information area 246 in the black ring 240 and be used by the developer in the film laboratory to correct ambient illumination induced color shifts in the image by determining the white point, as previously explained.

In operation, light from the image would enter the camera 210 through the lens system 214 and be reflected by the mirror 216 which would be in the down position. The light would be reflected upwards into the prism 220 and be reflected through the prism 220 to exit out the eyepiece 222 where the user could see the exact image seen by the lens system 214.

In the preferred embodiment, the light from the image would be sensed by the photodiodes 110 which is integrated in the camera 210. The photodiodes 110 would be positioned to receive light from the center of the image area. The spectrum of light from the image would fall on the top of the bandpass filters 116A through 116E.

The various spectrum segments passed by the bandpass filters 116A through 116E will provide different intensities of light at different regions of the spectrum on the photodiodes 110A through 110E depending upon the light illuminating the image. With proper selection of the bandpass filters 116, light can be identified as coming from natural and artificial sources, such as the following without being limiting early morning sun, mid- morning sun, late afternoon sun, cloud-covered sun, tungsten incandescent, halogen incandescent, standard warm white fluorescent, white fluorescent, standard cool white fluorescent, daylight fluorescent, neon, xenon flash, combinations of the above.

The outputs from the photodiodes 110A through E are provided to the sample-and- hold 120 which sequentially provides the signals to the analog-to-digital converter 122. The analog-to-digital converter 122 provides the digital signals to the ASIC 124 of the camera 210.

The ASIC 124 then provides the information as to the type of illuminants to the recording mechanism 230 which provides the information to the recording media 224. For a DSC camera, the information would be recorded in the digital recording medium 224 and for a film camera it would actually be recorded in the black ring around the picture 242 in the information area 246 of the film recording medium 224.

In an alternate embodiment, the optical sensor system 150 as shown in FIG. 12 has the diffraction grating 152 which breaks up the image and illuminant light, designated as the light 154, into its spectral components 156 and 158 which is spread across the plurality of photodiodes 110. The outputs of the photodiodes 110 then act in the same form as previously described for the optical sensor system 100. While the diffraction grating could be placed on a transparent resin, generally the diffraction grating 152 must be spaced away from the photodiodes 110, further than the bandpass filters must, in order to cover the spectrum from 400 to 700 nanometers.

Depending on the colorimetric properties of the camera 210 and its intended use, it may be necessary to measure the illuminant in a direction other than through the camera's lens from the image. In this situation, it may be necessary to use another non-integrated discrimination sensor, such as the photodiode 232, to provide another input to the ASIC 124.

The photodiode 232 would be directed away from the image area, and generally upward to assist in determining the white point. The weighting to the various optical sensor systems would be heuristically determined.

While the invention has been described in conjunction with a specific best mode, it is to be understood that many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the aforegoing description. Accordingly, it is intended to embrace all such alternatives, modifications, and variations which fall within the spirit and scope of the included claims. All matters set forth herein or shown in the accompanying drawings are to be interpreted in an illustrative and non-limiting sense.