Background of the invention

There has been a growth in the use of hyperspecti'al cameras for aerial imaging, environmental monitoring, forensic science, forestry, agriculture as well as in military and industrial applications. Hyperspectral cameras normally also cover a wavelength range outside the visible thus making the design of the optical system very challenging. There are different principles of hyperspectral imagers, but in this patent application a "hyperspectral camera" is defined as a hyperspectral imager of the "push broom" type.

One of the most important limiting factors in a

hyperspectral imaging system is spatial misregistration as function of wavelength. In hyperspectral data, each spatial pixel is supposed to contain the signals for different spectral bands captured from the same spatial area. This type of misregistration occurs when either the position of a depicted area (corresponding to one pixel) changes as function of wavelength {this error is commonly referred to as "keystone"}, or the borders or objects in the depicted area are blurred differently for different spectral channels, which is caused by variation in the point spread function (PSF) as function of wavelength. In practice this means that information from different positions on the scene depicted can be

intermixed and resulting in an erroneous spectrum for spatial image pixels positioned close together.

Another effect or limiting factor is a similar spectral misregistration commonly referred to as "smile". This effect will partly shift the spectrum of a depicted poin .

This invention solves or at least improves significantly the problem with keystone and PSF variations for a first embodiment, and the problem with keystone and smile for a second embodiment. Several approaches have been tried in the past in prior art to solve the problem, the two most common approaches being hardware correction and re- sampling of the hyperspectral image data. A further description of prior art is given in the section below.

Prior Art Most push-broom spectrometers operate as follows: they project, a one-dimensional image of a very narrow scene area onto a two-dimensional pixel array. A dispersing element (a diffraction grating or a prism) disperses light in such a way that, instead of a one-dimen ional image elongated in direction, each column of the array contains spectrum of one small scene area as shown in figure 2. The two-dimensional image of the scene is then obtained by scanning. The final data form a so-called "datacube", which represents a two-dimensional image with the third dimension containing the spectral information for each spatial pixel .

Due to various optical aberrations in real -world

hyperspectral cameras, the spectral information is not captured perfectly. Spectral misregistration occurs when a given pixel on the sensor is expected to receive light from a certain spectral band but instead receives

slightly wrong wavelengths . Spatial misregistration occurs when a pixel on the sensor receives a certain spectral band but looks at slightly wrong area of the scene. Misregistration is discussed in more detail in the article "Spectral and spatial uniformity in pushbrocrrt imaging spectrometers" by Pantazis Mouroulis, 1999.

Both types of misregistration may severely distort the spectral information captured by the camera. Therefore they should be corrected to a small fraction of a pixel . In hyperspectral cameras for the visible and near- infrared region (VNIR) spectral misregistration can be corrected by oversampling the spectral data and

resampling it in postprocessing (since most of the sensors have many extra pixels in the spectral direction, such a camera will still have good spectral resolution) . However, it is normally desirable to correct the spatial misregistration in hardware as well as possible.

Spatial misregistration in hyperspectral cameras is caused by two factors:

1} . Variation in position of the PSF's centre of gravity as a function of wavelength, usually called spectral keystone. This is show in figure 4. 2 ) , Variation in size and shape of PSF as a function of the wavelength. This is shown in figure 5.

Since the positions of PSF's centre of gravity for different wavelengths must not differ by more than a small fraction of pixel size {even as small deviation as 0.1 of a pixel may introduce noticeable errors in the measured spectrum) , optical design of such systems is very challenging. Keeping the size and the shape of PSF similar for all wavelengths is perhaps even more

difficult .

In general, pixel count in hyperspectral cameras is very modest (compared to traditional imaging systems such as phot©cameras) , since it is not enough just to produce a reasonably sharp image, and optical aberrations have to be corrected at subpixel level. Optical design becomes extremely difficult if higher spatial resolution is required.

Another serious challenge is to build the designed camera. Manufacturing and centration tolerances for hardware corrected cameras (even wit relatively modest. spatial resolution of 300-600 pixels) can be very tight.

As a result of such strict requirements to the image quality, the new hyperspectral cameras more or less converged to a few designs which offer somewhat

acceptable correction of spatial misregistration.

Offner relay with a convex diffraction grating has magnification -1 and is often used in hyperspectral cameras ("Optical design of a compact imaging

spectrometer for planetary mineralogy" ^' by Pantazis

Mourouiis et ai , Optical Engineering 466, 063001 June 2007; Patent US 5,880,834}. It can be designed to have reasonably small spectral keystone. Variations in PSF's size and shape can be corrected to some extent, but there are too few optical surfaces to make this kind of correction perfect. In a real system manufacturing and centration tolerances degrade the spatial misregistration further. Even though Offner relay is not very sensitive to decentration, the tolerances can be very tight in cameras with high spatial resolution. The minimum i.e., the fastest F-number in Offner cameras is limited to approx. F2.5. Polarisation dependency of the diffraction grating may be a problem.

Dyson relay is another example of an optical system used in hyperspectral imaging. (Optical design of. a coastal ocean imaging spectrometer, by Pantazis Mouroulis et al, 9 June 2008 / Vol. 16, No, 12 / OPTICS EXPRESS 9096), It has magnification -1 and uses a concave reflective, grating as a dispersive element. There is also at least one refractive element. The F-number can be significantly lower (faster) than in Offner systems. The system can be quite small. However, extremely tight centration

requirements make it difficult to achieve low

misregistration errors even with low resolution sensors. Both the slit and the detector face the same optical surface, therefore stray light is often a problem. Also, it is a challenge even to place the detector close enough to the first optical surface (as close as it is required in the Dyson design) . Due to extremely tight tolerances and practical difficulties in placing the detector, the resolution, of Dyson cameras is not particularly high.

Design and manuf cturing of good foreoptics (i.e. the foreoptxcs which would take full advantage of low misregistration error in the following relay) for both Offner and especially Dyson relays is very challenging.

Some manufacturers base their hyperspectral cameras on various proprietary lens systems. Performance of such cameras is more or less similar to the performance of the Offner based cameras. In a push-broom hyperspectral camera rays with different wavelengths are focused on different parts of the sensor. Therefore, compared to more traditional imaging systems (such as photolenses) , sensor tilt can be introduced

(United States Patent. US 6,552,788 Bl) . This tilt can be used as an additional parameter when correcting keystone, smile, and PSF variations in the optical system. Of course, the tilt will not eliminate keystone etc.

completely - it merely offers more flexibility in

optimizing system's performance. However, in relatively complex systems, where there are many such parameters available already, introduction of this additional parameter may not lead to any significant improvements in keystone, smile, and PSF variation correction.

A known alternative to precise correction of spatial misregistration in hardware is resampling. Since the most- challenging requirements are lifted in resampling

cameras, the optical design becomes similar to

traditional imaging optics. This gives a possibility to design optics with lower (faster) F-number, high

resolution etc. The negative side is the fact that the residual spatial misregistration after resampling with approx. Ix factor is quite large. In order to bring it down to acceptable level downsam ling by factor 2x or more is usually required. Therefore the full sensor resolution is not utilised. Also, the necessity to capture two times more pixels than in case of hardware corrected systems, may slow down the framerate and processing speed.

Variable filter camera is another way to capture

hyperspectral data. The principle of operation is described in the US Patent 5,790, 188. This camera can be quite small compared to the other hyperspectral cameras. However, it may be necessary to resample the captured data, unless the scanning motion is stable with subpixel accuracy. Also, the spectral resolution becomes very limited if a reasonably low F-number is required.

If a hyperspectrai camera has to work in a very wide spectral range, it may be beneficial to split this range between two or more sensors, as described in the NASA PIDDP Final Report (May 1st 1996) «A Visible-Inf ared Imaging Spectrometer for Planetary Missions. » Example of. such an imaging spectrometer is shown on Fig.3 of this report. Since both sensors share foreoptics and slit, it is possible to get very good spatial coregistration in the along-the-track direction. However, in tire across- the -track direction {where keystone is normally measured) it will be nearly impossible to achieve good {i.e. a few percent of a pixel) coregistration . This dawback of the multiple sensor approach is explained in the Final Report of the « Concept for Future Visible and Infrared Imager* Study, by Astrium GmbH for ESA, Doc. No FI-RP--ASG-0007 (Chapter 4.5.3, Page 4-33). If the slit 2040 in the camera from Fig. 2 is replaced by a thin plate with pinholes ar anged in a two-dimentional array, it becomes possible to capture the datacube in one exposure without scanning. This approach together with possible enhancements is thoroughly described in the US Patent Applications 2006/0072109 Al and 2008/0088840 Al . While being great for some applications {such as

multispectrai video) , this approach severily limits spatial and/or spectral resolution of cameras (if

compared with push-broom, approach) . Also, cameras built on this principle are just as prone to smile and keystone errors as the previously described push-broom cameras.

Necessity to correct, spectral keystone to a very small fraction of a pixel is the principal driver restricting the spatial resolution of the existing hyper pectral cameras, possible design approaches, speed (i.e. light gathering ability) , and the final data quality.

Resampling hyperspectral cameras, while being free of the first three restricting factors, normally require downsampling of the captured data by a large factor, which reduces the resolution (i.e. pixel count) of the final data. In other words, even though it is possible to design and manufacture a resampling camera with very sharp optics and very high pixel count, the captured data have to be dovmsampled by a relatively large factor in order to match, the data quality from a hardware corrected camera. This downsampled data may still have higher pixel count than the data from a hardware corrected camera, but clearly the full resolution of the sensor will not be utilized.

To sum u this means that in state-of-the-art systems the requirement for keystone correction in the optics typically limit the maximum resolution, the light collecting capabilities of the system, not to mention the number of feasible optical design solutions.

Even when the keystone has been corrected as well as possible using all available techniques, the remaining keystone will still be significant for a vast majority of systems. In the few systems where the keystone appears to be closer to acceptable the overall resolution or pixel count is relatively low. The cause of this is that the precision of the keystone correction is .linked to the pixel count for current state-of-the-art systems.

Physically in any given system keystone is a certain fraction of the image size. When one increases the pixel count for a given image size, keystone as a fr-action of the pixel size will therefore increase.

Description of the invention

The current invention introduces new "light mixing" elements, a kind of imaginary pixels, combined with a novel approach to restore image information corresponding to these imaginary pixels based on image data from a traditional image sensor. Compared to existing

hyperspectral cameras the current invention has several advantages, namely it:

1) Significantly reduces spatial misregistration relative to the pixel size compared to the existing hardware corrected solutions.

2) Removes the necessity to correct spatial

misregistration in the imaging optics. This opens up a possibility to use other optical designs than the traditional "hyperspect al solutions" such as Dyson or Offner relays. Therefore the spatial resolution (and the pixel count) as well as light throughput of the imaging optics can be greatly increased. The manufacturing and centration tolerances at the same time will not have to be tighter than in the current systems. On the contrary, they may be more relaxed .

3} Keeps resolution and pixel count of the recorded data nearly at the same level as the sensor pixel count . Figure 1 shows the basic principle of the current

invention. Foreoptics (2020) project the scene onto the slit plane (2040) .

The. slit plane (2040) contains an optical component

(3100) which mixes the light in such a way that the spatial content of the signal has slightly lower

resolution than the spatial resolution of the sensor. An example of such component is an array of square chambers (3000) with reflective walls as shown in figures 1 and 3 a) . The number of chambers has to be at least a few percent lower than the pixel count of the sensor (2100} in the spatial direction. Each chamber (3000) contains radiance and spectral content of a small scene area which will be recorded as one pixel in the final captured data. The projection of this scene pixel onto the slit will hereafter be referred to as a "mi el."

A mixel or an array of mixels have feature sizes in the micron and tens of microns range for the most common wavelengths and pixel sizes, therefore the machining tolerances are stringent. One possible method to

manufacture the mixels or array of mixels is to use a high- spect --ratio micromachining technique using deep X~ ray lithography. One feasible option is often called the ^W LIGA" process. A publication describing such processes is "High-Aspect-Ratio Micromachining Via Deep X-Ray

Lithography", H Guckel , Proceedings of the IEEE, Vol. 96, No. 8, August 1998. For systems with more relaxed tolerance requirements laser machining or possibly other methods can. be used to manuf cture the array of mixing chambers , For some optical designs it might be beneficial to have varying mixel lengths (ref . figure 7 h) and/or a curved mixel array (ref. figure 7 b, c, f , g) .

The slit with the mixel ax-ray can be protected from contaminants, such as dust particles and gas, by

transparent windows or lenses .

These windows can be placed far enough from the slit, so that any scratches, particles, and other imperfections on the surfaces of these windows would appear very blurry in the slit plane,, sensor plane, or any other corresponding intermediate plane.

The slit enclosure formed by the windows and/or lenses can be filled with gas such as nitrogen in order to prevent oxidation of the reflective surfaces of the mixel array i.e., introducing a protective atmosphere. This enclosure can be sealed or it can be purged with the gas in question. Another alternative is to have vacuum in the enclosure.

The relay optics (2060) projects the image of the slit (3100) onto the sensor (2100) . The image of each mixel has to be (at least a few percent) larger than one sensor pixel. Also, in order to reduce noise in the final data it is beneficial to make the projection of the mixel array at least 1 pixel shorter than the length of the sensor, so that for every mixel (including the mixels at the ends of the mixel array) most of the energy is captured by the sensor. The dispersive element {2080} in the relay optics (2060) ensures that the image of the slit is dispersed approximately perpendicular to the slit (3100) .

The processing unit restores the intensity content of each mixel (for each spectral channel) based on the sensor output.

Restoring equations

After the light was captured by the. sensor, it is possible to calculate with high accuracy the total amount of light inside each mixel for each wavelength. The result of these calculations will accurately represent the spatial and spectral content of the captured scene.

The method to perform these calculations is described below. First, the general equations are presented.

Then, a simple example which illustrates the principle for the case of infinitely sharp optics is provided (the Chapter "Flat light distribution"). Finally, it is shown how to take into account the blur introduced, by the optics (the Chapter "Flat light distribution with transitions"). Again, a simple numerical example is provided.

The general equations

Let us consider the situation where we want to restore N mixels from M recorded pixels (M>N) . The keystone is then equal to ( -N) pixels. This situation is shown in Figure 8. Figure 8 shows the mixels and the corresponding recorded pixels for the general case. We can now set up the following set of equations:

whe e :

S _n is signal content of mixel #n.

S ^R _m is signal content recorded in pixel #m on the sensor. q _Kn is the fraction of the signal content in mixel #n that is recorded in pixel #tn on the sensor.

N is the total number of mixels .

is the total number of pixels recorded on the sensor.

Here S ^K ,,, is known (measured) and q _ran is known (can be calculated) for ail m and n when the keystone for the system and the "shape" of the light distribution from the mixel (as it is projected onto the sensor) is known.

Equation 1 can be written in matrix form:

Since typically only a couple of mixels contribute to each recorded pixel, most of the coefficients q are equal to zero. The matrix will then typically have the form:

s..

si (Eq. 3)

where the coefficients ¾ are nonzero only along the diagonals and zero everywhere else.

The matrix q describes important properties of. the camera and can be called the ^camera model".

The matrix system {Eq. 2 and 3} can now be solved for the unknowns S _n . Note that the system has more equations than unknowns (M>N) , In fact, each extra pixel of keystone gives one extra equation. However, for the ideal case when there is no noise in the system, the matrix system is compatible, i.e., can be solved. For a real system with noise, the system is overdetermined and an

optimisation method such as for instance the least squares method must be used to obtain the solution. The process described above can be repeated for all spectral lines. This means that even though the spectral lines have different length (i.e. different keystone} they will all be converted to the same final grid with no loss of resolution with regards to mixel count. Of course the number of mixels has to be lower than the number of sensor pixels for the method to work.

Flat light distribution

Let us look at the case when the light distribution from each mixel is flat. An example with four such mixels recorded onto five sensor pixels is shown in figure 10 and in a somewhat simplified form in figure 9, Figure 10 shows the mixels with flat light distribution recorded onto the sensor pixels .

The corresponding matrix equations are given by:

and give the correct mixel values [10 30 100 50] when being solved. This example is noise free and the system is therefore compatible, but. in real life the system will be ove determined . Flat light distribution with transitions

Before the mixels with flat light distribution is recorded onto the sensor, the signal passes through optics which ^Λ smears' the signal somewhat. What, happens when the signal from a mixel is smeared, is that part of the signal leaks into the neighboring mixels, see figure 11. If i is known how the signal is smeared (i.e., the ^Λ shape' of the transition is known) , then the original content (S _n ) of the mixel can .be restored as before.

Figure 11 shows the mixel with flat light distribution and signal content S _n (upper- figure) , and the

corresponding mixel where the signal is smeared and has leaked into the neighboring mixels (lower figure) .

Figure 12 shows an example of a third order polynomial transition used to model the smear effect. The figure (figure 12) shows an example of the details of such a transition. The transition starts at ¾χ _χ and ends at x=x ₂ - The original signal content (before smearing) of mixel #1 is equal to yi, and the original signal content of mixel #2 is equal to y ₂ (the width of the. mixel is set equal to 1) . As an example, we will show how the

restoring is done for transitions that can be described by third order polynoms (however, the principle applies to any type of transition as long as the shape of the transi ion is known) .

The equation that describes a third order polynomial transition is

where a and c are two constants that can be determined from the boundary conditions y (¾ ) (or equivalently y and y' ( x _:1 ) f or equival ently y' (x ₂ ) -0) , which gives

2(x _? - )

The transition has odd symmetry about its centre where x ₀ and y _Q are given by

Y 4~ Y

χ — i — z-

(Eq, In order to find the signal content As" ¹ of the area that lies between x=x _a and =X _b of mixel #n and tlat is recorded onto pixel #m on the sensor, one must calculate

where

Here a _n arid c _n ax ^~ e the coefficients in the equation that describes the transition between mixel #(n~i) and mixel #n.

Figure 13 shows an example where four such mixels are recorded onto five (+ two} sensor pixels. The mixels with transitions are recorded onto the sensor pixels. The transition extends a fraction 0.2 into each side of the mixel . The corresponding matrix equations are given by:

and give the correct values (original mixel content) [10 30 100 50] when solved for the unknown mixel value This example is noise free and the system is therefore compatible , but in real life the system will be

overdetermined .

Keystone and PSF variations in the foreoptics

It is important to remember that the data is restored (i.e., the keystone and PSF are corrected) only for the relay optics. The foreoptics still needs to be as

keystone free as possible with as similar PSF for

different wavelengths as possible. The keystone part can be solved by using reflective optics which is (of course) completely keystone free. Now about the PSF variations. By using relay optics with magnification 0.2 , ..0 , 5x {or - 0.2... ~ 0.5x) it is possible to achieve F- numbers as low as F1-F1.3 in the image space and at the same time have very modest F- umber in the slit plane {-F4-F6), Therefore :

- The foreoptics can be designed and made diffraction limited.

- The size of the Airy disk becomes quite small compared to the pixel size (and therefore the disk is small compared to the raixel size too) , It means that even though there is a difference in PSF size for the shorter and the longer wavelengths,, this difference is quite small compared to the mixel size and therefore it will not affect the final data as much as it happens in conventional systems.

One optimisation that can be done for the array of mixing chambers is to put refractive optics in front of the array of mixing chambers to equalise the point spread function for different wavelengths. This refractive optics' will defocus different wavelengths differently to compensate for different point spread function as

function of the wavelengths in the fore optics .

For a given f -number of the fore optics in the standard embodiment without refractive optics there is an optimal ratio between the width of the mixing chamber and its length. For this ratio the best mixing will be achieved. This ratio should be adjusted in the embodiment using refractive optics in front of the mixing chambers. The optimal ratio for the different optical solutions can be achieved using advanced optical simulation software or can be found analytically. Yi-Kai Cheng and Jyh-Long Ch rn has in the academic paper "Irradiance formations in hollow straight light pipes with square and circular shapes" (Vol. 23, No. 2 /February 2006/J. Opt. Soc. Am A) investigated light propagation in hollow straight light pipes in more detail. The refractive optics in front of the array of mixing chambers may be implemented as a lens system or as one or more flats made of an optically transparent material where one takes advantage of the longitudinal chromatic aberration of the material. These flat (s) could be made of, but not limited to, any standard optical material.

Calibration

The relative position of mixels and pixels in the across - the-track direction must be known precisely in order to restore the mixel values correctly.

A single mixel at one end of the array (figure 3b) , which is separated from the rest of the array by a relatively large gap, would illuminate 2-3 pixels of the sensor for each spectral channel. If the PSF of the optics is known, the position of the sensor relative to the mixel array can be calculated. This single mixel may have any shape, it may also be a pinhole, or a dedicated light source. The main requirement is to have known intensity

distribution inside it.

Placing two such mixels (one on each side, of the mixel array, figure 3c) will provide the possibility to monitor the position and the length of the sensor relative to the mixel array.

Instead of using a part of the main sensor for measuring the position of the reference mixels or light sources it is possible to use small dedicated sensors besides the main sensor. Also, additional optics such as mirrors and/or prism can be used to direct, the light to these small sensors. As it has been pointed out, in order to restore the mixel content correctly the PSF of the relay optics (i.e. the shape of transition between every two adjacent mixeis) should be known. This can be found directly in the optical design software but it will not be very precise since the PSF is very affected by tolerances in the optical system. The best would be to measure the PSF directly on the assembled system for several wavelengths and field points. For this purpose a secondary array of mixeis (or pinholes) should be placed parallel to the primary array (figure 3d) . Every two adjacent elements of the secondary array should have sufficiently large gap in between - so a single PSF can be clearly observed. The simplest (but again not very precise) method of measuring PSF would be to use the signal from the 3 consecutive sensor pixels where a signal mixel from the secondary array is imaged. Much more precise information about PSF can be obtained by taking several measurements from those sensor pixels and changing the position of the mixel relative to the sensor for each measurement. This can be done by a high resolution translation stage.

When the PSF is known for several wavelengths and field points, it is possible to use the same secondary array for measuring the keystone of the optics. While it is possible to get the information about keystone from the optical design software, measuring keystone of the real assembled system allows to relax the tolerances. It is also possible to determine the position and the length of the mixel array relative to the sensor, as well as keystone and PSF, by restoring the data from a natural scene and minimising (by varying the assumed position, length, keystone and/or PSF) the inequalities in the solved overdetermined system of linear equations.

To restore the mixel values we must solve equation ( which can be written in short form-.

where

q - [M x N] matrix with M>N

S - [Nxl] vector

S ^R - [Mxl] vector

The system is overdetermined and must be solved by use of an optimisation method, such as for instance the least squares method. The least squares method searches for the solution ° that gives the best fit ° to the recorded data S ^R , i.e., that minimizes the error vector Δ where

and

The more noise that is present in the sensor pixels, the larger the error vector Δ will be. This fact can be used to calibrate the system with respect to relative position between mixels and pixels and/or relative length of the slit and/or keystone and/or PSF, based on the information in the captured image only .

Consider an example where we want to calibrate the system with respect to the relative position between mixels and pixels. An assumed relative position that is different from the correct one is equivalent to noise in the sensor pixels, and will make it more difficult to fit a solution {for the mixels values) to the recorded data (the sensor pixels) . This means that the resulting error vector Δ will be larger, and the more incorrect the assumed relative position is the larger vector Δ will be. The smallest value for the error vector Λ~Λ, _; · is found when the assumed relative position is correct. By varying the assumed relative position and determining when the error is smallest, the correct relative position between mixels and sensor pixels can be determined. Figure 14

demonstrates the principle. The figure shows the error Δ as a function of assumed relative position. The minimum error vector _m ± _n is found for the correct relative position X ₀ .

The same method can be used to calibrate for the relative length of the slit and/or keystone and/or PSF. Two-dimensional mixel array, correction of smile

The mixel array can be made two-dimensional and placed close to the sensor plane or in any intermediate image plane after the dispersive element. The most practical location for it is the sensor plane. This gives a possibility to correct not only keystone but also smile.

The array is distorted according to the smile and keystone of the optical system. Since the size of each mixel is slightly larger than the size of the sensor pixels, the light from each mixel will be distributed between 4 or more sensor pixels. Since the number of pixels is larger than the number of ixels, and the light distribution inside each mixel is known, it is possible to restore the intensity content of each mixel based on the sensor output similarly to what is done in the one- ditnensional case. The principle is shown in figure 15. The restored intensity content of the mixels forms a smile and keystone free image as long as the geometry of the two-dimensional mixel array replicates the smile and keystone of the optical system. The misregistration error caused by differences in PSF (for the optics before the two-dimensional mixel array) for different wavelengths will in this case remain uncorrected. Therefore, the optics does not need to be corrected for smile or

keystone but must have as similar PSF for different wavelengths as possible. Data processing from a system calibration perspective .

The PSF of the camera optics is measured for every extra mixel and/or pinhole (3150} (3170) (3190) (ref. figure 3) for one or more wavelengths. The PSF for the secondary mixel array (3200) is measured for several wavelengths. Measurements of PSF are done by moving the mixel array relative to the sensor (or vice versa) in small steps. This can be done with a commercially available or custom- made translat ion stage.

If we do not have a monochromatic light source or a light source with distinct spectral lines during the

calibration procedure it might be beneficial to use a shutter to close the first array of light mixing elements while the second array is being used for calibration. During normal use it will simiiax-ly be beneficial to close the secondary array using another shutter.

For the camera, with the secondary mixel array (3200) when the PSF is known, the secondary mixel array (3200) can be used to characterise the keystone of the camera optics. The main mixel array (3100) (ref . figure 3 d) should be obscured using a shutter or similar, and only the light passing through the secondary mixel array should be used. The measured keystone and the PSF data should be used to form the matrix q in the equation 2. The secondary mixel array should no be obscured, and the main mixel array with the end pinholes/mixels (3150) (3170) (31.90) should be opened. If the camera does not have the secondary mixel array, the keystone and the PSF data can be imported from the optical design software such as ZEMAX by ZEMAX

Development Corporation.

The signal from the end pinholes/mixels (3150) (3170) (3190) and the P5P data for the corresponding field points are used to determine the position and/or length of the mixel array relative to the sensor. In principle, this position and this length can be predetermined by the mechanics of the camera, but since it is desirable to know them with subpixel accuracy, the easiest way is normally to monitor the signal from the end

pinhoies/rriixels . Note that if the end pinholes/mixels are present, more or less every captured frame, i.e.

every spatial line, will have the data about the current relative position and/or length of the mixel array relative to the sensor.

The information about the position and length of the mixel array relative to the sensor is used to adjust the coefficients of the matrix q in Eq,2. Alternatively, several versions of the matrix q for different r-elative positions and lengths can be generated in advance in order to reduce the amount of calculations for finding mixel values.

The mixel values for different wavelengths can now be found by solving the overdetermined equation 2 and optimising the solution by minimising the inequalities in the solved system.

Any or ail of described calibration steps can be skipped. Instead, the data from a natural scene can be restored for different assumed positions and/or lengths of the mixel array relative to the sensor, and/or FSF' s and/or keystone. The solution, where the inequalities in the equation 2 are at the minimum, is the most accurate.

Also, if the image sensor contains a dead or hot pixel, the inequalities in the solved equation 2 will increase. If there is a reason to suspect that one of the sensor pixels is dead or hot (for example, by inspecting a few images) it is possible to exclude that particular pixel from the equation 2. The corresponding mixel can still be calculated because it is imaged onto at least two adjacent sensor pixels in each spectral band. Then it can be checked whether the inequalities in the solved equation 2 were lowered by excluding the suspected dead or hot. pixel from the calculations. If it is not known which of the sensor pixels is hot or dead, it is possible to exclude one of them at a time from the equation 2, and then solve that equation. This action should be repeated for every suspected pixel of the sensor (or even for every pixel of the sensor) . The solution of the equation 2 with the lowest inequalities will indicate which pixel is hot/dead and therefore is to be excluded from the calculations . In case of a perfectly calibrated camera and ideal mixing the inequalities in the solved equation 2 will occur only because of photon noise and readout sensor noise, i.e. they will depend on the brightness of the scene only. If the inequalities are larger, than the brightness of the scene would suggest, this would indicate that the

calibration of the camera (i.e. the coefficients in the matrix q from the equation 2) is not optimal. In other words, inequalities in the solved equation 2 provide useful diagnostic tool for the described hyperspectral camera .

If the information regarding position of individual mixels relative to the pixels is incorrect, there will be errors in the restored data . Such errors will appear on 2D images as artefacts on sharp borders between objects. One way to control, that the position and length of the mixels and pixels relative to each other are correct, is to control, that the objects with known, geometry (such as buildings) retain their rectangular shape. Also in such cases it is possible to adjust the coefficients of the restoring matrix using the objects with known geometry in the image as a reference . The energy in each raixel is calculated correctly only if the light distribution in the output of the mixing chamber is independent of the light distribution in the input of the mixing chamber. In a push-broom

hyperspectrai camera light in each spectral channel becomes partially coherent. The consequence of this partial coherence are ripples - small (in size)

variations in light intensity across the output of the mixing chamber. These ripples will have different aralitucses in different parts of the mixing chamber, also their amplitudes will depend on the position of the light source in the input of the mixing chamber, i.e., the light distribution in the output will not be completely independent of the light distribution in the input anymore. For realistic bandwidth of a single spectral channel it will not. be very noticeable, but. it is possible to further improve the accuracy of calculating the mixei values (i.e. when solving the equation 2} if the mixei array is moved a few times during every

exposure back and forth perpendicular to the optical axis and parallel to the slit. The optimal amplitude of such movement is larger than the size of ripples but smaller than the mixei width. As is obvious for a person skilled in the art, the PSF which is used for forming the matrix q in the equation 2 includes both the PSF of the optics and PSF introduced by the motion blur.

In a case when a mixei is clogged, it is possible to measure the amount of clogging and to adjust the

restoring coefficients accordingly - so that the image is restored correctly. This may improve yield in slit manufacturing and/or allow vise of a partially destroyed slit.

Three different scenarios are described below-.

« The input of a mixei is partially clogged,

β The output of a mixel is partially clogged.

* A mixel is fully clogged. If the input of a mixel is partially clogged, it still mixes light, i.e. it outputs even illumination but the intensity is lower than it is supposed to be according to the scene. This will appear on the image as consistently darker pixels. Knowing the attenuatio coefficient (i.e. amount of clogging) one can fully recover the original brightness in postprocessing, and the only penalty will be increase in noise. The amount of clogging can be measured by observing a uniform object (such as sky, an integrating sphere, an illuminated diffuse object placed between the foreoptics and the mixels) , or using

statistic of the natural scene when adjacent sensor pixels are assumed to capture similai" amount of energy over several exposures. If a mixel, when compared to its neighbours, is consistenlty darker equally in all spectral bands, then the input of this mixel is partially clogged.

If the output of a mixel is partially clogged, it stil mixes light, i.e. it outputs even illumination in its unobscured part, and outputs no illumination in the obscured part . Figure 19 a) shows a mixel 3000 partia clogged, on its output by a particle 3030. The mixel is imaged onto two sensor pixels 2100. The shape of the clogged part can be measured similarly to PSF: by look at a uniform object (such as an integrating sphere or an illuminated diffuse object placed between the foreoptics and the mixel array) and moving the mixels in small steps parallel to the direction of the mixel array (Figure 19 a,b,c5. This scanning motion, may not be nesessarv in case of large keystone. If there is large enough keystone in the system (as large as the clogged part of the mixel or larger) and the uniform object is illuminated by a broadband source,, then the shape of the clogged part can be found by comparing the illumination of the pixels where the part is imaged to the adjacent pixels in different spectral bands. In case of large enough keystone this will be equivalent to moving the slit in small steps.

If a mixel is clogged completely, the corresponding part of the scene cannot be captured, and will be recorded into the datacube as black. Alternatively _. , this mixel can be interpolated by using the data from the adjacent.

mixels .

The benefits of very low misregistration errors can also be utilized in systems where 2 or more sensor's share common foreoptics and slit. If an ordinary slit is used, it is more or less impossible to get precise {i.e. small fraction of a pixel) coregistration between different sensors, but when using mixels, it is fairly

straightforward: the calibration of the camera is handled independently for each sensor, but since each sensor captures the content of the same mixel array, then the calculated mixel values should have as good

coregistration for 2 or more sensors as they have for a single sensor. Also, with so good coregistration between 2 and more sensors it is possible to expand the use of such cameras to new applications such as high dynamic range

hyperspectral imaging, when one of two sensor captures the highlight information about the scene, and the second one captures the shadows of the same scene. And it is all done with nearly perfect spatial coregistration between highlights and shadows. Brief summary of the different embodiments

Two main embodiments of the current invention have been described in the above text. These two embodiments differ in the implementation of the actual array of light mixing elements, mixels, but use the same concepts and principles to process data i.e. , " .re s to re " data.

A first embodiment has a. linear array of mixels (3100) placed in the slit plane (2040) or in any of the other intermediate image planes . This embodiment might be a standard one, ref. figure 3 a) and figure 7 a) or it might be comprising other features as shown in figure 3 b, c, d or . It could also have shapes as described in figure 7 b, c , e , e, f, g or h. A combination thereof i.e., features and/or shapes will be apparent for the person skilled in the art.

A second embodiment is comprising a two dimensional array of light mixing element (3500) . This two dimensional array will be placed directly in front of the image sensor (2100) or in any other intermediate image plane after the dispersive element. A variation of this second embodiment is to make the two dimensional array fit the optical errors of the system i.e., so that it matches the smile and keystone of the optics . The processed or restored image data from this embodiment will be smile and keystone free .

The current invention is based on a novel principle of using light mixing elements constituting a kind of imaginary pixels which can be calculated based on data from image sensor pixels. The two embodiments described here cover all optical components all the way to the image sensor . At the output of the image sensor the electronics including the data processing system starts. In this section this will be referred to as the "data processing system:". The tasks of the data processing system is to acquire and digitise data from the image sensor and do all necessary processing of and calculation on the data so that restored image data corresponding to the mixels or imaginary pixels be outputted and stored. It will also do all necessary "house keeping" like controlling image sensor, translation stages etc,

The two outer ends of different embodiments for the data processing system are described in figures 16 and 18. The standard-component-based system (figure 16) is built using a standard camera head (2000} or industrial camera combined with a PC (5000) and an interface controller (5100) used for communication with the camera. The processing will be done in the PC processor (s) or in additional processor boards (5050) which might be comprising special processors suitable for the processing tasks. These processors (5050) could be CPUs , DPSs and its like. Custom software will be running on the PC controlling the system and acquiring and processing the data . A version of the camera data processing system in a self- contained embodiment is shown in figure 18, In this case the output is processed or corrected, and all this processing will be done by the electronics inside the camera so that it functions more or less like an

industrial camera, however, with optics for hyperspectrai use included. Figure IS and its figure text, give a more detailed description of this solution. As is obvious fox ^* a person skilled in the art, the im lementation of the data processing system can be done using the two versions described in this current

invention or any other solutions in between a custom self-contained system and a standard-component system or any other variations thereof .

The different processing steps can be grouped into logical blocks so that the functionality can be described even though it might be performed on different, hardware implementations. The term "image processing unit" refers to a combination of software, firmware and hardware performing the required image processing tasks. The image processing unit is comprising an "equation solver unit" which is the functionality described in the section "Restoring equations" and an "optimisation unit" which is the functionality described in "The general equations". As can easily be understood is that the equation solver unit is solving the equation sets as given in the

description and the optimisation unit, is optimising the solution. The matrix q in equation 2 describes important properties of the camera and can be called the "camera model" , Description of the figures

Figure 1 shows the basic principle of the current, invention, Foreopties (2020) project the scene onto the slit plane (2040) . The slit plane {2040} contains an array of light mixing elements (3100) . Each element

(called 'mixel') (30005 represents one spatial pixel in the final hyperspectral data.

The relay optics (2060} projects the image of the slit (3100) onto the sensor (2100) . The image of each mixel

(3000) is slightly (a few percent) larger than one sensor pixel. Also, to reduce noise in the final data it is beneficial to make the projection of the mixel array at least 1 pixel shorter than the length of the sensor, so that for every mixel (including the mixels at the ends of the mixel array) most of the energy is captured by the sensor. The dispersive element (2080) in the relay optics (2060) ensures that the image of the slit is dispersed approximately perpendicular to the slit.

Different wavelengths λΐ and λ2 are dispersed

differently.

Figure 2 shows the prior art. It illustrates the general principle of a push-broom hyperspectral camera. The foreoptics (2020) create an image of the scene in the slit plane (2040) . The slit (2200) acts as a field stop and cuts out a narrow portion of the image and transmits it further into the relay system (2060) . The relay system (2060) images this narro scene region onto the sensor (2100) . Due to the dispersive element (2080) in the relay, the image of the slit is dispersed on the sensor - so that the different wavelengths (λΐ, λ2) are imaged onto different parts of the sensor. Figure 3 shows a slit {3100) with mixing chambers and the additional holes/chambers for calibration purposes.

Various configurations are shown. a) . A single array of mixels (3100) . b) . A single array of mixels with an additional mixel at one end of the slit (3150} . This single roixel makes it possible to measure the relative position of the mixel array and the sensor or their relative length. c) . A single array of mixels with two additional mixels - one on each end of the mixel array (3150, 3170) . These single mixels make it possible to measure the relative position of the mixel array and the sensor as well as their relative length. d) . The array of mixels (3100) like in a) , and a

secondary array of mixels (3200) with relatively large gaps between the mixels . The secondary array allows to measure PSF and keystone of the optics before taking the measurements . e) . The mixels in the array (3100) are not necessarily squared. f) . The additional mixels (3190) at the ends of the slit are not necessarily squared. Of course all the features shown in figure 3 can be combined.

Figure 4 illustrates what a spectral keystone is. The centres of gravity of the PSF for different wavelengths are not located on the same vertical line _. , and the light coming from the same area of the scene is recorded as coming from slightly different areas depending on the wavelength.

Figure 5 shows that a keystone-Iike effect may occur in the final data even if the actual keystone of the optical system is zero. This phenomenon is caused by variation of PSF for ^* different wavelength,

Figure 6 shows various possible shapes of mixels.

Figure 7 shows that the mixel array can be straight (a) , or curved (b,c) in order to compensate optical

aberrations. Also, the mixels can have different size (d,e) . Of course, these two approaches can be combined (f) . Figure 7 (g) shows a curved mixel array, while (h) shows a mixel array with varying mixel chamber lengths . Figure 8 shows the general example of how mixels are recorded on the sensor in the case of perfectly sharp optics .

Figure 9 shows a simple numerical example of how mixels are recorded on. the sensor in the case of perfectly sharp optics ,

Figure 10 illustrates that the mixel content can be perfectly restored from the recorded pixels.

Figure 11 shows how an image of a mixel may look if the relay optics is not perfectly sharp. Figure 12 shows a possible representation of a. transition between two mixels after they have been imaged onto the sensor by the relay optics. Figure 13 shows how the mixel. content can be restored after the mixel borders have been blurred in the relay optics, and the amount of blurring is known.

Figure 14 shows how the deviation between the calculated sensor pixel values (from the approximate solution for the raixel values) and the recorded sensor pixel values changes as a function of the assumed position of mixels relative to the sensor pixels.

When the assumed relative position between the mixels and the pixels is correct, the inequality is minimum. This can be used for measuring the relative position of mixels and pixels by analysing the captured data. The method can be extended for measuring more than one parameter simultaneously: for example both the relative position of mixels and pixels, and the relative length of the mixel array and the pixel array. In addition the PSF and/or keystone can be measured.

Figure 15a shows a two-dimen ional mixel array (3500) on top of a two-dimensional pixel array or senso (2100) .

Figure 15b shows the same layout but this time the mixel array is distorted (3500) to fit the smile and keystone generated by the camera optics. The restored mixel intensities will represent smile- free and keystone-free hyperspectral data.

Figure 16 shows a system set-up based on a computer

{5000) with an interface controller (5100) for

communicating with and controlling the camera body (2000) with sensor (2100} . The camera body (2000) is connected to the spectrometer part (1000) with entrance

aperture {1050) . The computer (5000) will control the camera body and the spectrometer, and will also receive data acquired from the sensor {2100) and process them to restore corrected data in the standard CPU system of the computer or utilising one or more optional signal or graphics processing board (s) {5050). Temporary and processed data may be stored on a permanent storage device (5200) like a hard disk.

Figure 17 shows a simplified sketch of the data flow and the process steps in the signal processing related to the restoring of data. An analogue sensor signal (6000) is digitised giving digitised raw data (6050) which may undergo an optional pre-processing (6100) . At this level the raw data might be temporarily stored in a raw data storage (6200) for later processing or immediately go further in the chain to data restoring (6300) and/or calibration (6400) . The output of this is restored or corrected data (6500) which might be displayed right away or be stored in the final data storage means (6600) .

Figure 18 shows one possible implementation of the electronic system (7000) of a self-contained

hyperspectral camera according to the present invention. The figure shows the image sensor (2100) feeding analogue signals into an amplifier system (7100) which output is input to an analogue to digital converter (7200) . The figure also shows electronics for camera control (7300) which will control a positioning stage and possibly the image sensor. Electronics for I/O control (7400) like external sync signals is also shown. An interface controller (7500) is shown and this might be controlling interface types like CameraLink, FireWire, USB, Ethernet, GigE etc. A memory unit (7600) is shown and this might, be comprising one or more of Pl sh/EEPROM (7620) , DRAM (7640} , SRAM (7660}, buffer or FIFO circuits (7680} or any other types of memory devices. All this is

interconnected to the processor module (7700) comprising at least one processor (7780) which might be a general micro processor, a micro controller or a digital signal processor or an embedded processor in a field

programmable gate array. The processor module (7700) might also be comprising one or more additional processor units (7720) . These additional processor units (7720) might be custom modules optimised for tasks or sub-tasks related to equation solving and optimisation of

overdetermined equation sets.

Figure 19 shows some examples of a partially clogged light mixing element, Figure 19 a) shows a mixel (3000) partially clogged with matter (3030) , The clogging matter is only present in the projection onto the

rightmost pixel on the image sensor (2100) . Figure 19 b) shows another scenario where the clogging matter has started to cross the border between pixels, and figure 19 c) shows a case where the clogging matter is present in the projection onto both pixels.

The table below lists the numbers used in the figures-.

Description

Spectrometer, complete optical part

Entrance aperture of spectrometer

Camera body _. , comprising image sensor, control electronics, data acquisition system etc

Foreoptics

Slit plane

Relay optics

Dispersive element

Image sensor

Slit of prior art type

Mixing element, mixel

Array of mixel elements, mixels, slit One additional mixel for position measurement A second additional mixel for position and length measurement

Circular or pinhole shaped additional pixels Additional array of mixels for calibration purposes

Two-dimensional mixel array

Two-dimensional mixel array with distortion to correct for optical errors

Computer

Signal or Graphics processing .board

Interface controller in compute!" like CameraLink, Ethernet, USB, FireWire or similar Permanent storage device, hard disk, solid state disk and its like

Analogue sensor signal

Digitised raw data

Pre-processing step or pre-processed data

Storage means for raw data

Data restoring function

Calibration function

Restored / corrected data

Final data storage means

Electronic system of self contained camera Amplifier system

A/D converter

Electronics fox- camera control

Electronics for I/O control

Interface controller for CameraLink, Ethernet,

USB, Fireivire or similar

Memory unit

Flash EEPROM

DRAM 7660 SRAM

7680 Buffer or FIFO

7700 Processor module

7720 Custom or special task processor module

Processor like micro processor, micro 7780 controller or DSP