Field of the Invention The present invention relates to a microscope with an image sensor and a method therefor.

Background Art For studying small structures, microscopes are gene- rally used. There are a plurality of different microscop- ing methods, of which the most widespread one is optical- ly imaging microscoping. A common feature of all micro- scopy methods is that only a limited part of the object can be examined with high resolution on a separate occa- sion. This area is usually called the field of view of the microscope. The field of view of an optically imaging microscope is determined among other things by the size of the lenses in the optics. The resolution is determin- ed among other things by the numerical aperture of the optics. It is possible to achieve a large field of view and good resolution by using large lenses with a short focal length. However, it is extremely expensive to pro- duce to large lenses, especially if they should also have a short focal length.

As a result, it has only been possible to study relatively small areas on a separate occasion if at the same time high resolution is desirable.

A further problem of conventional microscoping is that it is necessary to study the object through the ocular and turn control means to move the field of view across the object. To sit studying the object through the ocular for a long time is tiring.

One frequently wants to study old preparations on a subsequent occasion. It has then been necessary to file the preparations and put them into the microscope once more, which however is time-consuming work. Moreover, it is not always possible to file the preparations. This applies in particular to preparations in vitro since the

preparations must be dried before being stored, which may cause a change of the preparations.

With a view to simplifying handling, it happens that images of relevant areas of a preparation are recorded and stored. It will consequently be possible to study the areas of the images later on without having to file the preparations. This also renders it possible to study relevant areas of wet preparations a long time after the recording of the images.

However, when the recorded images are studied, an image of the preparation in its entirety will not be obtained. Nor is it possible to get an idea of how diffe- rent images of different parts of the preparations are related to each other.

Thus there is a need for a microscope which simpli- fies handling when studying a preparation on a plurality of subsequent occasions.

Summary of the Invention An object of the present invention is to provide a microscope, which minimises the handling of preparations in repeated studies of an individual preparation.

A further object of the present invention is to pro- vide a microscope which is adapted to store images of an area which is larger than the field of view of the micro- scope while at the same time the relationship between different parts of the preparation can be recovered from the stored images.

One more object of the present invention is to pro- vide a method for microscoping, which facilitates repeat- ed studies of an individual preparation.

A further object of the present invention is to pro- vide a memory medium with a computer program, which faci- litates repeated studies of an individual preparation.

These objects are achieved by means of a microscope, a memory medium and a method according to the appended claims.

A basic concept of the invention is to form a digi- tal image with high resolution of a larger part of the object than has been possible with conventional equip- ment.

A microscope according to the invention comprises an object holder, a display, optics which in an image plane forms an image of an object placed in the object holder, and an image sensor for recording the image. The microscope is characterised in that it further comprises a storage means, an input means for selecting an area in the composite image and a resolution in the selected area, and a calculation means which is connected to the image sensor and which is adapted to digitally store in the storage means at least two recorded images, each of the images partly overlapping at least one further image, to join the at least two recorded images so that the com- posite image is an image of the area which was reproduc- ed on the at least two recorded images, and to store in the storage means information about the position of the recorded images in the composite image. The calculation means is further adapted to show the selected area with the selected resolution on the display.

By the microscope being adapted to form a digital image of a plurality of partial images, it is possible to later on study an optional area of the image without the preparation needing to be available.

By joining the recorded images is meant that the pixels in the recorded images are given positions in a common system of coordinates.

Preferably, the calculation means is adapted to store in the storage means the composite image, but alternatively each of the recorded images can be given a position which is stored in the storage means, the pixels obtaining a common system of coordinates.

According to one aspect of the present invention, a computer-readable memory medium is provided, on which a computer program is stored, which is adapted to be used

for storing digital images which have been recorded with an image sensor. The computer program comprises instruc- tions for making the computer read an input signal which represents at least two digital images, to join the digi- tal images to obtain a composite image of the area which is reproduced on the at least two digital images, to read information which defines an area and a resolution for the area in the composite image, and to output a digital output signal which represents the selected area with the selected resolution.

It is advantageous to record the partial images with the highest resolution allowed by the optics since the greatest possible freedom of action will then be obtained when later studying the composite image.

Preferably the calculation means is adapted to con- trol the object holder. The calculation means moves the object holder between each recording of a new image and stores at least two recorded images in the storage means.

The displacement is adjusted so that the recorded images partly overlap each other.

It is advantageous to record the position of the object holder together with the recorded image since such a position can be used when the images are put together.

The position of the object holder is obtained from e. g. stepping motors which are used to move the object holder.

According to a less preferred embodiment, the object holder is moved manually. The calculation means is then adapted to receive an input signal with information about the relationship between the recorded images so that it can join the images in the correct manner. Manual preci- sion displacement of the object holder, however, is very time-consuming, and therefore automatic displacement is preferred.

The joining of the images can be carried out in various ways. According to a preferred embodiment of the invention, the calculation means is adapted to join two recorded images by moving the images in relation to each

other so that an edge area of one image is fitted to an edge area of the other image, said edge areas overlapping each other. Consequently, high precision in the motion of the object holder is not necessary, which makes it possible to use a simpler and, thus, less expensive object holder.

The recorded images are stored either as black-and- white images or colour images. In both cases, each pixel is associated with at least one value which describes colour or luminous intensity. It has been found advanta- geous to fit the edge areas to each other by minimising the sum of squares of the difference between a value for each of the pixels in one edge area and the value of the overlapping pixel in the other edge area. The calculation is carried out for at least two different relative posi- tions between the images, whereupon the position which yields the smallest sum of squares is selected as the correct position.

To minimise the calculation work when the images are to be fitted to each other, it has been found advanta- geous to carry out the fitting of two images in at least two steps, the resolution for the edge areas increasing for each step. As a starting point for the fitting, use is made of approximate positions which have been recorded with the images. The resolution of the recorded image is reduced digitally, whereupon the images are fitted toge- ther. The square of the differences between the pixels is summed up with the images in at least two different rela- tive positions. Subsequently, a higher resolution is used, whereupon the fitting-together is repeated for this higher resolution. The optimal position from the first fitting-together is used as a starting point for the second fitting-together.

To further minimise the calculation work, it has been found advantageous to spatially filter the images before putting them together. This means that the effect of, for example, grains of dust which only appear on one

of the images, is minimised. The filtering improves the possibilities of minimising the sum of squares by mini- mising the sum of squares in one direction at a time.

Minimising the sum of squares in one direction at a time reduces the calculation work compared with the case where the sum of squares is calculated for all possible rela- tive positions in a predetermined number of pixels from the starting position. However, with unfiltered images it is not always possible to obtain the minimum sum of squares by minimising the sum of squares in one direction at a time.

When the images are stored as colour images, the difference between two pixels can be defined as the sum of the differences for each of the colour compo- nents included. Alternatively, the difference between two pixels can be defined as the difference between the intensity values of one of the colours included.

The put-together image has areas that are present in more than one of the recorded images. The value of the pixels in the put-together image is advantageously an interpolation between the value of corresponding pixels in the recorded images, the value of each of the pixels being weighted in relation to how close to the edge of the respective images they are located. Thus, pixels located far away from the edge are suitably taken into greater consideration than corresponding pixels located close to the edge of another recorded image.

According to one aspect of the present invention, the calculation means is adapted to select edge areas having high spatial frequencies and a high contrast and to join the recorded images by fitting-together the selected edge areas. By selecting edge areas with high spatial frequencies and a good contrast, a good fitting- together is obtained while at the same time the calcula- tion work is minimised.

When more than two images are to be put together, there are a plurality of possible orders in which the

images are put together. It is then advantageous for the microscope to be adapted to recognise high spatial fre- quencies with a high contrast in the edge areas and to put together pairs of the recorded images in an order determined by the spatial frequencies and the contrast at the edge of the images. Thus, the composite image is increased by a recorded image in each step. Consequently not all edges between the different images are adjusted in the composite image. This gives the advantage that the result in the composite image will be less sensitive to structureless areas in the edge area of an image.

Of course, it is possible within the scope of the invention to adjust all edges between all recorded images at the same time. However, extensive calculations are necessary to achieve simultaneous adjustment of all recorded images.

If all images are to be adjusted to each other at the same time it is, however, advantageous to calculate a comparative value for a plurality of different relative positions for each pair of adjoining images, the compa- rative value being a measure of how well the edge areas correspond to each other, and subsequently put together the recorded images in relative positions, so that a weighted sum of the comparative values for the pairs of images is maximised.

By this method, the calculation work is minimised while at the same time a better total fitting-together of the images is obtained. By weighted sum is meant that the different comparative values are weighted according to how reliable they are.

It has been found advantageous that the comparative values are based on the covariance of the areas where the images overlap each other for a number of different relative positions between the images.

In order to further reduce the calculation work, it is advantageous for the comparative values to be calcu- lated for displacements of the images in two orthogonal

directions, the images being projected onto rows and columns respectively in the images before the compara- tive values are calculated.

By calculating comparative values for one direction at a time, the calculation work is significantly reduc- ed. Projection of the images implies that the intensity values in the rows and columns respectively are added up.

It has also been found advantageous to calculate the sum of the comparative values in each of two orthogonal directions at a time. This significantly reduces the cal- culation work.

Each of the pixels has at least one intensity value.

Preferably the calculation of comparative values is car- ried out by calculating the covariance of the overlapping areas.

When putting together images and calculating compa- rative values, it is advantageous to carry out the cal- culations at a gradually increasing resolution. In a first step, images having a lower resolution are then generated digitally, starting from the recorded images.

This reduces the calculation work.

After putting together the images, a total image is obtained which reflects the entire recorded area.

It goes without saying that the above features can be combined in the same embodiment.

With a view to further elucidating the invention, detailed embodiments of the invention will be described below without, however, the invention being considered to be restricted thereto.

Brief Description of the Drawings Fig. 1 illustrates a microscope according to a pre- ferred embodiment of the present invention.

Fig. 2 is a flow chart of the method according to a preferred embodiment of the present invention.

Fig. 3 is a schematic view of images recorded by means of a microscope according to an embodiment of the present invention.

Fig. 4 shows two images recorded by means of a microscope according to an embodiment of the present invention.

Detailed Description of the Invention Fig. 1 illustrates a microscope according to a pre- ferred embodiment of the present invention. An object 1 is placed on a movable object holder 2 and illuminated from below by a bulb 3. An objective 4 images the object on a CCD 5 which has a number of image elements 6 at a mutual centre distance D. The CCD 5 is intended for recording of images and is connected to a calculation means 7 which in this case is a computer 8 with a display 9 and an input means 10 in the form of a keyboard and a mouse. Moreover, the computer is connected to and can control the object holder 2.

A user of the microscope initiates recording of an image of the object 1. In the initiation, a user inputs via the input means 10 information about how the image is to be recorded, i. e. the desired size of it, the desired resolution and how the object holder is to be arranged in relation to the objective at the beginning of the record- ing. The object 1 is, for example, a specimen of white blood cells which is to be studied in the microscope. By means of the inputted information, the microscope records images that are stored in a storage means 11 in the com- puter 8. Between each recording and storing of an image, the computer 8 controls the object holder 2 to be moved so as to image a new area, which partly overlaps the pre- ceding area that has been recorded. The storage means 11 can be a direct access memory or a hard disk in the com- puter 8. Thus, a number of images are recorded and stor- ed, which correspond to the area that was defined in the inputting via the input means 10.

Fig. 2 is a flow chart of the function of a micro- scope according to a preferred embodiment of the present invention. In step 12, information about which area is to be studied in the microscope is inputted via the input

means. With the aid of the inputted information, the microscope records in step 13 a number of images which at least partly overlap each other. In step 14, the recorded images are put together so that a composite image forms, which covers the entire area that was cover- ed with the recorded images. In step 15, the composite image is shown on the display 9 of the calculation means.

In step 16, a user inputs information about which area in the shown image is to be shown at higher magnification, the magnified image also being inputted in step 16. In step 17, the selected area at the selected magnification is shown on the display 9. Of course, the image in step 17 cannot be shown with a better resolution than that recorded in step 13. However, in step 17 it is possible to show the image at an optional magnification. According to the preferred embodiment, the images are recorded in step 13 with maximum resolution to allow as great freedom of action as possible in steps 16 and 17. The image shown in step 15 does not always reproduce the maximum resolu- tion that is found in the images recorded in step 13 since the size of the display is limited.

Fig. 3 shows a number of images such as 18 and 19 that have been recorded in step 13. The edge areas are graded in respect of high spatial frequencies and a high contrast. The images recorded in step 13 are put together to a composite image 20. When the recorded images are put together, they are put together in pairs in an order which is determined by how they have been graded with respect to contrast and the presence of high special fre- quencies. For example, one begins with image 18, to which image 19 is added. When images 18 and 19 are to be join- ed, an edge area of image 18 is compared with an image area of image 19 so that they correspond with each other within the area 21 in which they overlap each other. Sub- sequently, the remaining images are added in an order which is determined by which edge has the best contrast and the greatest presence of high spatial frequencies.

When all registered images have been put together, the composite image is shown on the display 9. A user inputs via the input means information about which area in the composite image 20 is to be shown at higher magnifica- tion. The selected area can be, for example, the area 22, which contains information from four recorded images. In the areas of the composite image 20 which contain infor- mation from more than one image, there is made in each pixel an interpolation between the values in the pixels from each of the recorded images, weighting being car- ried out so that pixels located close to the edge of a recorded image obtain less weight than such pixels as are located further away from the edge of another image.

Fig. 4 shows two images 23 and 24 which have been recorded in step 13. With reference to Fig. 4, the put- ting-together of two recorded images will now be describ- ed in more detail. Images 23 and 24 have been recorded with a certain overlap and their fitting-together is car- ried out by making a comparison within an area 25 so that correspondence is obtained between the two images in the area 25. By means of the computer 8, the area 25 is selected, in which there is a great presence of high special frequencies and a high contrast. In the area 25 there are structures that are to be found on both images, such as the structures 26 on the image 23 and the struc- tures 27 on the image 24. The resolution in the images 23,24 is reduced electronically by a factor 8, so that the number of pixels in the images is reduced by a factor 64. The image 24 is then moved in relation to the image 23 so that the sum of squares of the differences between the value in one pixel in the image 23 in the area 25 and the overlapping pixel in the image 24 in the area 25 is minimised. The sum of squares of the differences is in most cases minimal when the image 24 has been moved so that the structures 26 have the same position as the structures 27. Then the minimising is repeated for a gra- dually increasing resolution when the obtained relative

position is used as a starting position for the next minimising step. The resolution is increased by a factor 2 in each minimising step until the maximum resolution has been achieved. As a result, the images are placed gradually closer to their correct relative position. In each minimising step, the images are moved to different positions, which represent a combination of the orthogo- nal displacements from the starting position. The image 23 is composed of a number of pixels which are each asso- ciated with a value describing the luminous intensity of the pixel. If the image 23 is a colour image, it is asso- ciated with three different colours, red, green and blue, which each have a value describing the intensity of pre- cisely that colour in the pixel. If the images are colour images, the comparison is made in the edge area between each of the colours or between a converted image, in which case the converted image is a grey-scale image of the original colour image. It is also possible to use only one of the colours when comparing the images.

With reference to Fig. 3, an alternative method of putting together the images will now be described. First the optimal fitting between each pair of images 18,19 is determined. The fitting-together of two images is carried out by calculating the covariance of the area within which the images overlap each other for a plura- lity of different relative positions between the images 18,19. The covariance is used as a comparative value to find the best fitting-together of the images. Before cal- culating the covariance, a projection of the images is made onto each of two orthogonal directions in the images. Thus, the rows and the columns respectively are added up, after which the covariance is calculated in the direction of the columns and rows respectively for a num- ber of different relative positions. The covariance is calculated in the usual fashion except that no dividing by the number of measuring points occurs. For each of the pairs of images a set of covariances is obtained as a

function of the displacement. In each of the sets con- taining covariances for different displacements, its maxima are searched out. This point and two adjoining points are used to form a curve of the second degree, which together with the corresponding curves of the second degree for the other sets is used to put together all the recorded images. When fitting together the pairs of images, a weight W is calculated which is a measure of how reliable the fitting-together is. The weight W is given by: wherein ie [S (i) 20. 25max (S), |i-max|>2]-j E [S (j) 20. 50max (S), |j-max|=2], wherein S (i) is the covariance at the displacement i, and S (max) = max (S). If W becomes negative, the fitting- together is considered unusable and W is set at zero. If the images are not square, it is easier to find a good fitting-together of between the images along the long edge. Then the fitting-together is advantageously carried out in this direction first.

After determining the relative positions of each pair of images, all the recorded images are put together so that a weighted sum of the comparative values is maxi- mised. This means that a weighted sum of the covariances for the different pairs of images is maximised. This, too, is made first in one of the direction of the columns and the direction of the rows and then in the other direction. The maximisation is carried out by minimising the function

wherein m is the number of images in a column, n the number of images in a row, Ci, k, o is the optimal fitting- together of image [l, k] and its right neighbour and Cl, k, l is the optimal fitting-together of image [l, k] and its lower neighbour. Wl, k, o and Wl, k, l are correspondingly the weight of the respective two-image fittings. Xl, k is the displacement of the image in the x direction or in the y direction depending on the optimisation that has been carried out.

A person skilled in the art realises that the inven- tion is limited to the embodiments described, which can be varied within the scope of the invention. For example, the CCD can be replaced by a vidicon. Moreover the user need not necessarily initiate the recording of the par- tial images by specifying the size and resolution of the completed image since this can be carried out according to a predetermined schedule stored in the computer.