The present disclosure relates to a method and device for determining whether an obtained verify fingerprint image is genuine or impostor compared with an enrolled fingerprint.

BACKGROUND

Various types of biometric systems are used more and more in order to provide for increased security and/ or enhanced user convenience.

In particular, fingerprint sensing systems have been adopted in, for example, consumer electronic devices, thanks to their small form factor, high performance and user acceptance.

When authenticating a user by means of fingerprint, a verify finger print image of a user finger is obtained by means of a fingerprint sensor and compared with a stored enrolled fingerprint image. Features extracted from the verify fingerprint image are matched to a feature template of the enrolled fingerprint image to determine a number of matches of the features, based on which the verify fingerprint is classified as genuine or impostor and the user is authenticated or rejected.

SUMMARY

The matched features between features extracted from a verify fingerprint image and a template of at least one enrol image of an enrolled fingerprint may be categorized as inliers or outliers, depending on if they conform to the same chosen geometrical transform of the template features or not. The number of inlying matches, of a feature set in the enrol image and a corresponding candidate feature set in the verify image, is an indicator to whether the verify image is genuine, i.e. from a genuine user, or impostor, i.e. from an impostor user, and thresholding can be applied to take the final classification decision. An inlying match (or inlier) may thus be of verify and enrol features that conform to a geometrical transformation with sufficient accuracy. Each inlying match always involves exactly two features, one from the verify image and one from the enrol image, and the transform maps one of the features to the other with sufficient accuracy. When setting the inlier threshold, a trade-off is made between the desired False Acceptance Rate (FAR) and False Rejection Rate (FRR). This is also known as the operating point of the classifier. For smaller fingerprint sensor sizes, the number of features in the enrol and verify images will inevitably be lower, and it might be hard to reach the desired operating point by classifying on the inlier count only.

When matching the extracted features, the set of inlying matches maybe determined by simultaneously estimating a geometrical transform between the enrol and verify images, e.g. by means of a Random Sample Consensus (RAN SAC) algorithm. This transform maybe used to compute the amount of overlap between the enrol and verify images, e.g. expressed in number of pixels or relative in percentages of image size. In accordance with

embodiments of the present invention, the expected correlation between overlap and number of inliers, i.e. inlying matches of features, (also called inlier count) is used in order to create an improved way of classifying the verify fingerprint image as genuine or impostor.

According to an aspect of the present invention, there is provided a method performed in an electronic device comprising a fingerprint sensor for determining whether an obtained verify fingerprint image is genuine or impostor compared with an enrolled fingerprint. The method comprises, by means of the fingerprint sensor, obtaining the verify fingerprint image of a finger in proximity to a detection surface of the fingerprint sensor. The method also comprises, by means of image processing, extracting features from the verify fingerprint image. The method also comprises retrieving at least one stored template of features of at least one enrol image of the enrolled fingerprint. The method also comprises matching the extracted features to a geometrical transform of the features of the template, wherein positive matches between the extracted features and the template features are determined to be inlying matches. The method also comprises determining an area of overlap between the verify fingerprint image and the at least one enrol image based on the geometrical transform of the template features. The method also comprises classifying the verify fingerprint image as genuine or impostor based on the number of the inlying matches in relation to the determined overlap area.

According to another aspect of the present invention, there is provided a computer program product comprising computer-executable components for causing an electronic device to perform an embodiment of the method of the present disclosure when the computer-executable components are run on processing circuitry comprised in the electronic device.

According to another aspect of the present invention, there is provided a fingerprint sensing system comprising a fingerprint sensor, processing circuitry, and data storage storing instructions executable by said processing circuitry whereby said fingerprint sensing system is operative to, by means of the fingerprint sensor, obtaining a verify fingerprint image of a finger in proximity to a detection surface of the fingerprint sensor. The system is also operative to, by means of image processing, extract features from the verify fingerprint image. The system is also operative to retrieve at least one stored template of features of at least one enrol image of an enrolled fingerprint. The system is also operative to match the extracted features to a geometrical transform of the features of the template, wherein positive matches between the extracted features and the template features are determined to be inlying matches. The system is also operative to determine an area of overlap between the verify fingerprint image and the at least one enrol image based on the geometrical transform of the template features. The system is also operative to classify the verify fingerprint image as genuine or impostor based on the number of the inlying matches in relation to the determined overlap area.

According to another aspect of the present invention, there is provided an electronic device comprising an embodiment of the fingerprint sensing system of the present disclosure. It has now been realised that when the verify image is genuine, generally a relatively large overlap of the verify and enrol images is expected when the number of inliers (i.e. inlying feature matches) is large, and a relatively small overlap when the number of inliers is smaller. This is simply due to that the density of inliers is expected to be generally uniform throughout the overlapping area for a genuine verify image.

Conversely, if the verify image is impostor (i.e. is of a non-authorized finger) the number of inliers is usually relatively low, and the matching threshold should be set such as to make it unlikely that such a verify image is authorized. The overlap however, is not necessarily small since the false inliers may appear anywhere in the image, e.g. as a constellation of features concentrated in a relatively small area of the overlap which are similar in both the verify and enrol images, thus resulting in inliers.

Thus, the inventors have realized that for a genuine verify image, the overlap increases with the number of inliers, while for an impostor verify image, the size of the overlap is typically unrelated to the number of inliers.

It follows that, in accordance with the present invention, the verify

fingerprint image may be classified as genuine if the relationship between the number of the inlying matches and the determined overlap area implies that the density of inliers is generally uniform throughout the overlapping area, or classified as impostor if the relationship between the number of the inlying matches and the determined overlap area implies that the determined overlap area is unrelated to the number of the inlying matches

It is to be noted that any feature of any of the aspects may be applied to any other aspect, wherever appropriate. Likewise, any advantage of any of the aspects may apply to any of the other aspects. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings. Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated. The use of “first”,“second” etc. for different features/ components of the present disclosure are only intended to distinguish the features/ components from other similar features/components and not to impart any order or hierarchy to the features/components.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be described, by way of example, with reference to the accompanying drawings, in which:

Fig l is a schematic illustration of an embodiment of an electronic device comprising a fingerprint sensor interacting with a finger of a user, in accordance with the present invention.

Fig 2 is a schematic block diagram of an embodiment of a fingerprint sensing system of an electronic device, in accordance with the present invention.

Fig 3a illustrates an embodiment of inlying matches in an overlap between a verify fingerprint image and an enrol image, in accordance with the present invention.

Fig 3b illustrates another embodiment of inlying matches in an overlap between a verify fingerprint image and an enrol image, in accordance with the present invention.

Fig 4a illustrates an embodiment of inlying matches in an overlap between a verify fingerprint image and a plurality of enrol images, in accordance with the present invention. Fig 4b illustrates another embodiment of inlying matches in an overlap between a verify fingerprint image and a plurality of enrol images, in accordance with the present invention.

Fig 5 illustrates a decision boundary in a two-dimensional coordinate system of overlap (%) contra number of inlying matches, in accordance with the present invention.

Fig 6 is a schematic flow chart of embodiments of a method of the present invention.

DETAILED DESCRIPTION

Embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments are shown.

However, other embodiments in many different forms are possible within the scope of the present disclosure. Rather, the following embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like numbers refer to like elements throughout the description.

Figure l shows an electronic device l, here in the form of mobile phone, e.g. smartphone, comprising a display stack 2, e.g. comprising touch functionality (i.e. a touch display) and a fingerprint sensor 2. The fingerprint sensor 2 comprises fingerprint sensor circuitry, e.g. for outputting a grey-scale image or the like where different intensities in the image indicate the contact between a detection surface of the fingerprint sensor 2 and a finger 4 placed there on, e.g. as part of fingerprint authentication or navigation using the fingerprint sensor.

The fingerprint sensor 2 may operate according to any sensing technology. For instance, the fingerprint sensor maybe a capacitive, optical, thermal or ultrasonic sensor. Herein, a capacitive fingerprint sensor, which maybe preferred for some applications, is discussed as an example. The fingerprint sensor may comprise a two-dimensional array of fingerprint sensing elements, each corresponding to a pixel of the image outputted by the fingerprint sensor, the pixel e.g. being represented by a grey-scale value. The fingerprint sensor maybe located at a side of the display 3, outside of the display area of the display, as shown in figure 1, or it may be located within said display area. The outputted image is herein called a verify image since it is used for verification by comparing with a stored enrol image of an enrolled fingerprint. The verify image may for instance be in the form of a two- dimensional or one-dimensional pixel array, e.g. of grey-scale values.

Each image pixel may provide an image intensity, be it of a grey-scale value or other value. For example, for a capacitive fingerprint sensor, a high pixel intensity (e.g. white in grey-scale) implies low capacitive coupling and thus a large sensed distance between the detection surface and the fingerprint topography. A high pixel intensity may result because the finger does not cover the part of the detection surface where the sensing element

corresponding to the pixel is located. Conversely, a low pixel intensity (e.g. black in grey-scale) implies high capacitive coupling and thus a small sensed distance between the detection surface and the fingerprint topography. A low pixel intensity may result because the corresponding sensing element is located at a ridge of the fingerprint topography. An intermediate pixel intensity may indicate that the sensing element is covered by the finger topology but is located at a valley of the fingerprint topography.

From the verify image, features are extracted, e.g. minutiae such as bifurcations and ridge endings, as well as other characterizing features of the verify fingerprint. Each feature is associated with a set of x- and y-, or other location defining, coordinates, specifying their respective locations within the verify image. The features, with their respective associated coordinates extracted from the verify image may be collected in a verify template.

Similarly, the features of the enrol image, or plurality of enrol images, may be stored in an enrol template. A template, verify or enrol template, as discussed herein, may be regarded as a container for information associated with an image, including e.g. features (associated with coordinates), user identifier (ID), finger ID and/or other information. Since the fingerprint of the verify image may not have the same size, inclination, rotation etc., or be in the same location in the image, as the fingerprint in the enrol image, the features extracted from the verify image are matched to a geometrical transform of the features of the enrol template. Put another way, a geometrical transform is computed from the two feature sets, the set of features extracted from the verify image and the set of features of the enrol template, by aligning them (compensating for e.g. relative translation and rotation and (optionally) scaling) with each other to obtain inliers or to maximize the number of inliers. To reduce the complexity, e.g. three features (e.g. any three) extracted from the verify image may first be used for matching with a geometrical transform of a corresponding number (e.g. three) of the enrol features. The geometrical transform typically includes any or all of translation, rotation and scaling of the template features. Herein, translation is displacement/movement in the image space, i.e. where in the enrol image can a match be found for a verify feature, rotation relates to how the verify and enrol images (represented by the coordinates of their respective features) are rotated to find inlying feature matches, and scaling relates to how the distances between the features are zoomed in a

coordinated manner to find inlying feature matches. A RANSAC algorithm may be used to select a suitable transform, preferably an affine

transformation e.g. a rigid affme transformation (rotation and translation only), which results in an amount of inliers above a predetermined threshold for potentially classifying the verify image as genuine. However, in

accordance with the present invention, the area of overlap between the verify image and the enrol image in accordance with the selected transform is also taken into account before the verify image is classified. When comparing features from the verify and enrol images with each other, some feature pairs (one from each of the verify and enrol images) may form inliers when conforming to a first transform, while other feature pairs may form inliers when conforming to a second transform, and yet other features may not conform to any common transform at all. In this case, a choice is made between the first and the second transform when determining which of the first and second transforms to proceed with from the matching step of the present invention, e.g. based on the number of inliers and/or on the number of inliers in relation to the overlap area (similar to in the classification step of the present invention).

Figure 2 is a schematic block diagram of an embodiment of a fingerprint sensing system 10 of an electronic device l, e.g. as in figure l. The system 10 comprises processing circuitry n e.g. a central processing unit (CPU). The processing circuitry n may comprise one or a plurality of processing units in the form of microprocessor (s). However, other suitable devices with computing capabilities could be comprised in the processing circuitry n, e.g. an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processing circuitry n is configured to run one or several computer program(s) or software (SW) 15 stored in a data storage 14 of one or several storage unit(s) e.g. a memory. By running the SW 15, applications comprising at least a part of the processing circuitry 11 may be formed, such as fingerprint image obtaining circuitry 12 configured for obtaining e.g. the verify fingerprint image by means of the fingerprint sensor 2 in the system 10, and image processing circuitry 13 configured for image processing e.g. of the obtained verify fingerprint image e.g. for extracting features from the verify image and matching the features to a transform of the features of the enrol template. The storage unit is regarded as a computer readable means 14 as discussed herein and may e.g. be in the form of a Random Access Memory (RAM), a Flash memory or other solid state memory, or a hard disk, or be a combination thereof. The processing circuitry 11 may also be configured to store information in the storage 14, as needed, e.g. about the enrol template(s) 16 of the enrol image(s).

The electronic device 1 may be any device comprising a fingerprint sensor 2 and able to process images obtained by means of said fingerprint sensor. For instance, the device 1 may be any of a mobile phone e.g. a smartphone, a smart card, a tablet computer, a portable computer e.g. a laptop computer, or a stationary computer e.g. a desktop computer, a server or a mainframe computer. Figures 3a and 3b illustrate embodiments of inlying matches 34 in an overlap 33 between a verify fingerprint image 31 and an enrol image 32. The situation in figure 3a is typical for a genuine verify image 31, where the inliers 34, i.e. the positively matched features within the area of overlap 33, are

substantially evenly dispersed throughout the overlap. In contrast, in the situation in figure 3b, the inliers are concentrated to only a limited, relatively small part, of the overlap. The number of inliers 34 in figure 3b may in itself be above the classification threshold for a genuine verify image, but in view of the relatively large overlap a much higher number of inliers was expected, resulting in the verify image being classified as impostor. However, had the overlap been smaller, a false positive may instead have been the result of the classification.

In some embodiments of the present invention, a plurality of enrol images 32, relating to the same finger 4 maybe used to form the at least one stored enrol template 16. In some embodiments, the stored enrol template 16 is based on a plurality of enrol images 32. In some embodiments, the plurality of enrol images 32 may have been stitched together to form a single stitched enrol image. In other embodiments, the stored enrol template 16 comprises features of each of the individual (non-stitched) plurality of enrol images 32, implying for instance that a same feature may be present in the template from more than one enrol image if the enrol images are themselves overlapping. In some other embodiments, a plurality of enrol templates are stored, one for each of the plurality of enrol images 32.

Figures 4a and 4b illustrates examples of overlap 33 between a verify image 31 and a plurality of enrol images 32. In the example of figure 4a, there are three enrol images 32, denoted 32a, 32b and 32c, respectively. As can be seen in figure 4a, the enrol images 32 are spread out, i.e. all includes different parts of the enrolled fingerprint of finger 4. Thus, each enrol image 32 creates its own different overlap 33 with the verify image 31. Herein, enrol image 32a creates an overlap 33a, enrol image 32b creates an overlap 33b and enrol image 32c creates an overlap 33c. The situation of figure 4a may be preferable to the situation illustrated in figure 4b, since the combined overlap area of overlaps 33a, 33b and 33c may be larger than the overlap area 33 in figure 4b formed by the four enrol images 32a, 32b, 32c and 32d which all show the same part of the fingerprint. However, the enrol images of figure 4b may e.g. be of different quality, resulting in a combined image with improved quality, why there may thus still be an advantage of having a plurality of enrol images compared to having only one enrol image.

It is reasonable to believe that the matching in figure 4a conveys more information than the matching in figure 4b , since the enrol images 32 overlaps a larger part of the verify image 31. In order to distinguish between these two cases, it is possible to include the mutual overlaps 33 into the feature vector: x = [s(v, P), s(v, t ₂ ), ... , s(v, t _N ), ₀ (v, P), o(v, t ₂ ), ... , o(v, t _N ),

°(P 1 ₂ ) °(P £ ₃ ) _' °(hv- _1> hv)]·

Here, s(a,b) and o(a,b) are the score (number of inliers) and overlap between images a and b respectively. The enrol images ti are pre-sorted such that is the image with the highest number of inliers 34 to the verify image v. N is the amount of enrol images 32 we consider for one verify attempt.

In case of a plurality of enrol images 32, a score (e.g. probability or inlier count) may be obtained for each enrol image and the inlier number for classification maybe computed out of these scores. For instance, the highest observed score may be used as the final score in a winner-takes-it-all fashion. It may also be suitable to look at other candidates than the enrol image with the highest score. For example, if the image with the highest score is just below the classification threshold, we can examine the image with the second highest score. If this score is also close to the classification threshold, there may be a stronger evidence of the verify image being genuine. As discussed herein, especially in cases of a low number of inliers, e.g. close to the classification threshold, the number of inliers 34 (also called score or count) may not be enough to correctly classify the verify image as genuine or impostor. Rather, the additional property of area of overlap 33 is also considered.

User or manufacturer defined thresholds for the number of inliers 34 and the overlap 33 may be used, but it may be more preferable to use machine learning, e.g. a neural network model, e.g. a Multilayer Perception (MLP) model, a Convolutional Neural Network (CNN) model or a Support Vector Machine(SVM) model to determine a decision boundary 51 as shown in figure 5 for use in the classifying.

If an MLP classifier is trained to find the optimal decision boundary 51, one feature may represent a sub template match and is formed by concatenating the number of inliers 34 and the overlap 33: x = [inliers overlap].

In figure 5, the decision boundary 51 obtained by means of an MLP model is shown as a slanted line in a 2-dimensional coordinate system with number of inliers and the overlap as the two dimensions. The classifying may thus comprise determining a point in the two-dimensional coordinate system where the number of inlying matches 34 corresponds to a first coordinate of the point and the area of overlap 33 corresponds to a second coordinate of the point. The classification may then be done depending on on which side 52 or 53 of the predetermined decision boundary 51 in the coordinate system the point is located according to its first and second coordinates. In the example of figure 5, if the point is located in the upper right area 52 of the coordinate system, the verify image is classified as genuine, and if the point is located in the lower and left area 53 of the coordinate system, the verify image is classified as impostor. As can be seen, as the number of inliers 34 increases a larger range of the overlap is accepted for classification as genuine, while for a decreasing number of inliers the range of overlap accepted is narrowed since for a genuine classification we expect only a relatively small overlap if the number on inliers is smaller.

Figure 6 is a flow chart illustrating different embodiments of the method of the present invention. The method may be performed in an electronic device l comprising a fingerprint sensor 2 for determining whether an obtained verify fingerprint image 31 is genuine or impostor compared with an enrolled fingerprint. The method comprises, by means of the fingerprint sensor, e.g. by using the image obtaining circuitry 12, obtaining Si the verify fingerprint image of a finger 4 in proximity to a detection surface of the fingerprint sensor. The method also comprises, by means of image processing, e.g. by using the image processing circuitry 13, extracting S2 features from the verify fingerprint image. The method also comprises, e.g. by means of using the image processing circuitry 13, retrieving S3 at least one stored (e.g. in the storage 14) template 16 of features of at least one enrol image 32 of the enrolled fingerprint. The method also comprises, e.g. by means of using the image processing circuitry 13, matching S4 the extracted S2 features to a geometrical transform of the features of the retrieved S3 template, wherein positive matches between the extracted features and the template features are determined to be inlying matches 34. The method also comprises, e.g. by means of using the image processing circuitry 13, determine S5 an area of overlap 33 between the verify fingerprint image and the at least one enrol image based on the geometrical transform of the template features. The method also comprises, classifying S9 the verify fingerprint image as genuine or impostor based on the number of the inlying matches in relation to the determined overlap area.

In some embodiments of the present invention, the classifying S9 comprises determining whether the ratio of the number of inlying matches 34 to the area of overlap 33 is above a predetermined classification threshold.

In some embodiments of the present invention, the method further comprises, before the classifying S9, determining S6 a decision boundary 51 by means of a machine learning model, e.g. a neural network model, e.g. an MLP model, a CNN model or an SVM model, which decision boundary is used in the classifying S9. In some embodiments, the classifying S9 comprises determining a point in a two-dimensional coordinate system where the number of inlying matches 34 corresponds to a first coordinate of the point and the area of overlap 33 corresponds to a second coordinate of the point, and determining on which side 52 or 53 of the predetermined decision boundary 51 in the coordinate system the point is located according to its first and second coordinates.

In some embodiments of the present invention, the at least one stored template 16 comprises a single template based on a plurality of the at least one enrol images 32.

In some other embodiments of the present invention, the at least one stored template 16 comprises a single template based on a stitched imaged made up of a plurality of the at least one enrol images 32 stitched together to form the stitched imaged.

In some other embodiments of the present invention, the at least one stored template 16 comprises one template, which could also be called sub-template, for each of a plurality of the at least one enrol images 32.

In some embodiments of the present invention, the method further comprises, prior to the classifying S9, determining S7 that the number of the inlying matches 34 is above a predetermined first threshold, indicating that the verify fingerprint is potentially genuine.

In some embodiments of the present invention, the method further comprises, prior to the classifying S9, determining S8 that the number of the inlying matches 34 is below a predetermined second threshold, indicating that the verify fingerprint is potentially impostor.

In some embodiments of the present invention, the at least one stored template 16 is retrieved from a data storage 14 comprised in the electronic device 1.

Instructions, typically as SW 15 stored in a storage 14, in the form of computer-executable components in a non-transitory computer readable medium (e.g. the storage 14), forming a computer program product configured for enabling a device, e.g. the electronic device 1 and/ or the system 10, to perform embodiments of the present method as discussed above, when the instructions 15 are run on processing circuitry 11 in the device.

Thus, a computer program product 14 comprising computer-executable components 15 for causing an electronic device 1 to perform an embodiment of the method disclosed herein when the computer-executable components are run on processing circuitry 11 comprised in the electronic device.

The fingerprint sensing system 10 may thus comprise a fingerprint sensor 2, processing circuitry 11, and data storage 14 storing instructions 15 executable by said processing circuitry whereby said fingerprint sensing system is operative to, by means of the fingerprint sensor, obtain Si a verify fingerprint image 31 of a finger 4 in proximity to a detection surface of the fingerprint sensor; by means of image processing, extract S2 features from the verify fingerprint image; retrieve S3 at least one stored template 16 of features of at least one enrol image 32 of an enrolled fingerprint; match S4 the extracted features to a geometrical transform of the features of the template, wherein positive matches 34 between the extracted features and the template features are determined to be inlying matches; determine S5 an area of overlap 33 between the verify fingerprint image 31 and the at least one enrol image 32 based on the geometrical transform of the template features; and classify S9 the verify fingerprint image as genuine or impostor based on the number of the inlying matches 34 in relation to the determined overlap area. In some embodiments, the electronic device 1 discussed herein comprises an embodiment of the fingerprint sensing system 10.

The present disclosure has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the present disclosure, as defined by the appended claims.