AND AUTOMATIC TEMPLATE UPDATING USING FUSION OF MEDICAL AND NON-MEDICAL BIOMETRICS

FIELD OF THE INVENTION

[0001] The present specification relates to identifying a user, and more specifically to the field of medical and non-medical biometric sensing and processing.

BACKGROUND OF THE INVENTION

[0002] Biometric security systems have been deployed around the world and have been extensively used in past decades. Identity verification using biometric-based security systems is widely used in many fields of life, including financial transactions and tele-medicine. These systems are often used for access control wherever an aim is to restrict access to certain users.

[0003] However, it is widely admitted that this technology can be spoofed. Sensor level spoofing attacks seek to fool a sensor into providing a false positive determination. For example, biometrics may be stolen and copied; fingerprints may be left behind whenever we touch a glass surface like a phone screen, images of faces may be captured from a great distance, and images of iris may also be captured from a distance. From such samples, reproductions or otherwise fake spoof samples may be made, such as digital files or physical reproductions using gelatin, clay, playdoh, silicone, latex, or rubber. Such spoof samples can often be used to fool biometric security systems.

[0004] Medical biometrics such as electrocardiogram (‘ECG’) are among the newer additions to the biometric family, and unlike conventional biometrics such as fingerprint, iris, face, and other non-medical biometrics, ECG and other medical biometrics are vital signals and indicate liveness. In addition, it may be more difficult to steal a person’s medical biometrics, such as their ECG signature, than to copy their fingerprint, face, or iris. However, the accuracy of ECG biometric security and other medical biometric security may not be as high as more mature biometric security measures, such as fingerprint biometric security. [0005] Some combinations of ECG biometrics and conventional biometrics improve the recognition rate of a fused system but result in greater spoofing risk, as such systems recognize a user when the user passes either the ECG or the conventional biometric test. Such systems may accept an accurate reproduction of a fingerprint, even if the ECG signal does not match a stored record.

[0006] One area of improvement in biometric security is the introduction of liveness detection measures. For example, the main stream of the current methods is basically binary classifier; that given an image of for example fingerprint, classify it as fake or live based on analyzing, for example, its texture. Although they perform well on fake samples produced via certain spoofing methods and materials seen during the training phase, their performance on novel spoofing methods and materials are questionable.

[0007] Spoofing of biometric security systems continues to be an issue, and practical biometric security system spoofing and countermeasures continues to be an area of active interest.

SUMMARY OF THE INVENTION

[0008] Embodiments of the present invention fuse medical biometrics, such as ECG, and more mature biometrics, particularly non-medical biometrics such as fingerprint, for liveness detection in a fused system. Embodiments of the present invention also use fused biometrics for identity verification in a fused system.

[0009] Embodiments of the present invention are capable of automatically adapting biometric templates to operational data. As a result, template updating may be used to maintain the performance of the system in the long term without requiring a user to re-enroll or retrain the system from scratch.

[0010] Embodiments of the present invention include a stopping criterion to limit the length of recording sessions for certain biometrics. The stopping criterion takes into account the fact that some subjects have very repetitive biometric signals, such as their ECG signal, which can be recognized based on fewer heartbeats than that of subjects with less repetitive biometric signals. [0011] According to an aspect of the present invention, there is provided a system comprising: at least one non-medical biometric capturing device; at least one medical biometric capturing device; and a processing unit in electronic communication with the at least one non-medical biometric capturing device and the at least one medical biometric capturing device, wherein the processing unit is configured to analyze a set of signals received from the at least one nonmedical biometric capturing device to produce a non-medical biometric liveness score, configured to analyze a set of signals received from the at least one medical biometric capturing device against a medical biometric template to produce a medical biometric recognition score, combine the non-medical biometric liveness score and the medical biometric recognition score into a fused liveness score, and analyze the set of signals received from the at least one nonmedical biometric capturing device against a non-medical biometric recognition template to produce a non-medical biometric recognition score.

[0012] According to a further aspect of the system, the processing unit is further configured to fuse the non-medical biometric recognition score and the medical biometric recognition score to provide a fused biometric recognition score.

[0013] According to a further aspect of the present invention, there is provided a method of verifying the identity of a user, comprising: receiving a claimed identity from the user; receiving at least one non-medical biometric input from the user; producing a non-medical biometric recognition score by comparing the at least one non-medical biometric input to a non-medical biometric recognition template; producing a non-medical biometric liveness detection score by comparing the non-medical biometric input to a liveness detection model; receiving a medical biometric input from the user; producing a medical biometric recognition score by comparing the medical biometric input to a medical biometric template; fusing the non-medical biometric liveness detection score and the medical biometric recognition score to provide a fused liveness score; producing a liveness determination by comparing the fused liveness score to a liveness threshold; and producing a biometric recognition determination by comparing the non-medical biometric recognition score to a non-medical biometric recognition threshold.

[0014] According to a further aspect of the method, the step of producing a biometric recognition determination is replaced by: fusing the non-medical biometric recognition score and the medical biometric recognition score to provide a fused biometric recognition score; and producing a biometric recognition determination by comparing the fused biometric recognition score to a biometric recognition threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Reference will now be made to the accompanying drawings which show, by way of example only, embodiments and aspects of the invention, and in which:

[0016] FIG. l is a schematic diagram of a security system according to an embodiment;

[0017] FIG. 2 is a schematic diagram of an ECG signal collection apparatus according to an embodiment;

[0018] FIG. 3 is a schematic diagram of the front and back faces of a smartphone according to an embodiment;

[0019] FIG. 4 is a representation of experimental results of an embodiment;

[0020] FIG. 5 is a representation of experimental results of an embodiment;

[0021] FIG. 6 is a representation of experimental results of an embodiment;

[0022] FIG. 7 is a representation of experimental results of an embodiment;

[0023] FIG. 8 is a representation of experimental results of an embodiment;

[0024] FIG. 9 is a representation of experimental results of an embodiment;

[0025] FIG. 10 is a representation of experimental results of an embodiment;

[0026] FIG. 11 shows a smartphone embodiment of the present invention;

[0027] FIG. 12 shows a further smartphone embodiment of the present invention;

[0028] FIGs. 13 A and 13B show a further smartphone embodiment of the present invention; [0029] FIGs. 14A and 14B show a further smartphone embodiment of the present invention; [0030] FIGs. 15 A, 15B, 15C, and 15D show a padlock embodiment of the present invention;

[0031] FIG. 16 shows an embodiment of the present invention with a PPG

(photoplethysmogram) sensor;

[0032] FIG. 17 shows an embodiment of the present invention using a TEOAE (Transient Evoked Otoacoustic Emissions) sensor;

[0033] FIG. 18 shows an embodiment of the present invention using an EMG

(electromyogram) sensor; and

[0034] FIG. 19 shows a door handle embodiment of the present invention,

[0035] and in which like reference numerals indicate like or corresponding elements in the drawings.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Embodiments of the present invention fuse at least one medical biometric such as ECG (but not limited to, as many medical biometric signals such as electrocardiogram (ECG), photoplethysmogram (PPG), phonocardiogram (PCG), Transient Evoked Otoacoustic Emissions (TEOAE), Transient Auditory Evoked Potentials (AEPs), electromyogram (EMG),

electroencephalogram (EEG), could be used to indicate user identity) and at least one nonmedical biometric such as fingerprint (but not limited to, as many features that are unique to a user such as fingerprints, irises, ears, faces, etc. can be used to provide identification of the user) for liveness detection. Embodiments of the present invention also fuse at least one medical biometric such as ECG (but not limited to) and at least one non-medical biometric such as fingerprint (but not limited to) for biometric recognition. The at least one non-medical biometric may be captured by at least one non-medical biometric capturing device and the at least one medical biometric may be captured by at least one medical biometric capturing device.

[0037] In some embodiments, templates are used for medical biometric recognition and nonmedical biometric recognition, while non-medical biometric liveness detection is determined otherwise; for example, non-medical biometric liveness detection may be via a threshold determination, such as the presence of perspiration. [0038] Embodiments of the present invention relate to medical biometrics, of which ECG biometrics is generally used as an example herein, and non-medical biometrics, of which fingerprint biometrics is generally used as an example herein.

[0039] As depicted in FIG. 1, an exemplary embodiment security system 1000 collects fingerprint biometrics 1100 from a user and collects ECG biometrics 1200 from the user. The user may be an authorized user of a system incorporating embodiment 1000, such as an authorized user applying their own fingers to fingerprint sensors, or the user may be seeking to spoof security system 1000, such as an unauthorized user applying a synthetic fingerprint to fingerprint sensors.

[0040] The present invention may be used for a variety of security requirements. For example, in some embodiments, security system 1000 may be incorporated into a personal device such as a smartphone or smartwatch, or a combination of personal devices, and be used to secure one or more personal devices. In other embodiments, security system 1000 may be used to secure a chamber or vessel or room, or may be used to secure a door or accessway to a chamber or vessel or room, wherein authentication of a user using security system 1000 may grant access to the chamber or vessel or room. In a broader view, it can be used in any application that fingerprint biometric may be used by adding another layer of security against spoofing attacks.

[0041] In some embodiments, ECG biometric sensors are incorporated into, or placed on or adjacent to, fingerprint biometric sensors, such that a user can provide ECG and fingerprint information at the same time using the same sensors. The fingerprint sensor may incorporate an electrode, potentially by a mesh of transparent conductive materials, which may sample a user’s ECG signal as the user places a finger from a first hand on the fingerprint sensor while there is a second contact point from the opposite side of the body. For instance, one electrode could be a ring around a Home button on a smartphone while substantially the entire back surface of the smartphone may be a second electrode provided to be contacted by a part of the users second hand.

[0042] The use of fingertips to collect ECG signals may eliminate the need for a user to undress for electrode placement, as in a clinical setup for recording ECG from chest and may make fingerprint biometrics a suitable conventional biometric to be fused with ECG biometrics. [0043] Security system 1000 uses fingerprint biometrics 1100 to produce a fingerprint liveness detection or FpLD 1110 and a fingerprint recognition FpR 1120. Security system 1000 uses ECG biometrics 1200 to produce an ECG recognition 1210. A processing unit is in electronic communication with the at least one non-medical capturing device and the at least one medical biometric capturing device.

[0044] In another embodiment, in the security system 1000, fingerprint 1100 could be any nonmedical biometric such as fingerprint and face (but not limited to) or any combination of them which may be captured by a non-medical biometric capturing device.

[0045] Security system 1000 fuses fingerprint liveness detection 1110 and ECG recognition 1210 in a multimodal liveness detection or MmLD 1300. Security system 1000 then produces a liveness detection determination 1310. If liveness detection determination 1310 is negative, security system 1000 denies the user access 1320. If liveness detection determination 1310 is positive, security system 1000 moves on to recognition.

[0046] In the embodiment depicted in FIG. 1, security system 1000 includes a multimodal human recognition 1400. Multimodal human recognition or MmHR 1400 provides a recognition determination 1410 based on a fusion of ECG recognition 1210 and fingerprint recognition 1120. In other embodiments, a security system may produce a recognition determination using only conventional biometrics, such as fingerprint biometrics, without fusing ECG biometrics.

[0047] If recognition determination 1410 is negative, security system 1000 denies the user access 1420. If recognition determination 1410 is positive, security system 1000 accepts the user 1500 and grants access to the user.

[0048] Liveness detection determination 1310 and recognition determination 1410 may be system blocks. Multimodal liveness detection MmLD 1300, multimodal human recognition MmHR 1400, liveness detection determination 1310 and recognition determination 1410 may be performed by one or more computer systems, such as a processor unit executing computer readable instructions. [0049] Embodiments of the present invention are able to collect biometric signatures from users to form biometric templates to use as references, such as references to be used by security system 1000 in ECG recognition 1210 and fingerprint recognition 1120.

[0050] ECG recognition in the present invention, such as ECG recognition 1210, may be fiducial based or non-fiducial based.

[0051] Fiducial based ECG recognition relies on points of a heartbeat, such as onset and end of each wave, and requires P, R, and T waves to be located and features computed, such as peaks, slope, radius of curvature, and area to be computed in a region surrounding each of P, R, and T waves.

[0052] However, detection of such characteristic points may not always be possible, particularly in a relatively high noise situation. As a result, ECG recognition in the present invention may instead, or in combination, be non-fiducial based; considering an ECG signal holistically as a set of heartbeats or a time series without segmenting it into heartbeats. In embodiments, signals are segmenting in overlapping windows and autocorrelation features extracted and linear discriminant analysis (LDA) is used for dimension reduction. In

embodiments, short time Fourier transform (STFT) features are extracted and after a feature’s selection, log-likelihood ratio is used for classification. In embodiments, sparse coefficients of an over-complete dictionary are used as features. In embodiments, max-pooling is used for aggregation of samples to construct templates for comparison.

[0053] In some embodiments, a subset of the most stable features is selected. Such features may be less dependent on outside factors such as diet and emotion. These features may be used to form templates or classifiers of users of a biometric system, such as security system 1000.

[0054] ECG signals may be collected from a user for both security system set up and template building and for identity verification. In embodiments, ECG signals are recorded using at least two electrodes from fingertips as depicted in FIG. 2.

[0055] As depicted in FIG. 2, a negative electrode 2100 may be contacted by a fingertip 2110 on a first hand 2120 and a positive electrode 2200 may be contacted by a fingertip 2210 on a second hand 2220. In some embodiments, the positive and negative electrodes 2100 and 2200 may be provided on a common physical object 2300, which may be a personal device such as a smartphone or smartwatch, and may be part of a security system for the common physical object. A grounding or reference electrode 2400 also contacts the common physical object 2300.

Reference electrode 2400 may be used to reduce noise in an ECG signal.

[0056] In some embodiments, ECG electrodes and fingerprint sensors are combined. For example, as depicted in FIG. 3, a positive electrode may be incorporated into a first fingerprint detector 3100 and a negative electrode may be incorporated into a second fingerprint detector 3200. First and second fingerprint detectors 3100 and 3200 may be provided in a common physical object, such as a smartphone 3300. A reference electrode (not shown) may also be provided in smartphone 3300. First and second fingerprint detectors 3100 and 3200 may be provided on front face 3310 of smartphone 3300. Fingerprint detectors 3100 and 3200 may be separated from one another so as to be conveniently contacted by a finger of different hands of a user. In other embodiments one or more electrodes, fingerprint sensors, or combination sensors may be provided on other surfaces of smartphone 3300, such as on the back face 3320 of smartphone 3300. In other embodiments, fingerprint sensors may be built into the screen 3330 of smartphone 3300, and electrodes may also be built into the screen 3330 of smartphone 3300.

[0057] Fingerprint liveness detection in the present invention, such as fingerprint liveness detection 1110, may be hardware-based or software-based or a combination. Hardware-based techniques include specific components in a fingerprint sensor or other capture device to detect some evidence of liveness such as temperature, moisture or ECG pulses. Software-based techniques process an obtained fingerprint scan or image.

[0058] In embodiments, morphology -based and perspiration-based techniques may be jointly used for fingerprint liveness detection. In embodiments, pore perspiration may be used for fingerprint liveness detection, with the difference in quality of genuine and spoof samples, such as measured by a number of image quality measures, used to detect spoof samples. In

embodiments, local descriptors may be applied for fingerprint liveness detection. In some embodiments, local binary pattern (LBP) and its extensions are used for liveness detection, where LBP is a texture descriptor based on intensity difference between a pixel and neighboring pixels. In some embodiments, local phase quantization (LPQ) is used for liveness detection, where LPQ works on patches of pixels but instead of gradient it computes phase information by computing short time Fourier transform, phase information is then decorrelated and uniformly quantized. In some embodiments, weber local descriptor (WLD) is used for fingerprint liveness detection. In some embodiments, binarized statistical image features (BSIF) is used for fingerprint liveness detection, where BSIF is based on binarizing the response to a set of filters, including where filters are not fixed but are learned using independent component analysis (ICA). In some embodiments, local contrast phase descriptor (LCPD) is used for fingerprint liveness detection, which takes into account a spatial-domain component inspired by WLD and a phase component inspired by LPQ. Additional embodiments use deep neural networks for fingerprint liveness detection, where features are learned from training data, including where pre- trained deep networks are used to reduce the volume of training data required to train such networks.

[0059] In some embodiments, only one type of feature may be used to represent ECG signals in forming templates and characterizing ECG signals as spoof or genuine attempts. In other embodiments, different types of features may be concatenated to construct a more

comprehensive feature vector. Various embodiments employ a short-term Fourier transform with Hamming window of length 16 with step size of 13 computed over a 1 -second window centered at R peak, while mean of power, standard deviation of power, maximum amplitude, standard deviation of amplitude, kurtosis and skewness are computed on a 2-second window centered at R peak on the 8-13 Hz, 13-18Hz, 18-25Hz, 25-30Hz, 30-35Hz, and 35-50Hz frequency bands. Some embodiments also consider the signal’s amplitude itself. There may be up to 7198 features to choose from in forming a feature vector. Embodiments also use z-score normalization for every feature, so that features have zero mean and unit variance after normalization.

[0060] In selecting which features to use in forming templates and for classification of ECG signals, an auxiliary dataset of some generic subjects may be used where for every subject recordings of multiple sessions are available. Having a number of features, they may be analyzed to select features having weights over a determined threshold.

[0061] For example, where is the auxiliary dataset consisting of A

subjects, and there are Mi different sessions available for the i-th subject i.

consists of samples of j- th session of /-th subject. The weight of

the /-th feature consists of two terms. The first term w 1 (/) encourages class separability and is defined as follows: where d{ ) is the symmetric Kullback-Leibler divergence and /(·) denotes probability density function is pdf of /-th feature computed over all samples of i- th subject

and is pdf of /-th feature computed over all samples. We consider normal distribution

and use maximum likelihood estimates, i.e. sample means and variances to estimate the symmetric Kullback-Leibler divergence as follows:

where are two distributions whose distance is to be computed, wl is large when the overall distribution of every subject is different from the overall distribution of all subjects. However, wl only considers overall class distribution of a subject and disregards the session distributions of that subject. To address this limitation, we define the second term, w2, as follows:

The first summation is over all subjects and the second summation is over all sessions and denominator is a normalization factor. w2 encourages stability across multiple sessions and is smaller when a session can represent the actual distribution of the corresponding subject.

Finally, considering both wl and w2, weight of the /-th feature, w(/), is defined as follows: where l is a parameter that controls the trade-off between class separability and across session stability. Features can be selected by comparing their weights with a threshold.

Alternatively, features can also be sorted by weight, with the top features selected. A proper value for l can be determined using cross validation.

[0062] Embodiments of the present invention use a predefined ECG signal recording length. Other embodiments of the present invention include a stopping criterion for ECG recording sessions. Some users have a very repetitive ECG signal and may not need as many heartbeats as users with a less repetitive ECG signal. For example, heartbeat consistency (HC) may be defined and used as a stopping criterion as follows:

where corr is correlation function and „is n ^th heartbeat amplitude. In some other

embodiments, HC may include more or less heartbeats by taking the correlation of the average of b _n to _z and the average of to where z can take any value greater than or equal to 1. An ECG recording session may continue until HC(n) exceeds a predefined threshold.

Embodiments may also include hybrid stopping criteria, which includes a dynamic stopping criterion, such as HC, but also includes an upper limit, such as an upper limit of 30 seconds.

[0063] Easing a dynamic stopping criterion, such as HC, may be more convenient than a fixed criterion. A stable biometric signal, such as a stable ECG signal, is recognized more quickly, while a diverse ECG is sampled more extensively which may result in a more reliable decision.

[0064] In some embodiments, a minimum number of heartbeats is needed for analysis, for example in some embodiments a minimum of 4 heartbeats is needed, as in equation (5), and so a lower limit of 4 heartbeats may be used. In some embodiments, 4 heartbeats are needed to perform averaging and correlation, used to reduce noise and make correlation results more reliable. In some embodiments, any R peak appeared during the first second of recording cannot be used because the features measured require a 2-second window centered at R peak. As a result, in some embodiments, a minimum recoding time of 5 seconds may be set.

[0065] In some embodiments, a linear support vector machine (SVM) is used for ECG classification. SVM is a binary classifier, therefore in some embodiments, a one-versus-all strategy may be adopted. In some embodiments the score corresponding to a session (s) is determined as weighted sum of its heartbeats scores

Where the length of sessions is determined by HC and K is not fixed and the summation is over all heartbeats of each session. The denominator is a normalization factor and weights w _t are defined as:

Where Si denotes score of i-th heartbeat and m and s are mean and standard deviation of the scores in the corresponding session. The final decision for a session can be made by comparing its score against an acceptance threshold. Such definition of weights (after normalization) can be interpreted as probabilities assigned to heartbeats where higher probabilities are assigned to the heartbeats with larger scores. This makes the final decision less sensitive to outliers as outliers are artifacts that do not look like a regular heartbeat and hence usually do not match templates and their weights in (6) are small. Heartbeats with a very small HC value are usually very irregular due to muscle or electrode movements. Therefore, heartbeats with HC value below 0.5 are considered outliers and discarded. Note that this does not require further computation because HC values for the heartbeats that precede the stopping point are already computed. This method of removal of outlier heartbeats may be especially helpful as, even if the outlier removal module fails to detect a few artifacts, the effect of those artifacts will be mitigated by the weighting strategy in (6). [0066] In other embodiments, other outlier heartbeat removal methods may be used. For example, the DMEAN and DBSCAN methods, where the DMEAN method detects outliers based on their distance to mean template (i.e. mean of all heartbeats) and requires computing statistics including mean, standard deviation, min and max over entire session, while the

DBSCAN method is based on clustering all samples of a session and detects samples not belonging to any cluster as outlier.

[0067] In some embodiments, an enrollment session is necessary, in which a user provides an ECG signal to an embodiment of the invention for the construction of a template. The enrollment session may include a minimum of 22 heartbeats, setting the minimum length of an enrollment session to approximately 20 seconds. In some embodiments, obtaining 22 heartbeats or more may be achieved by setting HC to a high threshold to increase recording time and lower Equal Error Rate (EER) of the ECG recognition. Generally, increased recording time lowers the EER and improves the performance of ECG recognition.

[0068] Once an ECG template is obtained, embodiments of the present invention, such as security system 1000, may use an ECG template to determine ECG recognition, such as ECG recognition 1210.

[0069] Fusion of fingerprint liveness detection and ECG recognition to produce a fused liveness detection, such as using multimodal liveness detection 1300 to produce a liveness detection determination 1310, may use any of a variety of fusion methods. ECG recognition is used as an independent indicator of liveness because the presence of an authentic ECG signal inherently implies liveness.

[0070] ECG recognition may be fused with any fingerprint liveness detection method or system. For example, it may be fused with local contrast phase descriptor (LCPD), binarized statistical image features (BSIF), image quality assessment (IQ A), local binary pattern (LBP), local phase quality (LPQ), or Weber local descriptor (WLD) fingerprint liveness detection methods.

[0071] As an example, the fusion of ECG recognition and fingerprint liveness detection may be a fusion of ECG recognition as described above, and LCPD liveness detection. Fusion of ECG recognition and fingerprint liveness detection may be in any of a variety of ways, with different fusion methods offering advantages. For example, fusion may be realized in the score level through weighted sum, product, or maximum rules. In the sum rule, sum of the individual scores is used as the fused score. The product rule takes the product of individual scores to generate the fused score. In the maximum rule, the fused score is the maximum value among the individual scores. In an embodiment, a weighted sum rule may be used where weights may be determined using an evaluation set. In other embodiments, weights may be chosen proportional to the EER of the components, such as the EER of ECG recognition and the EER of LCPD liveness detection.

[0072] In some embodiments biometric recognition is monomodal, relying only on a conventional biometric, such as fingerprint recognition, or only on ECG recognition. However, in other embodiments biometric recognition is multimodal, and a fusion of ECG recognition and conventional biometrics, such as fingerprint recognition.

[0073] In embodiments, fingerprint recognition may be achieved using known methods. For example, embodiments may use fingerprint recognition software from NIST Biometric Image Software (NBIS). NBIS detects minutiae using the MINDTCT algorithm and computes fingerprint matching scores using the Bozorth3 matcher. MINDTCT performs image binarization, minutiae detection, false minutiae removal, neighbor ridges counting and minutiae quality assessment, and generates a list consists of location, orientation, type and quality of the detected minutiae to be used by Bozorth3. Bozorth3 is a minutiae-based fingerprint matching method invariant to rotation and translation. It considers the location and orientation of the top 150 high quality minutiae and computes the matching score.

[0074] For multimodal human recognition 1400, different methods of fusing fingerprint recognition and ECG recognition may provide different benefits. For example, fusion rules may include sum, product, and maximum rules, similar to the fusion of ECG recognition and fingerprint liveness detection in MmLD 1300. In embodiments, a weighted sum rule may be used, and the EER or other performance indicators may be used to set the weights of the components. [0075] In some embodiments, ECG templates may be automatically updated while being used. Automatically updating the ECG templates may allow a system to adapt to the intra-class variation of ECG over different sessions. As a result, the biometric system can adapt to temporal variations of ECG signal across different sessions. If the system ensures the liveness and genuineness in a trial, then the provided ECG and fingerprint samples can be safely added to the corresponding templates.

[0076] In some embodiments, to prevent updating of ECG templates using impostor samples, zero false acceptance rate for both liveness detection and human recognition tasks may be used. As a first example, the liveness fusion score, and the recognition fusion score, S _MmHR , can be compared against updating thresholds:

Where are thresholds corresponding to zero false acceptance rate operating point of MmLD and MmHR blocks, respectively and‘&’ is logical AND operator. Those subjects that pass the MmLD block are fed to the MmHR block and those that pass that block too are selected as candidate for equation (8).

[0077] As a second example of ECG template updating criteria, each of the ECG, FpLD and FpR blocks may be set to zero false acceptance operating point. Where ECG and fingerprint detection are used in parallel in the fusion blocks, MmLD and MmHR, the following criterion could be used to achieve the overall zero false acceptance rate:

Where S and are the scores and _D and are the updating thresholds

corresponding to zero false acceptance rate operating point of ECG, FpLD and FpR blocks, respectively,‘|’ is the logical OR operator, and‘&’ is the logical AND operator. Queries that satisfy the selected updating criterion may be selected for template updating.

[0078] Updating a multimodal template includes updating a fingerprint template and an ECG template. In order to update an ECG template, the new ECG samples may be added to the previously existing training samples of that subject and re-train the SVM model. In order to update a fingerprint template, the new fingerprint sample may be added to the set of previously existing samples of that subject. As Bozorth is a matcher, if a fingerprint template contains more than one sample, it may compute the match scores against each of the samples in the template and consider the average score as the matching score assigned to that trial.

[0079] Template updating using a multimodal system may be particularly practical. Automatic template updating using a single biometric may tend to add only those samples that are very similar to the existing samples. However, in a multimodal system there is potential to add more diverse samples. For example, in some embodiments, if the fingerprint sample can be determined to be live and genuine with high confidence, both ECG and fingerprint templates may be updated. Likewise, in some embodiments, if the ECG sample can be recognized with high confidence, both ECG and fingerprint templates may be updated.

[0080] Embodiments of the present invention were tested with several datasets.

[0081] Samples for an ECG database were collected at a lab at the University of Toronto. ECG signals were recorded using Vernier EKG sensor and Go!Link interface with l2-bit resolution and sampling rate of 200 Hz, as indicated in FIG. 2. 82 subjects had 2 or more ECG recordings, in sitting posture, while 46 of the 82 subjects had 5 sessions. Follow-up sessions were collected over a six-month period. The database was divided into 2 parts, the 46 subjects were used for enrollment and testing (i.e. testing dataset) and the remaining 36 subjects were used as an auxiliary dataset for feature selection.

[0082] Fingerprint data was taken from LivDet20l5 fingerprint databases, specifically the DigitalPersona, GreenBit, and Biometrika databases, as indicated in Table I.

[0083] The three LivDet20l5 databases contain testing of fingerprint liveness detection methods against live and fake fingerprint samples, where fake samples are constructed of 6 different spoof materials including Ecoflex, Gelatine, Latex, WoodGlue, Liquid Ecoflex and RTV. Each dataset originally included two sets: training and testing. The training set is the same as the testing set but does not include Liquid Ecoflex and RTV spoofs. In the LivDet20l5 databases, one experiment is performed for each of the six spoofs samples listed above, such that samples of that spoof are omitted from the training set. Thus, the type of spoof is always unknown in all 6 experiments of each scanner.

[0084] A chimeric dataset was constructed by combining the above ECG and fingerprint datasets. Each of the 46 subjects in the ECG dataset was randomly paired with a subject in the fingerprint dataset to generate a chimeric dataset of 46 unique subjects. Each finger was considered a subject. For each subject 4 live and 3 fake sessions were used: 1 live session for enrollment and 3 live and 3 fake sessions for subsequent test sessions. All experiments were repeated 50 times to cope with the randomness in constructing the chimeric dataset, and average results were used.

[0085] For the ECG dataset, FIG. 4 depicts the average recording time versus HC, taken from the testing dataset. As shown in FIG. 4, increasing HC threshold increases the average recording time as a larger HC threshold is harder to satisfy.

[0086] FIG. 5 depicts recording times for the testing dataset where HC was set to 0.92, with data for 46 subjects and each data point for each subject (subject index) representing one recording session. As depicted in FIG. 4, the average recording time was approximately 10 seconds, while for some users the upper limit of 30 seconds was reached. However, as shown in FIG. 5, when the average recording time is about 10 seconds (i.e. HC=0.92), the majority of the cases (i.e. about 60%) experience a shorter acquisition time of about 5 seconds; the use of dynamic HC stopping criteria has been able to cut the acquisition time in half for a majority of the subjects.

[0087] As depicted in FIG. 6, the use of a variable length recording time, such as HC, results in a significantly better Equal Error Rate (EER) across a variety of recording lengths. The use of a stopping criterion may reduce the recording time needed to achieve a desired EER. [0088] Recording beyond 30 seconds is of little value in a real-world scenario. However, increasing the maximum recording time to 1 -minute leads to an average EER of 3.8% and further increasing the maximum recording time to 2 minutes does not significantly improve

performance. Results in FIG. 6 are computed over the 46 subjects.

[0089] Performance of these above-mentioned six fingerprint liveness detection methods, namely LCPD, BSIF, IQ A, LBP, LPQ, and WLD, was compared to performance of a fused approach using the three datasets taken from the LivDet20l5 data, namely the DigitalPersona, GreenBit, and Biometrika, each collected using a different scanner.

[0090] As indicated in Tables II, III, and IV, each corresponding to a different dataset of the LivDet20l5 database. Of the six-example fingerprint liveness detection methods, LCPD outperforms the others, on average. Therefore, it may be preferred for use with fingerprint liveness detection FpLD 1110.

[0091] Fusing ECG recognition and LCPD liveness detection may be achieved by any of a variety of ways, with different fusion methods offering different advantages. For example, as shown in Tables II, III, and IV, below, fusion may be realized through weighted sum, product, or maximum rules.

[0092] In an embodiment, a weighted sum rule may use weights determined using an evaluation set. However, in other embodiments, weights may be chosen proportional to the EER of the individual traits. For example, as EER of ECG is roughly half of EER of LCPD, the weight of ECG and LCPD may be set to 2/3 and 1/3 respectively.

[0093] As indicated in Tables II, III, and IV, below, the sum rule may outperform other fusion rules.

[0094] Results in Table II, III and IV are computed over the 46 chimeric subjects discussed above, for each of which there are samples for 3 test sessions. Therefore, there are 46 ^c 3 positive trials and 46 ^c 3 negative trials (spoof trials) in each of 50 runs. Note that since there are more subjects in the fingerprint dataset than ECG dataset, we randomly picked 46 fingerprint subjects in each run and report the average and standard deviation values computed over 50 runs.

[0095] Performance of tested embodiments was also compared to the performance of participants of the LivDet20l5 competition. As mentioned, the LivDet20l5 test set includes 6 different spoofs, of which LiquidEcoflex and RTV are not in the training set. Since tested embodiments of the present invention were trained with a training set similar to the training set used in the LivDet20l5 competition, they were also tested on LiquidEcoflex and RTV. This is the same as the protocol used in the LiveDet20l5 competition for unknown spoofs, except that instead of using all test samples of LiquidEcoflex and RTV (i.e. 500 fake and 1000 live samples) at once, we use 276 spoof and 276 live samples (i.e. 46 ^c 3 ^c 2) in each run. This corresponds to 46 subjects, 3 samples and 2 spoofs. Note that since the experiments are repeated 50 times, all samples are involved in the experiments.

[0096] Table V shows the Half Total Error Rate (HTER) for participants of the LivDet2015 competition on the aforementioned unknown spoofs as well as EER of the proposed method for various ECG lengths. For example, ECGFP-5 denotes the proposed method with 5 seconds of ECG. Results for LivDet20l5 participants are indicated in Table V. It can be seen that the proposed method (i.e. fusion of the ECG and LCPD) performs significantly better than the state- of-the-art methods in the LivDet20l5 competition.

[0097] While performance of LCPD used in our method is in the range of top performers of the LivDet20l5 competition, fusion with other liveness detection algorithms could produce further advances. In addition, although in this study we investigate fusion of ECG and a software-based liveness detection method, a hardware-based method can also be used in the same way because the proposed invention performs the fusion on score level.

[0098] In addition to the 12 participants listed in Table V, one complete fingerprint system is also submitted to the LivDet20l5 competition. Participants were provided with three spoof receipts and their submitted fingerprint system were tested on those spoof types as well as two unknown spoof materials. HTER of the submitted system was 8%, which is by far behind the proposed invention.

[0099] Table VI, below, showing a summary of the performance of three algorithms from various LivDet competitions, shows the reported rate of misclassified fake fingerprints (ferrfake) when the rate of misclassified live fingerprints (ferrlive) is 1%. This represents the percent of spoof attacks that have been able to fool the system when only 1% of live attempts are mistakenly rejected. For comparison, ferrfake of the MmLD (fusion of ECG and LCPD) is also presented. The average ferrfake of these methods at ferrlive=l% is about 45%. This indicates the poor performance of the state-of-the-art liveness detection methods, while the fusion of ECG and LCPD results, as discussed above, performs significantly better. The positive and negative trials used in Table VI are the same as Table V above; i.e. 276 positive and 276 negative trials per experiment.

TABLE VI. ferrfake (in percent) at ferrlive=l%. Results for 3 participants of LivDet2015. ferrfake of the MmLD (fusion of ECG and LCPD) for different ECG lengths is also presented. To be consistent with LivDet2015 competition, only LiquidEcoflex and RTV are considered as unknown spoofs. Standard deviations are in parentheses.

[00100] Table VII, below, compares the performance of three recognition fusion methods and a Bozorth3 fingerprint recognition method. The EER of Bozorth3 on 3 different datasets is shown in the last column of Table VII.

[00101] Fusion of ECG and Bozorth3 recognition is by sum, product, or maximum rules. Since the EER of fingerprint recognition and ECG recognition modules are approximately the same, we use equal weights for both traits when applying a weighted sum rule. As there are 46 subjects, as discussed above, for which there are 3 test sessions available, the number of positive and negative (zero-effort) trials in Table VII is respectively 46 ^c 3 and 46 ^c 45 ^c 3 for each spoof in each run.

[00102] Comparing the results of different scanners in Table VII, one may observe that while Bozorth results in an EER of 2.7% on the GreenBit scanner, its performance degrades to an EER of 15.1% on the Biometrika scanner. However, performance of the multimodal system is less sensitive to the scanner, i.e. a better generalization on different scanners. As indicated in Table VII, the sum rule generally outperforms the product and maximization rules.

TABLE VII. EER (in percent) of MmHR (fusion of ECG and Bozorth) for different fusion rules and different values of HC threshold. Standard deviations are in parentheses.

[00103] FIGs. 7 and 8 compare the EER of ECG and fingerprint recognition with and without template updating (‘adaptive’), where FIG. 7 shows a first test session using the 46 chimeric subjects discussed above, and FIG. 8 shows a second test session using the same subjects, with results averaged over 3 datasets and bars representing standard deviations. The length of test sessions was determined by HC criterion and the 30-second upper limit.

[00104] Two types of attack are considered: zero-effort attack and spoof attack. In the zero- effort attack, an intruder provides their own fingerprint sample but claims to be someone else. In the spoof attack, an intruder provides a fake fingerprint sample of the claimed identity. In both cases, an ECG signal is also recorded. For each of 46 chimeric subjects there are 3 fake and 3 live samples corresponding to 3 rounds of testing. Therefore, in template updating experiments, in each testing round there are 46 positive, 46 spoof, and 46 x 45 zero-effort trials. However, for computational simplicity we only consider 500 zero-effort trials.

[00105] At the end of the first test session, on average about 79% of chimeric subjects are selected for template updating. Likewise, at the end of the second test session about 76% are selected. In FIGs. 7 and 8, it can be seen that the performance of both ECG recognition and fingerprint recognition using Bozorth improves due to template updating. The recognition rates reported in FIGs. 7 and 8 are based on 46 positive and 500 negative (zero-effort) trials for each type of spoof in each dataset. The results for different types of spoofs and datasets are averaged and the entire process is repeated 50 times. The average and standard deviation are reported.

[00106] FIG. 9 shows the EER of the MmLD block in the third test session after performing automatic template updating in earlier test sessions. For comparison, EER of other methods are also shown as horizontal lines. In order from top to bottom, the horizontal lines in FIG. 9 correspond to WLD, IQ A, LPQ, BSIF, LBP, and LCPD, respectively. It can be seen that the proposed approach performs significantly better than comparison methods for a wide range of ECG lengths. In computing the liveness detection results in FIG. 9, 46 positive and 46 negative (i.e. spoof) trials have been used for each type of spoof in each dataset. The results for different types of spoofs and datasets are averaged and the entire process is repeated 50 times and average and standard deviation are reported.

[00107] FIG. 10 compares the recognition rate of the proposed system (with template updating) with the baseline i.e. Bozorth in the third test session. It can be seen that the proposed invention performs significantly better than Bozorth. In computing the recognition rates in FIG. 10, 46 positive and 500 negative (i.e. zero-effort) trials has been used for each type of spoof in each dataset. The results for different types of spoofs and datasets are averaged and the entire process is repeated 50 times as explained before and average and standard deviation are reported.

[00108] FIG. 11 shows one possible configuration of medical and non-medical sensors on a smartphone. The smartphone includes three ECG electrodes and one fingerprint sensor. A first ECG electrode is combined with the fingerprint reader in combined sensor 11100. A second ECG electrode 11200 is on a first side of the smartphone, and a third ECG electrode 11300 is on a second side of the smartphone. This arrangement may enable a first finger of a user’s first hand to be scanned by combined sensor 11100; resulting in a fingerprint scan of the first finger of the user’s first hand and an ECG signal of the user from the three ECG electrodes.

[00109] The combined sensor 11100 in FIG. 11 can be equipped with another sensor for capturing another biometric such as the medical biometric photoplethysmogram (PPG) as shown in FIG. 12. Photoplethysmogram (PPG) is one alternative biosignal to ECG, and which may only require one hand. A PPG sensor may measure the absorption of light through a finger as the heart pulses.

[00110] Photoplethysmography sensors have been developed for the fingertip and the ear lobe, and they provide a convenient, noninvasive means of measuring heart rate and heart rate variability. PPG-based signals also have the potential to be used for biometric authentication, even though PPG signals may convey less information than ECG. [00111] Similarly, biosignals like electroencephalogram (EEG), and electromyography (EMG), etc., may also be used as biometrics.

[00112] FIG. 12 shows an embodiment in which two medical biometrics and one non-medical biometric are used; a set of ECG electrodes and a PPG sensor are used with a fingerprint scanner In the embodiment depicted in FIG. 12, the first medical biometric sensor 12100 is an ECG electrode in the form of a ring around a button 12300 on a smartphone 12400, while the second medical biometric sensor 12200 is a photoplethysmogram (PPG) sensor which is on or incorporated into the button 12300. Button 12300 may be or incorporate a non-medical sensor as well, such as fingerprint reader. Further ECG electrodes or other sensors may also be

incorporated onto or in a surface or a button or other surface feature of smartphone 12400. PPG is a vital signal like ECG. Instead of using ECG alone, as described before, an embodiment may combine ECG and PPG to form a fused medical biometrics recognition score to be used it in the same way an ECG recognition score was used in the tested embodiment.

[00113] In other embodiments, only one type of medical biometric sensor, such as a PPG capturing device, may be used. The medical biometric sensor may be combined with a nonmedical biometric capturing device, such as an iris scanner or fingerprint sensor.

[00114] FIGs. 13 A and 13B show a front view and a back view, respectively, of a smartphone 13300 according to an embodiment of the invention. According to this embodiment, a nonmedical biometric sensor, namely a fingerprint sensor 13200, is provided, with a first ECG electrode 13100. Fingerprint sensor 13200 and first ECG electrode 13100 are provided on a button on the front of the smartphone 13300. A portion of the back of the smartphone 13300, such as the entire back of the smartphone 13300, may be a second ECG electrode.

[00115] FIGs. 14A and 14B shows a front view and a back view of a smartphone according to a further embodiment of the invention. According to this embodiment, a first ECG electrode 14100, is provided as a decorative ring on or around a frontal button of smartphone 14200. Further ECG electrodes may be on or incorporated into other buttons of smartphone 14200. For example, an ECG electrode could be on or incorporated into or could replace power button 14300, further ECG electrodes could be on or incorporated into or could replace volume buttons 14400. Further medical biometric sensors could be formed or incorporated into portions of the body of smartphone 14200, such as an ECG electrode forming or incorporated into a first portion 14500 of the back of smartphone 14200 and an ECG electrode forming or incorporated into a second portion 14600 of the back of smartphone 14200. Further medical biometrics such as PPG can be on or incorporated into buttons or other surfaces or components of smartphone 14200.

[00116] FIGs. 15A to 15D depict an embodiment in which medical and non-medical biometrics are used to secure a padlock 15000. Padlock 15000 includes a fingerprint sensor 15200 and two ECG electrodes 15100. Padlock 15000 may also include an ECG electrode built into or on or near fingerprint sensor 15200 to enable the user’s hand interacting with the fingerprint sensor to also provide an ECG signal.

[00117] As depicted in FIG. 16, in embodiments incorporating a PPG (photoplethysmogram) medical biometric, the PPG can be recorded using combination of a LED 16200 and Photo- Diode (PD) 16300 from any part of the body, such as from a finger 16100. LED 16200 may input rays into finger 16100 while PD 16300 may receive rays from finger 16100. PPG sensors have been used in other personal devices to measure heart rate, such as disclosed in ETnited States patent application publication number 20160038045 and United States patent number 8,948,832.

[00118] As depicted in FIG. 17, in some embodiments TEOAE (Transient Evoked Otoacoustic Emissions) may be used with another medical or non-medical biometric in a security system. TEOAE is an acoustic response generated from the cochlea. It is related to inner ear structure and can be recorded using an earphone with a built-in microphone, such as one built into earphone 17000.

[00119] As depicted in FIG. 18, in some embodiments EMG may be used with another medical or non-medical biometric in a security system. Electromyogram (EMG) is related to the electrical properties of a muscle and may be sensed from an EMG sensor such as sensor 18000 or may be built into another product, such as the armband disclosed in United States patent application publication number 20140198034 and United States patent application publication number 20140198035 and related patent disclosures.

[00120] FIG. 19 depicts an embodiment in which medical and non-medical biometrics are used to secure a door handle 19000. Door handle 19000 includes a fingerprint sensor 19200 and an ECG electrode 19100. Door handle 19000 may also include an ECG electrode built into or on or near fingerprint sensor 19200 to enable the user’s hand interacting with the fingerprint sensor to also provide an ECG signal.

[00121] Embodiments of the present invention may include or be provided upon computer readable media. Steps or modules or aspects of embodiments of the present invention may be performed using computer hardware and software, such as computer processing units and computer operating systems.

[00122] The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Certain adaptations and modifications of the invention will be obvious to those skilled in the art. Therefore, the presently discussed embodiments are considered to be illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than the foregoing description and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.