Government Rights In The Invention

[0001] This invention was made with government support under National Science Foundation

No. CNS-1149611. The government has certain rights in this invention.

Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No.

62/176, 108, filed February 9, 2015.

Background of the Invention

[0003] Many smart building applications require indoor identification of individuals for

personalized tracking and monitoring services. For example, in a nursing home, identifying monitored patients and tracking individual activity range helps nurses understand the condition of patients. Similarly, such identification information can also be used in smart stores/malls to analyze shopping patterns of customers.

[0004] Various methods and apparatuses have been explored for identification of individuals.

These methods and apparatuses utilize biometrics (face, iris, fingerprints, hand geometry, gait, etc.) and sensing technologies (vision, sound, force, etc.). Some biometrics, such as iris, fingerprints and hand geometry achieve relatively high identification accuracy and are widely used for access control. However, they often require human interactions, and, as such, they have limited usefulness for ubiquitous smart building applications. With other methods, such as facial and gait recognition, it is often difficult to get enough sensing resolution required for recognition from a distance, particularly when used in surveillance applications. Numerous sensing technologies have been explored and proven useful and efficient, but all have limitations. Vision-based methods often require line-of- sight, with performance dependent upon lighting conditions, and may require high computational costs, which limits their viability. Likewise, sound-based methods have limitations when deployed in conversation sensitive areas, as they are prone to be affected by ambient audio. Force-based methods typically utilize specialized floor tile sensors for footstep detection, resulting is the requirement for dense deployment at a high installation cost.

Summary of the Ivention

[0005] This invention performs identification of individuals via footstep induced structural vibration analysis. People walk differently, and therefore their footsteps result in unique structural vibrations. The invention measures these vibrations, detects signals induced by footsteps, extracts features from these signals, and applies a hierarchical classifier to these features to identify each registered user with a high confidence level.

[0006] Due to better wave attenuation properties in solids, with proper amplification, the

invention can detect individuals at a relatively large range. As a result, the invention has a sensing density that is low compared to known force-based methods. Compared to vision-based and sound-based methods, the invention measurement suffers less interference from obstacles that move around, because the vibrations travel in the structure itself. Furthermore, the installation of the invention is non-intrusive, consisting of one or more geophones installed on or near the floor surface, which can be accomplished without alteration the structure.

Brief Description of the Drawings

[0007] Fig. 1 shows recorded step events from three different people wearing soft-soled shoes, with each row showing a different person. The left column shows the time domain of the step event, while the right column shows the frequency domain of the same step event.

[0008] Fig. 2 shows three separate recordings of step events from the same person. The left column shows the time domain of the step event, while the right column shows the frequency domain of the same step event.

[0009] Fig. 3 presents the schematic overview of the components of the invention, highlighting the required functional modules.

Detailed Description of the Invention

[00010] Each person has a unique walking pattern due to many factors, including, for example, individual physical characteristics, the center of gravity position during the walk, the way feet contact the ground, etc. Due to each person's unique walking pattern, there is a uniqueness and consistency of the footstep induced floor vibration for each person.

[00011] The floor vibration signal induced by a footstep is referred to herein as a step event. A sequence of step events from a continuous walk is referred to herein as a trace.

[00012] The floor vibration signal is captured by one or more sensing modules, each of which consists of three major parts: a geo-phone, an amplifier, and an analog-to-digital converter. The geophone is set on the floor of the structure to capture floor vibration signals. The analog signal is then amplified. In the preferred embodiment, the amplification is performed by connecting the geophone to an op-amp with an empirical amplification gain of approximately 1000, which allows approximately a sensing range of about 10m for particular factors including floor type, shoe type, etc., however, as would be realized by one of skill in the art, many methods of amplification could be used. A sampling rate of 25 kHz allows the capture of a wide frequency range of signal characteristics, but other sampling rates could be used.

[00013] In tests of the system using this sensor module, step events from different people were recorded, showing distinguishable variations in both time and frequency domains. Fig. 1 shows step events from three people, labelled (a), (b) and (c). The left and right columns show corresponding time and frequency domain signals from the same step event for each person, respectively. In addition, dotted lines and dashed/dotted lines indicate locations of peaks and valleys in the frequency domain, respectively. As shown in Fig. 1, the locations of peaks and valleys vary among different people, which can be used as features to identify them.

[00014] Step events from one person bear resemblance between each other. Fig. 2 shows three step events from one trace (i.e., from a series of steps by the same person). The left and right columns show corresponding time and frequency domain signals, respectively. Note that the step events span similar time duration with nearly identical velocity profiles in the time domain. The frequency domain patterns are well aligned across the three step events. This data demonstrates that a step event is a feasible metric for identification of individuals. [00015] As shown in Fig. 3, the identification system of the present invention contains three modules: sensing 10, footstep analysis 20, and decision-making 30. Fig. 3 displays the relations of these modules.

[00016] Sensing module 10 (described above) performs floor vibration sensing 12. The vibrations sensed are those that are induced by a person walking across a floor surface Sensing hardware 10 amplifies the signal received from the sensor and outputs a digital signal derived from the amplified analog output of the sensor. As discussed above, in a preferred embodiment of the invention, the sensor is a geophone of a type that is well known and commercially available, however, other types of sensors may be used. The system may use multiple sensing modules 10, depending upon the desired area of coverage.

[00017] Footstep analysis module 20 takes a trace of step events and extracts individual step events therefrom. Features representing characteristics of each step event are then extracted.

[00018] The key to identification of individuals is to extract and analyze the characteristics of step events. There are two major components in the footstep analysis module. The first is step extraction 22 to obtain step events, and the second is feature extraction 24, which characterizes step events via feature extraction.

[00019] Step events contain a person's identity information, while the interval between step

events is mainly noise. Therefore, to identify people, step events need to be extracted from the trace containing the entire vibration signal by step extraction module 22. The noise is modeled as a Gaussian distribution, and then an anomaly detection method is used to extract step events. The threshold value to detect a step event is determined by an allowable false alarm rate.

[00020] Two detection algorithms have been developed for extracting step events from the trace containing the entire vibration signal. The first detection algorithm is threshold-based method and uses the time representation of the signal. This method finds the threshold using the background noise distribution and a footstep event is indicated whenever the energy of signal exceeds a defined threshold. The second detection algorithm uses the time-frequency representation of the signal. This approach is able to deal with signals with very low signal-to-noise ratio where it is difficult to differentiate between the background noise and footstep-induced vibrations and improves the accuracy by distinguishing between footsteps and other sources of non- stationary excitation. Some examples of such sources include vibrations induced by dropping an object and shutting a door. This algorithm uses the characteristics of structure to find the frequency

components of the signal which are more robust to background noise. Furthermore, it includes a classification algorithm which distinguishes between footstep-induced vibrations and vibrations induced by non-stationary signals.

[00021] Feature extraction module 24, extracts features from selected step events. The events from which to the features are extracted are selected based on their signal to noise ratio. Features can be more efficiently extracted from step events in a trace having a high signal-to-noise ratio. Features of the selected steps are then extracted to characterize the footsteps.

[00022] Step events in one trace may have different signal-to-noise ratios depending on the

relative distance of the location of each step event to a sensor. This leads to a variation in classification performance. A small number of step events closest to the sensor, and consequently with the highest signal-to-noise ratio, are selected for classification.

[00023] Once the step events are selected, they are normalized to remove effects of the distance between the footstep location and the sensor, and for different types of floor surfaces, for example, a hard floor versus a carpeted floor. Step events closer to the sensor have a higher signal energy, which is calculated as the sum of squared signal values. Each selected step event is divided by its signal energy to normalize for differences in the distance of each step event from the sensor, thereby removing the distance effect, the distance of each step event from the sensor is irrelevant to characterizing the step event for a particular person and contains no identify information.

[00024] After normalization, features are computed in both time and frequency domains to

present different characteristics of step events for each person. Time domain features may include, but are not limited to standard deviation, entropy, peak values, partial signal before and after the maximum peak, etc. In the frequency domain, features may include, but are not limited to spectrum centroid, locations and amplitudes of peaks, power spectrum density, etc.

[00025] Once these features are extracted, decision-making module 30 takes the features and runs through a hierarchical classifier, which includes both step level classification 32 and trace level classification 34. The identification individuals is modeled as a hierarchical classification problem in the invention. A hierarchical classifier includes step level classifications 32 and trace level classifications 34. Identification accuracy is increased by utilizing the fact that steps from the same trace belong to the same person. The classified step events are compared against a database 36 of previous step events from identified individuals to accurately identify the individual.

[00026] The system takes features of step events from different people's traces to generate a classification model using a Support Vector Machine, which maximizes the distance between data points and the separating hyper-plane. The step level classification 32 returns both the identification label and the confidence level from testing the step event.

[00027] By classifying identity at trace level 34, classification uncertainty is reduced by

eliminating outlier step events from the step level classification 32, thereby enhancing the overall identification accuracy of the system.

[00028] Each step event classified obtains an identification label and a confidence level as the result of the step level classification 32. Since multiple steps events with the highest signal-to-ratio are referenced a confidence matrix P _sxn is created, where n is the number of people to be classified, and s is the number of step events selected from the trace. The identity of the step event with highest confidence level is selected to be the identity of the entire trace.

[00029] Achieving high accuracy for the classified step events is important. When a new person's trace is detected, it is possible that step events in the new trace are not similar to any of the footsteps in database 36. In this case, the confidence levels of all steps in a trace are equally low, and the system detects such situations. The confidence level threshold CLthreshoid is set to determine a reliable classification result. The trace is considered to be identifiable when the confidence level is higher than the confidence level threshold. Otherwise, the trace is determined to be unclassifiable (i.e., the trace of a previously un- identified person). The system can adjust this threshold to obtain different identification accuracy based on the application.

[00030] In tests of the system, various numbers of persons, and various types of structures were used, and the system was found to provide a high identification accuracy.

[00031] Many applications of the system have been identified in the areas of individual

monitoring, analysis of group behavior and security.

[00032] Individual identification and monitoring can be used to detect children or elderly patients in an in-home setting, where the system can detect and identify individuals and respond accordingly, for example, if they appear alone in designated area (e.g., the kitchen or bathroom), or if they leave the premises. For elderly subjects, the system can be used to analyze walking patterns to predict fatigue level, which may be useful in and prevent fall events from occurring. Finally, individual identification can be used to identify

individuals in a smart space, and personalize the environmental settings, for example, by detecting the identity of an individual as they walk through the front door, the smart system can start their computer before their arrival, then, by tracking the individual to the elevator, the smart system can play their favorite songs in the elevator. Likewise, the system could also set customized temperature, turn on lights, unlock doors, etc.

[00033] The system may also be applied to monitor and analyze group behavior. In a supermarket, shopping mall or airport environment, the system may recognize individual shopping patterns and understand the group shopping pattern based on the characterization from the footstep induced vibration signals (e.g., height, weight, gender, etc. ). In a smart office type environment, the system could recognize the activity range of each individual and assign resources/space and manage energy consumption based on the optimized convenience.

[00034] Lastly, there are security applications for the system. For example, the system may be used to authorize access to a particular area by determining if the detected footsteps fit the profile of an authorized individual. The system may also be useful in theft detection, by detecting changes in the pattern of individual footsteps due to hidden objects on the body of the individual. Lastly, the system may be able to detect specific gait patterns due to individuals carrying weapons on their body.

[00035] Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limiting to the details shown. Rather, various modifications may be made in the details without departing from the invention.