[Title of Invention]

METHOD AND SYSTEM FOR GENERATING EMBEDDING FOR SIGNAL

[Technical Field]

[0001]

This invention relates generally to signal processing, and more particularly to comparing signals subject to transformations.

[Background Art]

[0002]

In many applications, it is desirable to compare signals, such as signals representing images, sound, or industrial automation signals that have undergone transformations. These transformations can include translation, rotation, scaling, etc. Furthermore, the signals can be parametrized with a parameter controlling some aspect of the transformation. For example, if the transformation is a rotation, then the parameter can be an angle of rotation. Similarly, if the transformation is translation, then the parameter can be an amount of translation.

[0003]

There are several instances where such comparisons are necessary. For example, in many applications, it is necessary to identify a component of an image being processed, such as an image of a corner, a vehicle, or a face. To identify these components, an example or template, the image of the face or the car should be compared with the elements of the image being processed in order to be identified. However, the face or the car in the image being processed does not necessarily have the same scale, position, or orientation as the face or car in the example image.

[0004] Similarly, in many industrial applications it is desirable to compare signals, obtained from equipment monitoring sensors, with template signals, that are examples of proper equipment operation. When the signals are dissimilar, this is an indication of equipment malfunction, and an operator should be alerted. However, such signals are often translated in time (delayed), compared to the template signals.

[0005]

There are several approaches in performing such comparisons. The most common is to compare a first signal with all the transformations of a second signal and select the the signal that matches best. The quality of this match indicates how similar the signals are under the transformation. One example of this approach is to determine the autocorrelation of the signals and select the peak of this

autocorrelation, which is an approach very common in a number of applications. This comparison can often be performed "in-place," i.e., without explicitly determining all the transformations.

[0006]

Fig. 1 is a schematic of a conventional signal comparison technique wherein signals 101-102 might be similar but transformed in some way, e.g., due to a translation in time. Alternatively, two signals 103-104 might be dissimilar even when the transformation is taken into account. It is desired to determine a distance metric that characterizes the similarity. For example, in one embodiment, the distance is a Euclidian distance. Typically, this is performed using the

cross-correlation 1 10, which determines the correlation of the two signals under all the possible transformations (i.e., translation in time). This results to a

corresponding signal distance 115 of the signals under all the transformations. From the cross-corralation and the corresponding distance a maximal correlation 1 11, and a corresponding minimal distance 116 are determined. [0007]

A problem with that approach is the inherent computational complexity when a single signal needs to be compared with more than one signals, e.g., stored in a database, to determine similarity. Because the comparison of pairs of signals essentially compares many signals, according to the various of transformations considered, and selects the best match. The computation can be significant, even when the comparison is "in-place." This issue is alleviated in an alternative invariant approaches described below.

[0008]

One alternative approach is to further transform the original signals in a way that is invariant with respect to the original transformation. In other words, the transformed signals would be the same, even when the original signals have a different shift, rotation or scale, etc., according to the invariance desired. The transformed signals are subsequently compared and the output of this comparison is used to determine the similarity of the signals.

[0009]

The transformation is typically much less computationally intensive than the comparing of all the pairs of signals as described above. Thus, the comparison of a signal with many signals in a database can be made much more efficient. On the other hand, one of the problems with the invariance approach is that, often, it is difficult or impossible to find a transformation with the appropriate invariance that also preserves sufficient information to perform meaningful comparisons.

[0010]

A similar approach is to extract features from the signal that are invariant to the desired transformations, and not transform the entire signal. Then, the features are compared to determine a match. This is the approach taken in many popular image descriptors, such as a scale-invariant feature transform (SIFT) or a compressed histogram of gradients (CHoG), and other similar methods. The benefit of that approach is that it is easier to extract invariant descriptors than to transform the entire signal. The drawback is that these descriptors are usually designed with particular applications and tasks in mind, such as detecting objects in an image, and are tuned specifically for this task. For each different task, a different descriptor needs to be designed with the appropriate properties tuned to the task.

[Summary of Invention]

[0011]

To overcome the above problems and to preserve computational efficiency, the embodiments of the invention provide a method for generating operator invariant embeddings. These embeddings provide a way to combine the benefits of conventional approaches without their drawbacks. The embeddings provide a feature extraction mechanism which, in contrast to conventional approaches, is generic and does not need to be designed specifically for each application or each type of signal.

[0012]

Furthermore, if it is desired to preserve certain information derived from the embedding generation, the embedding can be considered as an operator-invariant transformation. In this case, the signal can be recovered from the embedding. The computation of the embedding can be determined by comparing the signal with a set of intermediate signals and all their transformations.

[0013]

In the later case, each signal is compared with all the transformations of a small number of intermediate signals. These comparisons serve as a feature set that are compared with the features extracted from the other signals. Thus, the computational cost of comparing the signal with all the transformations of all the other signals is reduced because the exhaustive comparison only happens with the transformation of a few template signals. The outcome of this comparison is a feature, which is compared with the features extracted from all other signals, which is a much faster comparison.

[Brief Description of the Drawings]

[0014]

[Fig- 1 ]

Fig. 1 is a schematic of a conventional signal comparison technique.

[Fig. 2]

Fig. 2 is a schematic of a process to generate an embedding according to an embodiment of the invention.

[Fig. 3A]

Fig. 3A is a schematics of the process to extract a single coefficient or feature accoring to embodiments of this invention.

[Fig. 3B]

Figs. 3B is a schematics of the process to extract a single coefficient or feature accoring to embodiments of this invention.

[Fig. 4]

Fig. 4 is a schematic of a signal query execution according to an

embodiment of the invention.

[Fig. 5]

Fig. 5 is a schematic of a signal query execution according to an alternative embodiment of the invention.

[Fig. 6]

Fig. 6 is a schematic of a client/server query generated remotely according to an alternative embodiment of the invention.

[Fig- 7] Fig. 7 is a pseudocode that corresponds to the schematic shown in Figures 3 A and 3B according to an embodiment of the invention.

[Description of Embodiments]

[0015]

Operator-Invariant Embedding

Fig. 2 shows an embodiment of a method for generating an embedding for a signal which is invariant to an operator. Each of the measurements vectors e, 210 is used to measure an acquired signal x 201 and generate an embedding of x as y 240.

[0016]

The signal can be acquired 205 in a preprocessing step using, e.g., a sensor 205, e.g., a cameras, a microphone, or other types of sensors. The signal can be a visual signal, e.g., an images or a videos, an audio signal, e.g., speech and music, or industrial automation signals. Alternatively, or addditional the signal can be acquired by an application, e.g., an image or a speech processing application that extracts features from the signals. The signals can also be obtained by sensors sensing factory equipment, mechanical systems, electrical systems, such as power storage systems including batteries.

[0017]

The signal 201 is measured with each of a set of measurmement vectors e, 210 using a measurement process 220 to produce a set of corresponding

coefficients 230. All the coefficients are combined in a vector to produce the embedding 240 y.

[0018]

The method can be performed in a processor 200 connected to memory and input/output interfaces by buses as known in the art.

[0019] Subsequently, the embedding can be used to compare the similarity of signals as described below.

[0020]

Signal Similarity

In order to compare two signals a similarity measure or a dissimilarity measure is often used. The similarity measure is high when the two signals are similar and low when the two signals are dissimilar. The dissimilarity measure is low when the signals are similar and high when the signals are dissimilar.

[0021]

An example of a dissimilarity measure is a signal distance, which is a measure very commonly used in signal comparisons. Given two signals x, x' G S, where S is the set of signals of interest, the -6 _V distance is defined as

where p > 1. Very similar signals have small distance when the signals are similar, i.e., x = x', and the distance is zero.

[0022]

If one signal is a multiple of the other, then the _v distance can be high, even though certain applications can consider those signals as the same. For that reason, an alternative similarity measure is often used, which comprises of first normalizing the signals, and then determining their £ _p distance. This measure is

X'

llxllp llx' llp (2)

V

[0023]

Another class of useful similarity measures are divergences, which are similar to distances. For example, a Bregman divergence is often used to compare signals. A Bregman divergence may or may not satisfy neither the triangle inequality nor symmetry.

[0024]

One of the most useful similarity measures is signal correlation c, defined as c(x, x') = (x, x') =∑t XjXj, . (3)

[0025]

As with distance measures, multiplication of signals by a scalar can affect their similarity as measured by the correlation measure. In such cases, the correlation coefficient, which uses the normalized signals, is

= (5 S7>- < ⁴ >

[0026]

In a number of applications, two signals that are the opposite of each other, i.e., x «— x', or x *— cx' for some c > 0, should still be considered similar. In such applications, the magnitude of the correlation or the correlation coefficient, i.e., I c(x, χ') I or |p (x, χ') | is the more appropriate similarity measure of the signals.

[0027]

An alternative similarity measure, often preferred, is based on functions known as radial basis kernels. Radial basis kernels have a defining property that the kernels are only a function of the distance II x— x' || _p of two signals, i.e., the kernels are defined as

tf(x, x') = /c(ll x - x' ll _p ), (5) for some appropriate /c(-) and p.

[0028]

Two of the most commonl radial basis kernels are the Laplacian and the Gaussian kernel, defined as KL (X, X') = - e- ^l|x - ^x " ^ll , and (6)

K _G (x, x') = e-ciix- 'Hi, (7) respectively.

[0029]

Of course, several other similarity measures exist and can be used in this subsequent development.

[0030]

In order to design the operator-invariant embeddings according to the embodiments of the invention, in addition to the signals x, x', consider an operator T _a : S→S that is parametrized by a parameter a. The embeddings are designed to be invariant to this operator. The operator T _a characterizes how a shift, scale change, rotation, etc., transforms a signal, such as an image. The operator can also characterize multiple transformations, e.g., both a rotation and a scale change, or a rotation and a shift, or all three.

[0031]

The parameter characterizes the specific properties of the transformation. For example, if the transformation is a rotation, then a may be the angle of rotation. If the transformation is a translation, then a may be the amount of translation. If the operator combines multiple transformations, then a may be a tuplet of parameters, as required for each transformation, e.g., includes both a translation and a rotation amount. The parameter a can take a range of values a E 31, e.g., in a rotation, where 31 can be the range of angles, from 0 to 2π, or — 7Γ to 7Γ.

[0032]

For each value of the parameter a, there is an inverse value E 31,

depending on the transformation, which would invert T _a , i.e., Γ„(Τ _α (χ)) = x. For example, the rotation by to the right is inverted using a rotation by to the left, i.e., a rotation by— to the right, which is also equivalent to a rotation by

2π— a to the right. Similarly, a scale change parameter a is inverted using a scale change by 1/ .

[0033]

The embedding is also an operator, E: S→V, where S is the signal space, as above, and V is the embedding space. Typically, the embedding space is V is ]R ^M or C ^M for some M. This operator is approximately invariant to T _a , if

E(x) « E(T _a (x)) for all 6 R " and all x £ <S ^" , with exact equality if £(·) is exactly invariant to _α (·).

[0034]

A dissimilarity measure is generally denoted using d(x, x'), or a similarity measure s(x, x'), as appropriate for the application. Similarity has high value if x is similar to x' and low if not. Dissimilarity has low value if x is similar to x' and high if not.

[0035]

In order to generate £(·) _> a number of measurement vectors e _t , i = 1, ... , M 210 are used. In typical embodiments, the vectors are generated randomly. Their number M depends on the application. For example, ean be based on a

trade-off between speed and accuracy. A higher M gives higher accuracy.

Generally, M should be much less than the dimensionality of the signal, and also can depend on the number of signals that will be compared. For example, in the database query and search application, as describde below, M increases

logarithmically with the number of signals in the database. Typical numbers depend on the application type. For example, when using the embedding to look for a template in an image, one would expect M to be less than a few hundred, typically less than 200.

[0036]

Depending on what the signal space S is, the vectors can be generated in a number of ways. For example, the vectors can be generated using a uniform distribution on a unit sphere in S. Or, if, for example, S c . or C ^N , then each ε _έ can be generated as a random real or complex N-dimensional vector, with each vector component drawn from an i.i.d. real or complex normal distribution.

Alternatively the distribution can be a uniform, Rademacher, or other subgaussian real or complex distributions.

[0037]

Embodiments of the measurement process 220 are described in more detail with reference to Figs. 3A and 3B.

[0038]

In order to measure the signal 201 with the measurement process 220, an embodiment of the invention generates a sufficiently fine finite grid Q from the parameter set, i.e., Q c R " , where G = \Q\ <∞ is the number of grid points. In general, the grid sufficiently fine if for all parameters a E R, then sup _a , _e ^ | a— cc'\≤ ε for sufficiently small tolerance ε. A similar guarantee, with multiple tolerance values, should be provided for a multiparameter grid, e.g., if captures both translation and rotation. In most implementations, the grid would contain parameters and their inverse, i.e., a 6 Q <= a 6 Q, although this is not necessary.

[0039]

Fig. 3A shows by example how the grid points are used transform each measurement vector 210, with multiple transformations, one for each grid point. In particular, for each grid point, a transformation T _a , j = 1, ... , G, 320 is used to transform the measurement vector ej. Each transformed measurement vector is compared 310 with the signal 201 using the similarity or the dissimilarity metric. Matches are selected and the inner product of the signal with the matching transformed measurement vector is determined 340. In the preferred embodiments the matches are selected to be the best matches according to the selected metric is 330. However, the matches can be, for example, the second or third best match, or more than one matches using different transformations of the same measurement vector. In the example in the figure, the best match is the transformation through T _a2 . The result of the inner product y _t , is the i ^th coefficient 230 of the embedding 240 y.

[0040]

Alternately or additionally embodiment is shown in Fig. 3B, in which the signal 201 is transformed using each parameter in the transformation 320 and the result is compared with the measurement vector 210 using the similarity or the dissimilarity metric 310. The selected matches 330 are used to compute the inner product of the transformed signal with measurement vector 340 in order to produce the coefficient 230.

[0041]

The pseudocode for the procedure that determines the embedding according to the schematics of Figs. 3 A and 3B is shown in Fig. 7.

[0042]

In the preferred embodiment, for each measurement vector, the procedure first determines the transformation in the measurement vector that maximizes its similarity with the signal (step 2). To do so, the procedure minimizes the

dissimilarity function d(v) or maximizes the similarity function s(v), depending on which one is available by the application. Then, the procedure measures the signal using this transformation (step 3). Note that steps 2 and 3 can be replaced by α; = argmind(T' _a (x), θί), or, cc _t = argmaxs(7 _a (x), and (8) aeg aeg

y _i = (T _a (x), e _i ), (9) respectively. In most transformations, but not all, both formulations are equivalent and can be used interchangeably.

[0043]

The formulation in Fig. 7 corresponds to the embodiment shown in Fig. 3 A, and the alternative formulation in (8) and (9) coresponds to the embodiment of Fig. 3B. The pseudocode in Fig. 7 is usually easier to implement in practical systems, especially when measuring a large number of signals.

[0044]

The main output of the embedding function is the vector of measurements y. Optionally the function also returns the optimal transformation parameters, , if the application requires the shifts. In some embodiments, the parameters can be used to reconstruct or approximately recontruct the signal from the embedding. The parameters can also be used to find the relative transformation of similar signals.

[0045]

Given two signals χ, χ' and their embeddings y = E(x , y' = E(x'), their similarity or dissimilarity can then be evaluated using s(y, y') or d(y, y') using an appropriately defined similarity or dissimilarity function in V. This similarity or dissimilarity is approximately representative of the maximum similarity or minimum dissimilarity when comparing all the transformations of x with all the transformations of x', with parameters taken from the parameter set 31.

[0046]

If the embodiment has also preserved the optimal transformation parameters used to compute the embedding, then parameters used to embed the two signals can be compared to produce the optimal transformation that would align or match the signals. For example, the average or the median relative value of the

parameters, or the average or the median relative value of the best matching coefficients can be an estimate of the relative transformation one signal should indergo to match the other signal.

[0047]

The embedding vector can also be quantized to q = Q (y) using an appropriate quantization function. This is very useful if, for example, the

embedding should be stored or transmitted, and the bit-rate used needs to be reduced.

[0048]

A hash of the embedding h = H(y) can also be determined, which can further be a locality sensitive hash (LSH), in order to further improve the speed of signal retrieval based on comparisons with the embedding.

[0049]

Uses and Protocols

There are several uses of these embeddings. One of the most important is operator-invariant signal retrieval from a database, e.g., a data base of images. In this application, a database of signals is searched, using an operator-invariant method, for signals similar to incoming query signals. The database is prepared using the appropriate operator invariant embedding by determining the embeddings of all the signals in the database and storing the embeddings as indices to the database.

[0050]

When the query is executed, the embedding of the query signal is first determined. The embedding is compared with the embeddings of all the signals in the database. Depending on the application, the closest match or a number of closest matches are returned, or an indication that a close match is or is not in the database. This is a much faster comparison, when contrasted with the alternative of comparing the query signal with all the transformations of all the signals in the database for all the parameters in the parameter space. The initial computational cost to prepare the database is small, especially when many queries need to be executed.

[0051]

The query time can be further reduced if from all the signals in the database a locality sensitive hash is determined from their embedding and stored together with the signal and its embedding. When a query is executed, a hash is determined from the embedding using the same hash function. Only the few signals in the database that have the same hash are selected for comparison.

[0052]

Figs 4 and 5 schematically show for preparing and querying a database of signals according to embodiments of this invention.

[0053]

Fig. 4 shows how the embedding described above can be used without hashing to prepare and query a database 450 of signals. The database is first prepared 420 as follows. For each signal x 421 the embedding y is computed 422. The embedding together with the signal forms a pair (x, y) 423. This pair is an index entry. All the index entries are stored in an index which is stored in the database 450 and used to index the database using indexing methods known in the art.

[0054]

To query the database with a query signal x' 41 1, a query is prepared 410 by computing the embedding y' 412 of the query signal. The query is executed by comparing 413 the embedding of query signal y' with the embedding y for each of the signals in the database 450. Matching signals are selected as the query result. The best match or several matches comprise a query result 414. These steps can also be performed in the processor connected to the database.

[0055]

The output of the query 415 is based on the query result. For example, the output can be the matching signals themselves, or metadata associated with the matching signals, or a decision based on the matching signals, such as to allow or disallow access to a user.

[0056]

Fig. 5 shows how the embedding can be used with hashing to prepare and query a database of signals. Similar to Fig. 4, the database is prepared 520 by first computing 522 the embedding Of each signal x 521. A hash h is computed from the embedding 523. The hashes from all the signals are index exntries and are used to build an index 524 for the database 550. The index indexes the signals and their embeddings according to their hash, using indexing methods known in the art.

[0057]

The query is prepared 510 in a similar fashion. The embedding of the query signal 511 is computed 512 and the hash of the embedding is computed 513. The hash and, in some but not all embodiments, the embedding itself is used to query the database.

[0058]

The query is executed 514 by retrieving from the index of the database a list of all the signals with the same hash as the query hash. These are all matching signals from the database. If the query embedding is also used in the retrieval, then the embeddings of the retrieved signals can also be compared with the embedding of the query signal to decrease the number of matching signals. The matching signals are the query result 515. In some embodiments multiple hashes can be used to increase the accuracy of the retrieval using methods well known in the art.

[0059]

Similar to the embodiment in Fig. 4, in this embodiment the output of the query 515 is based on the query result. For example, the output can be the matching signals themselves, or metadata associated with the matching signals, or a decision based on the matching signals, such as to allow or disallow access to a user.

[0060]

In a preferred embodiment shown in Fig. 6, the query is generated remotely, i.e., there is a client 601 executing the query preparation 510 and querying the database 550 stored at a server 602 using a network. The database is already prepared using the protocol 520 described above. In such cases, the client can transmit the signal, or a compressed version of the signal, and the database follows the protocol above, possibly after decompressing the signal. Alternatively, the client can determine the embedding, quantize the embedding or encode or compress the embedding in some other way, and transmit the embedding to the server to continue the search protocol 514. The server retrieves the matching signals 515 and transmits the query output 516 to the client. While Fig. 6

demonstrates a client/server embodiment using the indexed database described in Fig. 5, it is also possible in a similar manner to implement a client/server embodiment based on the database preparation described in Fig. 4.