The present disclosure relates in general to the technical field of mmWave radar networks and more in particular to a method for self-calibration of mmWave radar networks starting from moving target trajectories.

BACKGROUND

In the past few years there has been an increasingly growing interest in the use of millimeter waves (mmWaves) for human tracking, person identification and activity recognition. This demands solutions to improve the usability and practicality of such systems. Radars working in the mmWave band have emerged as valid alternatives to cameras for ambient monitoring, as they enable highly accurate environment sensing thanks to the small wavelength of the transmitted signal and they are robust to changing and poor lighting conditions. Moreover, the most important strength of the mmWave sensing technology with respect to traditional cameras is that they do not raise privacy concerns, as no visual representation of the scene is captured. This enables collecting the information of interest, such as the number of people in a room or recognizing a human gesture to control an electronic device, while preserving the privacy of the users. However, commercial mmWave radars have limited range (up to 6-8m) and are subject to occlusion. Covering medium-to-large spaces thus requires multiple radars (radar networks), with known position and orientation. This raises the problem of how to automatically obtain the positions and orientations of the radars, as it is often impossible to know them in advance, and it is impractical to manually input these settings at deployment time.

This problem remains unsolved in the literature and existing works tackling it are quite limited in terms of usability.

In particular, the present disclosure is based on the recognition by the author that the limitations stem from the need for a dedicated calibration phase in controlled conditions, where either a single human target walks along a linear trajectory, or multiple human targets remain still in a fixed position. These requirements significantly reduce the possible applications and deployment scenarios of mmWave radar networks, which should generalize to unseen, uncontrolled, real-world scenarios.

Stated in different words, the main problem of the existing mmWave radars selfcalibration techniques is that they require a dedicated calibration phase where the number of targets and movement trajectory must be known and controlled. In addition to this being impractical and time-consuming, errors in implementing these restrictions during the calibration phase negatively impact the quality of the calibration. In real- world scenarios, a fully automatic self-calibration should adapt to any movement trajectory of the, possibly multiple, tracked targets of any nature, i.e., not limited to humans. Only this level of robustness makes it possible to avoid external intervention.

A technical problem underlying the present disclosure is that of providing a method able to solve the problem of handling free movement trajectories and multiple targets even when occlusions occur, and the trajectories that can be generated by any uncontrolled target in the environment, thus broadening the applicability of the system to non-human targets.

The present disclosure aims to provide a self-calibration method for selfcalibration of a mmWave radar network and able to solve said technical problem with reference to the prior art and/or to achieve further advantages.

This is obtained by means of a self-calibration method as defined in the respective independent claim. Secondary characteristics and particular embodiments of the subject of the present invention are defined in the corresponding dependent claims.

The present disclosure is based on the recognition that the only way in which different radars can track the same target consists of a certain level of overlap between the fields of view of the radar to be calibrated, and that this physical requirement allows the development of an automatic calibration method.

In particular, the method according to the invention can work if multiple targets are present on the scene. The strengths of the proposed solution are the following:

- The dedicated cost function is able to select the track couples providing the best calibration performance, among all the available.

- Through the solution of the track-to-track association problem between the radars using a purposely designed cost function, the proposed technique exploits the presence of multiple targets moving in the environment to improve the calibration results, whereas existing solutions do not work in this case.

- It supports any kind of target movement trajectory with high calibration performance.

- It works even if occlusion events occur.

In addition, thanks to the present method, after the computing step, the radar sensors can be kept still or in the same positions without any further movements. The important condition is that the at least two radar devices have at least partially overlapping fields of view. Further characteristic features and modes of use forming the subject of the present disclosure will become clear from the following detailed description of embodiments thereof, provided by way of a non-limiting example.

It is, in any case, evident that each embodiment of the subject of the present disclosure may have one or more of the advantages listed above; in any case, it is not required that each embodiment should have simultaneously all the advantages listed.

DETAILED DESCRIPTION

The self-calibration method according to the present disclosure operates in the context of multiple mmWave radar sensors and allows one to estimate the relative position and orientation of each radar with respect to a common reference system (for example, one of the radars) from the movement of targets moving in the common part of the field of view (FoV) of the radars.

The method operates by pairs of radars, so, in the following, the method steps are described considering a network of two radars, referred to as first radar sensor and second radar sensor. For networks with more than two radars, it is sufficient to iteratively repeat the method for each pair of radars. In particular, according to the present disclosure, it is assumed that a set of tracks coming from the same target(s) of opportunity is available at each radar.

More specifically, a self-calibration method according to the present disclosure is for self-calibration of a mmWave radar network and the method comprises:

- A tracking step of tracking targets, which can be defined as targets of opportunity, by means of a first radar sensor and a second radar sensor having at least partially overlapping fields of view (FoVs) and obtained a set of first tracks and a set of second tracks;

- An association step of associating the first tracks obtained by the first radar sensor with the second tracks obtained by the second radar sensor to obtain a set of associated track pairs; in particular the association step is for associating one or more portions of the first tracks obtained by the first radar sensor and one or more portions of the second tracks obtained by the second radar sensor acquired during a same time interval thus obtaining associated pairs of time-aligned tracks tracked by the first radar sensor and time-aligned tracks tracked by the second radar sensor; the association step is based on the computation of an association cost; - A computing step of computing rotation and translation parameters of the first radar sensor with respect to the second radar sensor using all the associated track pairs.

By “targets of opportunity” it should be meant targets that are not cooperative with the system and are randomly moving in the environment without the system or the operators having any control over their movement. Examples of this can be people randomly passing by in the proximity of the radar sensors.

The computing step uses all the associated track pairs by stacking them into a single sequence. In other words, the computing step uses all the associated track pairs by stacking them into two respective sequences, one for the first set of tracks and one for the second set of tracks.

It should be understood that the set of first tracks and a set of second tracks may include a single fist track and a single second track. In this case each of the first stacked track and second stacked track corresponds to the respective first track and second track. The stacking operation is linked to the computing step detailed in the equation below.

In particular, the computing step is for computing the best rotation matrix R and translation vector t that minimize a computing cost between the stacked tracks, denoted by q _stack , from the first radar, and u _stack , from the second radar, according to the following least-squares cost function

M argmin > | |(Ru _st a _Cfe + t) - q _stack 11

7?eso(2),teiR ² m “=i where ||-|| represents a suitable norm function.

In the expression of the cost function SO(2) denotes the special orthogonal group of 2x2 matrices, M is the number of total time steps in the stacked tracks.

R and t represent the best rotation matrix and translation vector that match the trajectories of the stacked tracks u _stack from the second radar and the trajectories of the stacked tracks q _staC k, from the first radar, wherein the obtained rotation matrix and translation vector are, respectively, estimates of the relative orientation and position of second radar sensor with respect to first radar sensor.

Said tracking step, the association step, and the computing step may be repeated for different pairs of radar devices in the radar network that have partially overlapping fields of view (FoVs). Preferably when one radar has a partially overlapping field of view (FoVs) with all the other radars, the tracking step, the association step, and the computing step are repeated for all the possible radar pairs in the network. More preferably, the tracking step, the association step, and the computing step may be repeated sequentially according to any ordering of the radar pairs with overlapping fields of view (FoVs).

According to a preferred aspect of the invention, the association step includes a saving step to save a value of the cost function for each the first track and the second track, wherein the rotation matrix and translation vector minimize the cost between the tracks, according to the cost function.

The association cost function may be composed of terms that account for (i) the residual error of two tracks after the application of the roto-translation, (ii) the time synchronization between the two tracks, (iii) the length of the time-aligned tracks. The technique is then able to automatically evaluate the quality of the calibration by computing the residual matching error and re-calibrate if needed.

Preferably, the association cost function for the track pair (q, u) is computed as where p K) is a corrective term that depends on the length of the aligned portion of the tracks, K, according to (iii), y(r) is a measure of the quality of their alignment in time, according to (ii), while /(q, u) is the residual error between the tracks after the second track has been rotated and translated into the reference frame of the first radar, according to (i). E is a non-zero constant to avoid division by 0, preferably E = 1. Preferably, /(q, u) is the residual sum, according to a norm function, of the distances between the rotated and translated track of the second radar and the track of the first radar

Preferably, y(r) = 1 / (1 + T) where T is the mean time alignment error between the two tracks where and T/ ^are the timestamps associated to the k-th state in track q and u, respectively. Preferably, p(J ) = log KT, where T is the duration of a radar timestep. Preferably, rotation matrix R and translation vector t, in the computation of /(q, u), are computed as argmin

Reso(2),teK

Preferably, according to a threshold, the association step further includes a removal step of removing all the associated track pairs whose cost is above the threshold. Alternatively, the method includes a selection step of arbitrarily selecting one of the remaining track pairs.

It appears that preferably the computing step includes a stacking step of stacking together the associated tracks from the first radar into a q _stack and stack together the associated tracks from second radar, obtained in the removal step, into a u _stack . The stacking procedure consists in concatenating the states of all the tracks from the first radar together in a single sequence comprising all the tracks of the first radar. Denote by Qi, q ₂ - ■ ■ ■ - Qp the P tracks from the first radar that have been associated in the association step to the same number P of tracks from the second radar, called u ₁ ,u ₂ , ... ,u _P . The stacking step obtains q _stacfe [QI» Q2' ^{, , ,} ' QP] ^rid ^stack = [u ₁ ,u ₂ ,...,u _P ]

Noting that the number of components of each state in a track is 2, and denoting by L ₁ , L ₂ , ... , L _P the common number of timesteps in the tracks q ₁ ,q ₂ ,...,q _P and u ₁ ,u ₂ ,... ,u _P , the total number of timesteps in q _stack and u _stack is M = The stacked tracks q _stack and u _stack are thus M x 2 matrices.

Using the stacked tracks obtained in the stacking step, the computing step allows to compute the rotation matrix and translation vector that minimize the computing cost between the stacked tracks, according to the cost function.

The obtained rotation matrix and translation vector are, respectively, the estimates of the relative orientation and position of the second radar with respect to the first radar.

According to the present disclosure the term “track” can be intended as a time series of estimated target states.

A target should be intended as a moving object in the FoV of the radars. The target state may include, but it is not limited to, the target position on the horizontal plane parallel to the ground (x-y). Further states may be considered for the present disclosure.

The term “trajectory” may be intended as an ordered sequence of positions of a target in the physical space.

For example, target tracking at the local sensors is performed using Kalman filtering on local radar measurements. Local radar measurements include (but not limited to) the distance and azimuthal angle of the target with respect to the radar device.

It should be noted that local sensor tracks refers to the tracks (or stacked track sequences) collected by each sensor, having no knowledge of the global target states. The local tracks are again the same stacked tracks mentioned in claims 1-13. The adjective “local” refers to both the tracks and the sensors here. I think it’s better to define “local” in an earlier claim to avoid confusion

Preferably, the first track and the second track are aligned in time. Being aligned in time means that there must exist side information that makes it possible to identify which elements of the tracks are acquired at (almost) the same time instant. Therefore, before the association step, the method includes an alignment step for aligning track pairs obtained by the respective first radar sensor and second radar sensor.

The time alignment can be achieved in many ways. For example, possible ways are as follows: a) by means of a hardware trigger signal for the frame acquisition of the radar devices; b) by means of timestamp labelling of the data acquired by the radars; c) by means of a post processing of the data to find the optimal time alignment.

In other words, the alignment step may be carried out using time information about the tracks. Preferably, the time information comprise timestamps associated with each state of a local track time series. The track time series is said stacked track sequence. More preferably, the timestamps are used to align the tracks by finding an association between states of the two tracks that have been acquired at approximately the same time instant.

In a most preferred solution, the association is performed by optimizing a timealignment cost function of the timestamps associated with the states of the two tracks. For example, an optimization is performed associating each state of the first track, denoted by s _x , with timestamp with the state of the second track s _t whose timestamp is closest to T _X in terms of absolute difference. Specifically, s _x is associated to s _t where i satisfies argmin |T - T _X |. i

Timestamps may be obtained by means of a synchronization between the first radar sensor and second radar sensors. Alternatively, the synchronization may be performed via a network synchronization protocol, or the synchronization may be performed between the clocks of the radar sensors via hardware components. As a consequence, in the association step of the method of the present disclosure for all the combinations of tracks from the two radar devices, the portions of tracks that have been acquired during the same time interval are paired, forming a track pair.

For all the track pairs, the method provides a computing step of computing the rotation matrix and translation vector that allow obtaining the lowest cost between the tracks (preferably according to the specific cost function) once the trajectories of the second radar are transformed into the reference system of the first radar, using the rotation matrix and translation vector. The rotation matrix R and the translation vector t between the tracks, here denoted by q and u, are computed minimizing the following least-squares cost function argmin

Reso(2).teK

In the expression of the cost function, SO(2) denotes the special orthogonal group of 2x2 matrices, K is the number of time steps in the time aligned tracks.

More in particular, and preferably, the association step of association between local sensor tracks is obtained from an association algorithm, and during the computing step the algorithm associates each track from the first radar sensor with one or no track from the second radar sensor. In this regard, it should be noted that the association step happens between two sets of tracks collected at two different sensors. Some tracks from the first sensor can find no matching track from the other sensor, but this does not mean that the association step is not present. In this case, the association step is still performed, but it only returns the valid associations between some tracks of the first sensor and some tracks of the second sensor. The important thing is that it’s clear that the association happens in any case, as it is an operation between sets of tracks and not between single tracks.

In addition, as anticipated in the previous paragraphs, the association step includes the saving step of saving the value of the cost function for each pair and using a one-to-one association algorithm that takes as input the costs, and then an association step of associating the time-aligned tracks of the first radar with the time- aligned tracks of the second radar.

In other words, the one-to-one association algorithm minimizes a total association cost computed as the sum of multiple track pairs costs. More in particular the computation of the optimal association based on the minimum cost is performed using a one-to-one optimization algorithm. Preferably, the optimization algorithm is the Hungarian (or Khun-Munkres) algorithm.

More preferably, the track pairs costs are a function of a similarity measure between the states of a pair of aligned tracks. For example, the similarity measure between tracks can be a function of the sum of similarity measures between the single states of the aligned tracks.

More preferably, the similarity measure is a function of the Euclidean distance.

For example, the Euclidean distance between the states of the local tracks is computed after transforming the track pairs to have the same coordinate system and in particular by computing the rotation and translation parameters of one radar device with respect to the other.

According to an alternative embodiment, each track pair cost is a function of an alignment quality measure between the tracks in the pair. In particular, the alignment quality measure depends on a distance function of timestamp values attached to each local track state. Timestamps contain information about the time instant when the states were estimated/measured.

For example, the distance function is the average absolute difference between the timestamps.

Preferably, in the computing step, the rotation and translation parameters are computed solving a least-squares (LS) minimization between the stacked tracks of a reference radar q _stack and the associated, rotated and translated, stacked tracks of the other radar u _stack . In such preferred condition, the minimization finds the optimal rotation matrix R and translation vector t that minimize a computing cost between stack and u _stack , according to the following LS cost function

M argmin > | | (Ru _st a _Cfe + t) - q _stack 11

7?eso(2),teiR ² m “=i where ||-|| represents a suitable norm function.

In the expression of the cost function SO(2) denotes the special orthogonal group of 2x2 matrices, M is the number of total time steps in the stacked tracks.

For example, the LS minimization is solved efficiently via the singular value decomposition method. The singular value decomposition method computes the optimal rotation matrix R and translation vector t as

R = UV ^T and t = q - Ru. q = ^u k are the average values of tracks q and u. U and V are the left and right singular vector obtained by computing the singular value decomposition of matrix XY ^T = U .V ^T , where X = q - q and Y = u - u. S is the matrix of singular values of XY ^T .

The method according to the present disclosure, is also able to automatically evaluate the quality of the calibration by computing the residual matching error and re-calibrate if needed.

Preferably, the method further comprising the steps of evaluating a quality measure of the obtained rotation and translation parameters; repeating the calibration in case such quality measure is below a pre-specified threshold.

For example, the quality measure is a function of the residual error between the associated local tracks after the corresponding rotation and translation. The function of the residual error may be the root-mean squared error or the mean absolute error.

According to the present disclosure, local sensor tracks refer to the tracks (or stacked track sequences) collected by each sensor, having no knowledge of the global target states. The local tracks are again the same stacked tracks. In other words, “local” means that each single radar sensor does not have “global” knowledge about the tracks or the target states. The adjective “local” may refer to both the tracks and the sensors.

Detailed embodiment

According to an embodiment of the present disclosure, it is assumed that a tracking algorithm is available, for example, a Kalman filter. Therefore, a tracking step is performed.

The tracks obtained by the algorithm are aligned in time, by that meaning that there must exist side information that allows to identify which elements of the tracks are acquired at (almost) the same time instant. A possibility is to label each element of the tracks with the timestamp corresponding to the time instant when it is generated. For this, either a global or local time reference can be used: in case of a local time reference, a time synchronization protocol shall be used to align the local clocks. From now on, it is assumed that time synchronization with use of such timestamps is adopted.

If the radar devices are connected to more than one edge computer, the local time of all the edge computers must be synchronized. A possibility is to do this by exploiting a software protocol for clock synchronization such as the Network Time Protocol. A set of tracks coming from the same target(s) of opportunity and generated as described in the previous tracking step, time alignment step, and synchronization step is assumed to be available at each radar.

In particular, for all the combinations of tracks from the two radar devices, the method includes an identification step of identifying the portion of tracks which have been acquired during the same time interval, by exploiting the timestamps attached to the data. The identification is performed associating each state of the first track, denoted by s _x , with timestamp with the state of the second track s; whose timestamp is closest to T _X in terms of absolute difference. Specifically, s _x is associated to s _t where i satisfies argmin IT; - T- . i

Subsequently after the identification step, for all the portions of track pairs identified in the identification step, the method comprises the computing step of computing the rotation matrix R and translation vector t that make it possible to obtain the lowest cost between the tracks, according to some cost functions, once the trajectories of radar 2 are transformed into the reference system of radar 1 , using the rotation matrix and translation vector. The optimal rotation matrix R and translation vector t that yield the minimum quadratic error between the tracks, here denoted by q and u, are computed minimizing the following least-squares (LS) cost function

K argmin > ||(Riz _fe + t) - q _k \ \

Reso , tea. ²

In the expression of the cost function, SO(2) denotes the special orthogonal group of 2x2 matrices, K is the number of time steps in the time aligned tracks. The LS problem is solved using the singular value decomposition method, that computes the optimal rotation matrix R and translation vector t as

R = UV ^T and t = q - Ru. q = are the average values of tracks q and u.

U and V are the left and right singular vector obtained by computing the singular value decomposition of matrix T ⁷ = U .V ^T , where = q - q and Y = u - u. S is the matrix of singular values of XY ^T .

The method provides a saving step of saving the value of the cost for each pair. A cost function may be a function of the sum of distance functions between the single states of the aligned tracks, called /(q, u), and the distance function may be the Euclidean distance. /(q, u) may be computed as

The cost function can be also a function of the time alignment between the two time-aligned tracks and of the length of the tracks.

The function of the time alignment between the tracks may be computed as y(r) = 1 / (1 + T) where T may be the mean time alignment error between the two tracks and T/ are the timestamps associated to the k-th state in track q and u, respectively. The function of the length of the tracks may be computed as p(J ) = log KT, where T is the duration of a radar timestep.

The total association cost function for the track pair (q.u) may be computed as s is a non-zero constant to avoid division by 0, preferably £ = 1.

Then, the method includes a further association step of using a one-to-one association algorithm, such as the Hungarian (or Khun-Munkres) algorithm, that takes as input the costs from the saving step, the method provides the step of associating the time-aligned tracks of radar 1 with the time-aligned tracks of radar 2.

Subsequently, according to a threshold A _th , the method includes a removal step of removing all the associated track pairs whose cost function is above the threshold, that is, all track pairs (q, u) such that, A q,u) > A _th , are removed.

For each associated pair of tracks obtained from the removal step, in a stacking step, the method provides for stacking together the tracks from radar 1 into q _stack and stack together the transformed tracks from radar 2, obtained in the computing step, into u _{s ack} .

Using the stacked tracks obtained in stacking step, the method includes the computing step of computing the rotation matrix and translation vector that minimize the cost between the tracks, according to the cost function M argmin > 11 (Ru _stacfe + t) - q _stack 11

7?GSO(2),telR ² m “=l where ||-|| represents a suitable norm function.

In the expression of the cost function SO(2) denotes the special orthogonal group of 2x2 matrices, M is the number of total time steps in the stacked tracks.

As in the previous saving step, the cost function may be a function of the sum of distance functions between the single states of the aligned tracks and the distance function is the Euclidean distance.

The obtained rotation matrix and translation vector are, respectively, the estimates of the relative orientation and position of radar 2 with respect to radar 1.

Preferably it is possible to automatically assess whether the radar network is well calibrated or not. Therefore, the method, according to the present disclosure, allows for a check of the calibration status. In order to check the calibration, at operation time, the following steps are performed.

In particular, it is assumed that a calibration has already occurred and, so, that a rotation matrix and a translation vector are available and that they have been computed so as to transform data from radar 2 into the reference system of radar 1.

Subsequently the method includes a step of considering a set of tracks coming from the same target(s) of opportunity. For all the combinations of tracks from the two radar devices, the method includes again to repeat all the steps starting from the identification step.

In particular, an identification step of identifying the portion of tracks which have been acquired during the same time interval, by exploiting the timestamps attached to the data.

For each of the portions of track pairs identified in the identification step, compute the cost of the pair as the residual error of the Euclidean distances between the rotated and translated track from radar 2 and the track from radar 1 , using the rotation matrix R and rotation vector t in the previous calibration.

Then the method includes a saving step of saving the value of the cost for each pair.

Using a one-to-one association algorithm, such as the Hungarian (or Khun- Munkres) algorithm, that takes as input the costs from the saving step, the method includes an association step of associating the time-aligned tracks of radar 1 with the time-aligned tracks of radar 2. According to a threshold, the method includes a subsequent removal step of removing all the associated track pairs whose cost function is above the threshold.

It is possible to consider another threshold A _rec < A _th (lower than the one used in the removal step). If, for all the associated pairs of tracks from removal step, the cost is higher than the threshold, the system needs to be re-calibrated.

Subsequently as reported above a set of tracks are available, and it is possible to re-calibrate the system staring from the identification step.

The subject-matter of the present disclosure has been described hitherto with reference to embodiments thereof. It is to be understood that other embodiments relating to the same inventive idea may exist, all of these falling within the scope of protection of the claims which are attached below.