TECHNICAL FIELD

The present invention relates to determining the direction between a detection point and an acoustic source, and in particular a method and system for determining the direction between a detection point and an active speaker, such as a speaking participant in video conferencing.

BACKGROUND

Video teleconferencing systems are used to create virtual meetings between two or more people, or two or more groups of people, located in separate locations.

Determining the direction between a detection point and an acoustic source is useful in video conferencing. Such determining may, e.g., be used in automatic camera pointing in a video conferencing system. A typical situation at a video conference end-point (the terminal equipment at a video conferencing site) is a meeting room with a number of conference participants sitting in front of or around a table watching the display device of the end-point, while a camera positioned near the display device is capturing a view of the meeting room. If there are many

participants in the room, it may be difficult for those who are watching the view of the meeting room at a far end side to determine the speaker or to follow a discussion between several speakers. Thus, it would be preferable to localize the active speaker in the room, and automatically point and direct the camera onto that participant. In addition, the camera may be zoomed in order to obtain an appropriate view of the speaker.

One example of audio source localization in the background art is shown in U.S. Pat. No. 5,778,082, which describes, i.a., a method and a system using a pair of two spatially separated microphones to obtain the direction of an audio source. By detecting the beginning of the respective signals of the microphones representing the sound of the audio source, the time delay between the received audio signals may be determined, and the direction to the audio source may then be calculated. This principle has been illustrated in fig. 1 . Two microphones A and B arranged at a distance D receive an acoustic signal from the acoustic source C. The angle Θ of incidence represents the direction between a detection point (the mid-point between microphones A and B) and the source C. A time delay r represents the difference between the time of arrival of the acoustic signal at microphones A and B. This time delay is calculated as the maximum point of the cross correlation of the signals provided by microphones A and B, respectively. Herein, "maximum point" refers to the argument, i.e. the time, corresponding to the maximum value of the cross correlation of the microphone signals.

C T

The angle Θ is then calculated as Θ = arcsin wherein c is the sound velocity, r

D

is the calculated time delay, and D is the distance between the microphones.

The above-mentioned background art approach has certain drawbacks. In particular, noise generated by the microphones themselves has proven to have adverse effect on the resulting angle determination. Hence, it has been necessary to use expensive, high quality microphones in order to obtain a sufficiently accurate and reliable determination of the direction between the detecting point and the active speaker.

SUMMARY OF THE INVENTION

It is an object of the present invention to overcome at least some disadvantages of background art solutions. The invention is defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and its advantages will be readily obtained and understood when studying the following detailed description and the accompanying drawings. However, the detailed description and the accompanying drawings should not be construed as limiting the scope of the invention.

Figure 1 is a schematic diagram illustrating a principle of direction determining in the background art,

Figure 2 is a schematic diagram illustrating certain aspects of a method and a system for determining a direction between a detection point and an acoustic source, Figure 3 is a schematic block diagram illustrating principles of a first embodiment of a system for determining a direction between a detection point and an acoustic source,

Figure 4 is a schematic block diagram illustrating principles of a second

embodiment of a system for determining a direction between a detection point and an acoustic source,

Figure 5 is a schematic block diagram illustrating principles of a third embodiment of a system for determining a direction between a detection point and an acoustic source,

Figure 6 is a schematic flow chart illustrating principles of a first embodiment of a method for determining a direction between a detection point and an acoustic source, and

Figure 7 is a schematic flow chart illustrating principles of a second embodiment of a method for determining a direction between a detection point and an acoustic source.

DETAILED DESCRIPTION

In the following, the present invention will be discussed by describing preferred embodiments with reference to the accompanying drawings. However, those skilled in the art will realize that other applications and modifications exist within the scope of the invention as defined by the claims.

Figure 1 has already been referred to in the Background section above.

Figure 2 is a schematic diagram illustrating certain aspects of a method and a system for determining a direction between a detection point and an acoustic source.

In figure 2, it is assumed that an acoustic plane wave, originating from the acoustic source, impinges on a linear microphone array that is symmetrical about a central detection point 190. Thus, the microphone array includes a number of microphone elements which are arranged in-line, and the distance from the detection point to any of the microphone elements on one flank of the array is the same as the distance to a corresponding microphone element on the other flank. For explanation, four microphone elements have been illustrated: a first pair of microphone elements 1 10, 120, and a second pair of microphone elements 130, 140. Furthermore, for explanation, a Cartesian coordinate system is organized so that the microphone array and the position of the plane wave source span the x-y plane, and origin, corresponding to the detection point, is positioned in the center of the microphone array. In this example, in more general terms, there are M pairs of microphone elements in the microphone array, i.e. the total number of microphone elements is 2M.

The position of the i'th microphone is x i e{-M; -(M-l), ... (M-l), M).

In the example illustration of fig. 2, M=2, i.e. there are 4 microphone elements in the microphone array. Further, in fig. 2, x, = iA and M =2. However, M could assume any integer number 2, 3, 4, 5 or higher.

A mathematical deduction of the angle of arrival Θ, which is an expression of the direction between the detection point 190 and the acoustic source, will be presented in the following.

The sound pressure at the i'th microphone, at time t, is given by p(t, = s{t - ^{Xi si ( 6>)} ) * Oi(f , Θ) + m{t) (1) where s(t) is the time-dependent source acoustic signal, c is the speed of sound in the medium between the source and the microphones, usually air, cij(t, Θ) is the angular dependent impulse response of the i'th microphone, n,(t) the i'th

microphone's self noise, and * denotes the convolution operator.

Assume now that the time is considered at discrete points rather than as a continuous variable, and that that the microphone signals have been discretized (and possibly quantized, i.e. analog-digital-converted) at a sampling frequency 1/T _S .

Further assuming, for simplicity of explanation, that the impulse response of each microphone equals the dirac delta function, i.e. that aft, Θ) = S(t), leads to the following expression of the sound pressure at the i'th microphone: p(n, i) = s{[n - ^{Xi S {0)} }T _s ) + ni (n _s ) (2)

l _s c

Performing the Fourier transform leads to the following expression ω sin Θ

where ω= 2nfT _s and kx , c being the speed of sound.

c

Now, the direction between the direction point and the acoustic source, i.e. the angle of arrival, can be estimated, that is, Θ may be determined.

Complex conjugating the Fourier domain microphone signals in the negative x- plane (corresponding to utilizing the negative frequency components if s is real) and multiplying the signals by the remaining Fourier domain microphone signals in the positive x-plane leads to the array cross spectrum'

3 _χ (ω, θ) (4)

given that the microphone noise signals and sound source are uncorrelated (which is a reasonable assumption).

One can easily see that there will be an increased signal to noise ratio when the number of microphones increases. Calculating the inverse Fourier transform results in

R _S M is the autocorrelation function of the source convolved with itself M times. The angle Θ of arrival can be found by locating position of the maximum of Rx Θ . (6)

For the array configuration wherein M=2, i.e. in the case of two pairs of

microphones, the maximum can be found by θ— arcsin ^ ~^[ ^ατ 9 ^α ^χ _η Κ _χ (n)] J . (7)

The above reasoning is basically related to the configuration of microphones as schematically illustrated in figure 2. However, it should be understood that the above teachings, principles, deductions and assumptions may also be applied, individually or in combination as appropriate, when reading the following disclosure of embodiments of the method and system.

Figure 3 is a schematic block diagram il lustrating principles of a first embodiment of a system 100 for determining a direction Θ between a detection point 190 and an acoustic source. The acoustic source has not been illustrated. However, the line of incidence 184 illustrates the direction of the incom ing acoustic waves, originating from the source. It is assumed that the distance between the acoustic source and the detection point 190 is so much larger than any distance between microphones that it is reasonably to consider the angle of incidence Θ to be the same for all the microphones. The system comprises a first pair of microphone elements 1 10, 120, arranged symmetrically about the detection point 190, receiving acoustic signals originating from the acoustic source.

The system further comprises a second pair of microphone elements 130, 140, arranged symmetrically about the detection point 190, also receiving acoustic signals originating from the acoustic source.

In the illustrated embodiment, the first ( 1 10, 120) and second ( 130, 140) pairs of microphone elements are arranged in a linear microphone array 180, located at the detection point 190. More specifically, as will also be understood from the illustration in fig. 3, the microphone array 180 may be arranged in such a way that the microphone elements are located pairwise symmetrically about the axis of incidence 182 passing through the detection point.

As illustrated, the first ( 1 10, 120) pair of microphone element may be an inner pair of microphone elements, i.e. they are arranged closest to the detection point 190. Further, the second ( 130, 140) pair of microphone elements may be an outer pair of microphone elements, i.e. they are arranged at a larger distance from the detection point 190.

Each microphone element in the first pair of microphone elements is

communicatively connected to a respective input of a first cross correlator 210, which is configured to calculate a cross correlation of the two input microphone signals. As a result, the first cross correlator 210 outputs a first cross correlation signal.

Each microphone element in the second pair of microphone elements is

communicatively connected to a respective input of a second cross correlator 220, which is configured to calculate a cross correlation of the two input microphone signals. As a result, the second cross correlator 220 outputs a second cross correlation signal.

The expression "communicatively connected" has been used in order to emphasize that there may, or may not, be arranged intermediary circuits, processing elements etc. between the microphones and the remaining circuitry shown on fig. 3. Such intermediary circuits may, e.g., include any number of elements selected from amplifiers, filters, discretizing devices, quantizers, samplers, A/D-converters, signal processors, etc.

The first and second cross correlation signals are fed to respective inputs of a convolver 310, followed by a direction calculator 410. The convolver 310 and the direction calculator 410 are configured to calculate the direction between the detection point and the acoustic source based on the first cross correlation signal and the second cross correlation signal.

More specifically, the convolver 310 is configured to convolve the first cross correlation signal by the second cross correlation signal.

Even more specifically, in an embodiment, the convolver 310 is configured to perform the convolving of the first cross correlation signal by the second cross correlation signal in a frequency domain. Alternatively, the convolver may be configured to perform the convolving in a time domain. The direction calculator 410 is configured to calculate the direction between the detection point and the acoustic source by identifying a maximum point of the result of the convolving.

More specifically, the direction calculator 410 may be configured to calculate an angle of incidence, representing the direction between the detection point and the acoustic source, based on the identified maximum point.

The elements denoted as cross correlator(s), convolver(s) and direction calculator have been described as separate functional modules for the purpose of explanation of the principles of the invention. The skilled person will realize that such functional modules may be implemented as processing instructions in software. In this case the cross correlators, the convolver and the direction calculator may be software modules, controlling a processing device which inputs the input signals, processes them according to the present disclosure, and calculates a value representing the desired direction or angle as an output signal. The processing instructions may be held in memory or storage. In particular, the functional modules, and/or other parts of the functionality provided by the invention, may be implemented by means of a digital signal processor. Other alternatives may exist, for instance may special integrated circuits, such as FPGAs, be used for the practical implementation of the invention.

A special embodiment of the system has particular utilization in video conferencing. In such an embodiment, the acoustic source may be an active participant in a video conference, i.e., an active speaker in the conference. Further, the detection point coincides wholly or substantially with a location of a variable-direction video camera in a video conference equipment, such as a video conference endpoint. The system further comprises a camera controller, which is configured, i.a., to control the camera, in particular its rotation, e.g. about a substantially vertical and/or horizontal axis, to make it point in the direction calculated by the method and system disclosed, i.e. in the direction towards the acoustic source, which is in this case the active conference participant. This results in that an image of the active participant may be captured by the camera. In order to improve the capturing of the active participant's image, zooming may be used to select the proper detail of the image. Figure 4 is a schematic block diagram illustrating principles of a second

embodiment of a system 101 for determining a direction between a detection point 190 and an acoustic source.

The system 101 illustrated in fig. 4 corresponds to the system 100 illustrated in figure 3, and the same reference numerals have been used for illustrating identical or corresponding elements.

However, the system 101 illustrated in fig. 4 further comprises a third pair 150, 160 of microphone elements, which are arranged pairwise symmetrically about the detection point 190, and which receive acoustic signals originating from the acoustic source (not shown).

Further, the system 101 includes a third cross correlator 230, configured to calculate a cross correlation of signals provided by the third pair 150, 160 of microphone elements, resulting in a third cross correlation signal.

Further, in system 101 , the convolver 310, in this example referred to as a first convolver 310, a second convolver 320, and the direction calculator 410 are configured to calculate the direction between the detection point and the acoustic source based on the first, second and third cross correlation signal.

More specifically, the first convolver 310 is configured to calculate the convolution of the first and second cross correlation signals. The result of this convolution, referred to as a first convolved signal, is fed as a first input to the second convolver 320. The third cross correlation signal is fed as a second input to the second convolver 320. The output of the second convolver 320 is fed as input to the direction calculator 410, which is configured to calculate the direction between the detection point and the acoustic source by identifying a maximum point of the result of the output of the second convolver.

It will be understood that it is possible, and within the scope of the present invention, to re-use the principle of adding another pairs of microphone elements in the linear microphone array, and adding further cross correlators which calculates the cross correlation of the added pair of microphone elements, and modifying the arrangement of convolvers and the direction calculator in order to calculate the direction calculator's output signal, indicating the value of the direction or angle of incidence, based also on the output of the further cross correlators. Thus, in the case of M pairs of microphone elements, wherein M is an integer greater than 2, the M pairs of microphone elements are arranged pairwise symmetrically about the detection point, receiving acoustic signals originating from the acoustic source. The system then also includes M cross correlators, referred to as the first, second, ... and so on up to the M-th cross correlator. Each cross correlator is configured to calculate a cross correlation of signals provided by the respective pair of microphone elements, resulting in M cross correlation signals.

Further, M-l convolvers are arranged in the system. The first convolver is configured to calculate the convolution of the first and second cross correlation signals, resulting in a first convolved signal. The second convolver is configured to calculate the convolution of the first convolved signal and he third cross correlation signal, and so on. The (M- l )-th convolver is arranged to calculate the convolution of the (M-2)-th convolved signal and he M-th cross correlation signal.

Further, still in the case of M pairs of microphone elements, the direction calculator 410 is configured to calculate the direction between the detection point and the acoustic source based also on the M-th cross correlation. More specifically, the direction calculator is configured to calculate the direction between the detection point and the acoustic source based on the output of the (M-l )-th convolver, as specified above.

Figure 5 is a schematic block diagram illustrating principles of a third embodiment of a system 102 for determining a direction between a detection point 190 and an acoustic source.

The system 102 corresponds substantially to the system 101 illustrated in figure 4, and the same reference numerals are used for identical or corresponding elements on the figures. However, in figure 5 it has been indicated that the distance between adjacent microphones is not necessarily the same, as long as each pair of

microphone elements is arranged in-line and symmetrically about the detection point 190 (or symmetrically about the axis of incidence crossing the detection point 190). For instance, the distance between microphone elements 1 10 and 130 (which equals the distance between microphone elements 120 and 140) may be less than the distance between the microphone elements 130 and 150 (which equals the distance between the microphone elements 140 and 160). Other possibilities exist as well, as will be realized by the skilled person.

Figure 6 is a schematic flow chart illustrating principles of a first embodiment of a method for determining a direction between a detection point and an acoustic source.

The method starts at the initial step 600.

In the first receiving step 610, acoustic signals originating from the acoustic source are received at a first pair of microphone elements, which are arranged

symmetrically about the detection point.

The first and second pairs of microphone elements may be arranged in-line, in a linear microphone array located at the detection point, in such a way that the microphone elements are located pairwise symmetrically about an axis of incidence that passes through the detection point. The first pair of microphone element may be an inner pair of microphone elements, and the second pair of microphone elements may be an outer pair of microphone elements.

Next, in the second receiving step 620, acoustic signals originating from the acoustic source are received at a second pair of microphone elements, which also are arranged symmetrically about the detection point.

Next, in the first cross correlation signal calculating step 630, a first cross correlation signal is calculated as the cross correlation of the signals provided by the first pair of microphone elements.

Next, in the second cross correlation signal calculating step 640, a second cross correlation signal is calculated as the cross correlation of the signals provided by the second pair of microphone elements.

Cross correlation signals may conveniently be calculated in frequency domain, by first calculating a Fourier transform of each microphone signal, and by multiplying the Fourier representation of one microphone signal with the complex conjugated Fourier representation of the other microphone signal in the microphone pair..

Calculating the inverse Fourier transform of this product yields the time domain cross correlation of the microphone signals. Alternatively, the cross correlation signal may be calculated in time domain, by convolving one microphone signal with a time reversed signal provided by the other microphone in the microphone pair.

Next, in the convolution calculating step 650, the convolution of the first cross correlation signal and the second cross correlation signal is calculated, resulting in a first convolved signal.

Such a convolving of the first cross correlation signal by the second cross correlation signal may be performed in a frequency domain. More specifically, if the cross correlation signal has been provided by a Fourier transform, the

convolution may conveniently be performed in the Fourier domain (frequency domain) my multiplication, followed by inverse Fourier transforming of the result. Alternatively, the convolving may be performed in a time domain.

Next, in the direction calculating step 660, the direction between the detection point a and the acoustic source is calculated, based on the first cross correlation signal and the second cross correlation signal. More specifically, the direction is calculated based on the first convolved signal resulting from step 650.

The direction calculating step 660 may, in an embodiment, include identifying a maximum point of the first convolved signal. The maximum point represents the argument, i.e. the time, corresponding to the maximum value of the first convolved signal. Identifying the maximum point may be performed by a search approach.

More specifically, in the direction calculating step 660, an angle of incidence, which represents the direction between the detection point and the acoustic source, may be calculated based on the identified maximum point.

Further details about the method, in particular further aspects of calculating the angle, i.e. the direction to the acoustic source, have been disclosed above with reference to figure 2.

It will be understood that the illustrated sequence of steps in figure 6 has been presented for explanatory purposes, and that the same result may be achieved with a sequence that departs from what has been illustrated. For instance, the steps 610 and 620 of receiving acoustic signals may be performed simultaneously or concurrently. If the receiving steps are performed at distinct point of times, the time difference between them should be kept small compared to the nature of the acoustic signals in question. In this case, the sequence of the receiving at the first and the second pair of microphone elements may be interchanged. Likewise, it will be understood that the calculating of first and second cross correlation signals may be performed in any order, or simultaneously or concurrently, if desired.

Figure 7 is a schematic flow chart illustrating principles of a method for

determining a direction between a detection point and an acoustic source, using first, second and third pairs of microphone elements.

The first, second and third pairs of microphone elements may be arranged in-line, in a linear microphone array located at the detection point, in such a way that the microphone elements are located pairwise symmetrically about an axis of incidence that passes through the detection point, e.g. in the manner as described above with reference to figures 4 or 5.

The method starts at the initial step 700.

In the first receiving step 710, acoustic signals originating from the acoustic source are received at the first pair of microphone elements.

Next, in the second receiving step 720, acoustic signals originating from the acoustic source are received at the second pair of microphone elements.

Next, in the second receiving step 730, acoustic signals originating from the acoustic source are received at the third pair of microphone elements.

Next, in the first cross correlation signal calculating step 740, a first cross correlation signal is calculated as the cross correlation of the signals provided by the first pair of microphone elements.

Next, in the second cross correlation signal calculating step 750, a second cross correlation signal is calculated as the cross correlation of the signals provided by the second pair of microphone elements.

Next, in the third cross correlation signal calculating step 760, a third cross correlation signal is calculated as the cross correlation of the signals provided by the third pair of microphone elements. Cross correlation signals may conveniently be calculated in frequency domain or in the time domain, e.g. as described above with reference to figures 3, 4, 5, or 6. Next, in the first convolution calculating step 770, the convolution of the first cross correlation signal and the second cross correlation signal are calculated, resulting in a first convolved signal.

Next, in the second convolution calculating step 780, the convolution of the first convolved signal and the third cross correlation signal is calculated, resulting in a second convolved signal.

The calculating of convolved signals may be performed in frequency domain or in the time domain, e.g. as described above with reference to figures 3, 4, 5, or 6.

Next, in the direction calculating step 790, the direction between the detection point and the acoustic source is calculated, based on the first cross correlation signal, the second cross correlation signal and the third cross correlation signal. More specifically, the direction is calculated based on the second convolved signal resulting from step 780.

The direction calculating step 790 may, in a corresponding manner as described with reference to fig. 6, include identifying a maximum point of the second convolved signal.

In still another embodiment of the method, acoustic signals originating from the acoustic source are received at M pairs of microphone elements which are arranged pairwise symmetrically about the detection point, wherein M is an integer greater than two. The integer M may, e.g., be 3, 4, or 5. Alternatively, M may be another integer, greater than 5. The cross correlation of signals provided by each pair of microphone elements are calculated, resulting in M cross correlation signals.

Further, the step of calculating the direction between the detection point and the acoustic source is based on the M cross correlation signals. In such a case the method includes M steps of calculating cross correlation signals, each step corresponding to a respective pair of microphones. Such steps may be referred to as the first, second, ... and so on, up to the M-th step of calculating cross correlation signals. The M steps of calculating correlation signal results in M cross correlation signals. Further, M-l steps of calculating convolution are included in the method. The first step of calculating convolution calculates the convolution of the first and second cross correlation signals, resulting in a first convolved signal. The second step of calculation convolution calculates the convolution of the first convolved signal and he third cross correlation signal, and so on. Finally, the (M-l)-th step of calculating convolution calculates the convolution of the (M-2)-th convolved signal and he M- th cross correlation signal.

Further, still in the case of M pairs of microphone elements, a step of calculating a direction calculates the direction between the detection point and the acoustic source based also on the M cross correlation signals. More specifically, the direction may be calculated based on the output of the (M- l)-th convolving step, as specified above.

Thus, the convolving procedure may be used for the next outer microphone element pair, recursively, in combination with the already calculated convolved cross correlations of the signals provided by the inner microphone element pairs in the linear, symmetrical microphone array.

In a particularly useful application of the method, the acoustic source is an active participant in a video conference, and the detection point coincides with a location of a variable-direction video camera in a video conference equipment. In this case, the method further comprises controlling the camera to point in the calculated direction for capturing the active participant by the camera.

As described above, the method and system for determining the direction between a detection point and an acoustic source is particularly useful in video conferencing, e.g., for automatic camera pointing towards an active participant (a speaker) in a video conferencing system. However, the determining of direction between a detection point and an acoustic source may be used in several other applications as well, e.g., in the production of TV programs or movies, in stage settings in theatrical shows or during music performance, and in surveillance, monitoring or security technology.

The present invention results in a higher signal to noise ratio, compared to the conventional time delay technique for angle of arrival estimation referred to in the Background section. Thus, the invention allows use of cheaper microphones with higher noise levels, without sacrificing overall performance and accuracy.

Alternatively, the use of high quality microphones with the principles of the present invention allows better performance and accuracy in the determination of arrival direction, also in cases where the source has a weak output signal, or if the source is located at a substantial distance from the detection point.

It should be understood that the described method and system are completely corresponding, and that any feature that may have been described specifically for the method, should be considered as also being disclosed with its counterpart in the description of the system, and vice versa.

Numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.