Introduction

It has been a long-time objective in the audio industry to increase listener engagement and immersion in recorded and subsequently reproduced sound. This quest was very much alive already in 1931 when Alan Blumlein invented stereo. Over the years sound quality and consequently immersion has gradually improved. Although various forms of surround sound were present earlier, in the seventies Dolby introduced Dolby Stereo which despite its name was the first commercially successful surround sound format. Surround Sound provided a higher level of immersion than previously attainable. In recent years object-based audio formats like Dolby Atmos and Sony 360 have emerged, increasing the level of immersion even further.

One of the major challenges connected to all surround formats is the reproduction of the surround sound field. Although a Dolby Atmos commercial theater with hundreds of speakers located around the cinema room can sound very impressive it is not practical to replicate such a setup in a private home. The industry has also struggled to create a convincing replication of a surround sound field over headphones. Despite considerable research efforts, the present-day technologies do not manage to produce a sound field that is perceived to be significantly outside of a listener’s head with headphones. The sound is typically felt to be mostly inside the head and not surrounding the listener as intended. Furthermore, the small amount of sound outside of the listener’s head is predominantly positioned to the immediate left and right of the listener’s ears or slightly behind. It is not possible to provide a stable front hemisphere location which is obviously very desirable.

To locate sound sources in space so called Head Related Transfer Functions (HRTF) are commonly applied. Surround sound produced for movies, video games etc. and many stereo recordings contain HRTF coding of sound. HRTF coding of location is present both in surround sound and stereo recordings and is suitable for both loudspeaker and headphone playback. Several playback algorithms, such as Dolby Atmos for headphones, also employ HRTF coding to locate sound.

Several HRTF databases containing measurements from hundreds of test subjects have been published on the web by the research community and are available for download. The databases typically contain frequency responses, Head Related Frequency Response (HRFR), associated with multiple locations around each test subject. Some databases also include the associated time domain response called Head Related Impulse Response (HRIR).

Typically, HRFR responses for hundreds of individuals are averaged to produce average HRFR for each location. The averaged HRFR data is then used for location coding of audio sources in recordings and playback.

As discussed earlier, this type of average HRFR coding does not produce convincing results over headphones and it requires a multitude of speakers spread around a room. Despite the averaging of measurements across many test subjects the perceived location also changes significantly from individual to individual.

Successful results are however achievable using individually measured HRIR for each listener. Convolving playback material with the individual HRIR using an ordinary FIR filter can create a fully realistic immersion in surround sound through headphones but only for the person whose personal HRIR is used during playback convolution. Producing individual HRIR data for everyone that is going to listen to a recoding is clearly not possible. Several attempts have been made to customize the commonly used average HRFR data from information about personal physical properties provided by the individual, but none have provided any breakthrough.

With HRIR the latency in the FIR filter also becomes a problem. For good results, the HRIR must be rather long and the latency introduced will cause significant problems in virtual reality, gaming and other similar applications where significant latency is unacceptable.

A successful straight forward averaging approach like HRFR averaging is also not possible in the time domain. Fig. 1 illustrates the difficulty of time domain HRIR averaging. Traces 1 , 2 and 3 in fig. 1 show HRIR data from three different test subjects. Due to different physical sizes, and associated sound wave travel times, the second bumps in the HRIR data occurs at different points in time in relation to the larger first arrival to the left on the traces. Trace 4 illustrates an average of 1 , 2 and 3. Clearly this is not a good average from the three physically different test subjects. Trace 2 would be the best average between the individuals’ sizes in this example but trace 4 does not look at all like trace 2. The three individual bumps on traces 1-3 have been smeared out in time. Instead of a clear wave front arrival at the average point in time, trace 2, the wave front has been time smeared and suppressed which is not the desired outcome.

The presented invention solves the location coding by breaking down the localization process in a novel manner introducing a new time domain focused approach. The approach is called Multiple Order HRTF. The approach allows averaging across individuals and with its time domain coding provides more stable localization of audio sources that is clearly positioned outside and in front, if desired, of the listener’s head through headphones. It is also possible to create virtual surround sound sources around a listening room using only two stereo speakers by embedding coded position information into the direct sound from the stereo speaker pair.

Summary of the present invention

The present invention is directed to a method for sound reproduction, said method comprising location coding with multiple order head related transfer functions (HRTF), wherein the method involves sound reproduction with at least a first order HRTF to the left ear and then a second order HRTF from the left ear to the right ear, and at the same time a first order HRTF to the right ear and then a second order HRTF from the right ear to the left ear. In relation to the above it should be mentioned that “multiple order” may imply second order, third order or up to any level of order. In relation to this it may also be mentioned that according to one embodiment, the method involves at least a third order HRTF going from the left ear to the right ear in the same way as from the right ear to the left ear, preferably at least a fourth order HRTF going from the left ear to the right ear in the same way as from the right ear to the left ear.

Furthermore, the concept according to the present invention is further described below and in relation to the figures, especially in relation to fig. 2.

Moreover, in relation to the present invention it should be mentioned that there are many known methods using several I multiple HRTFs, e.g. as disclosed in US2020/0037097, however this is not the same concept as disclosed and provided by the present invention. Again, the present invention provides a method comprising sound reproduction with at least a first order HRTF to the left ear and then a second order HRTF from the left ear to the right ear, and at the same time a first order HRTF to the right ear and then a second order HRTF from the right ear to the left ear. This should not be confused with using several I multiple HRTFs, which is utilized in many known methods.

Detailed Description of Multiple Order HRTF

It is well known from psycho acoustic research that human hearing is extremely sensitive to time domain properties of sounds. The sonic difference between wood and metal is heard in the first few milliseconds after a knock on the material. The startup waveforms of a violin and trumpet note are quite dissimilar and the difference is easily heard. However, if the sustained note from each instrument is heard without the startup it becomes difficult to differentiate between the two.

In the same manner sound source location is interpreted not only by HRFR but also from time domain information. Preceding localization solutions have focused on average HRFR data ignoring time domain information due to the discussed difficulties. The results have been less convincing. Individual HRIR data captures the time domain information but only for one individual at a time and manages to provide a good surround sound field impression for the individual in question.

Fig. 2 illustrates sound paths from a sound source to and around a listener’s head. Number 1 is the listener, 2 the sound source and 3 to 8 are visualized sound wave paths to and around the head. Fig. 2 only illustrates one sound source location but any location in three-dimensional space has got a similar set of imaginable sound paths associated with it. Fig. 2 shows the general principle and paths for other sound source locations should be easy to extrapolate.

Each sound path 3 to 8 have a time delay, a frequency response and attenuation associated with it. Path 3 has a time delay, the travel time of sound from the sound source 2 to the right ear but in this special case, since this is the first arrival of sound to the listener, the delay is zero as there is no need to have a delay that parallels the sound travel time to reach the listener. Attenuation in this specific first order path is also zero since the sound travels directly to the ear without any obstacles that can produce attenuation. The frequency response would typically be the well know average HRFR for the source location for the right ear. The sound wave will however not stop when it has reached the right ear. It will continue along the path 6 around the head to the left ear. This path has an interaural time delay due to sound travel time, a frequency response due to the shadowing of higher frequencies by the head etc. and attenuation caused by the travel around the head to the other ear. This second wave path is the second order HRTF. When the sound wave has reached the left ear, it will again continue to travel along the path 8 back to the right ear and once more this path has a time delay, a frequency response and attenuation associated with it. This is the third order HRTF. For reasons of clarity fig. 2 does not illustrate higher order HRTF, but the principle should now be obvious and it is easy to extrapolate any higher order HRTF by just continuing with the paths around the head. The time delays associated with the paths between the ears are directly tied to the physical distance between the ears and in the order of 200ps to 1ms, typically about 600ps. Frequency response alteration caused by the head when sound waves travel across it from one ear to the other is in general a down shelving of the higher frequency spectra beginning at 400Hz to 2.5kHz and continuing all the way up to the limit of human hearing at 20kHz and above. A few dips and peaks related to the specific path will also be present due to the physical properties of the human head and shoulders. Attenuation typically varies from 0-6dB in the first order path, 3-12dB in the second, 6- 24dB in the third and 9-48dB in the fourth. The methodology and techniques involved in obtaining the exact time delays and attenuations associated with each path should be straight forward for someone skilled in the art using standard methods and it is therefore not further discussed.

The frequency responses involved can be determined from readily available HRTF data. Fig. 3 shows the frequency response, as magnitude (dB) to frequency (Hz), associated with sound location 2, sound path 6 in fig. 2.

Acoustic measurements have shown that sound waves do propagate around an object several times as described and it is quite audible and clear that when the second, third and fourth order HRTF are added the sound is perceived as more natural and the localization of the sound source is greatly improved. Localization and naturalness become better and better as more orders are added up to fourth order after which the improvement becomes less noticeable. Any order of HRTF can of course be used from second to as many as one can imagine, hundreds or even thousands but as stated orders above fourth only provide small benefits.

The sound paths starting with path 4 from the sound source to initially the left ear also has time delays, frequency responses and attenuations associated with each of them like the paths described above starting with path 3. The delay along 4 is however not zero as with path 3, it is delayed due to the interaural time difference. The frequency alteration that occurs would again typically be the well know average HRFR for the sound source location for the left ear. Attenuation along path 4 is typically 4.5dB with the sound source located as shown in the example. The following second and third order paths 5 and subsequently 7 also have associated time delays, frequency responses and attenuations.

The sound path from one ear to the other across the front of the head is slightly longer than the path behind the head. It also produces a slightly different attenuation and frequency alteration than the sound path behind the head. Considering this it becomes obvious that the head and ears is an excellent localization device where different sound source locations would produce unique sets of Multiple Order HRTF sound paths. Consequently, Multiple Order HRTF makes it possible to achieve stable localization of sound sources both in front of and behind the head.

As Multiple Order HRTF separates the frequency response alteration, attenuation and time delay for each path averaging across test subjects become straight forward. Frequency responses for each path across many individuals can easily be averaged using familiar methods and attenuations and delays just become averages of attenuation and travel distances across test subjects for each path. Averaging of many individuals’ properties is crucial to achieve stable and similar results for all listeners.

The frequency alterations associated with each path can easily be implemented using standard HR filters eliminating the latency associated with FIR filters. Multiple Order HRTF thus works without any introduction of latency making the approach well suited to virtual reality, gaming and any other application that requires zero or extremely low latency. Fig. 4 contains a block diagram of a typical Multiple Order HRTF DSP implementation. A fourth order implementation for one sound source position is shown. It is of course possible and obvious that one can implement Multiple Order HRTF in many other ways and fig. 4 just shows an example of one of many possible topologies. Blocks 11 , 21 , 31 , 41 , 51 , 61 , 71 and 81 are delay blocks applying the delays associated with each set of four paths for each ear in the fourth order implementation. Block 12, 22, 32, 42, 52, 62, 72 and 82 apply the frequency alterations associated with each path. Block 13, 23, 33, 43, 53, 63, 73 and 83 are gain blocks applying attenuation present in each path. Finally, 100 is an adder block that is simply summing all outputs from the four paths to the left ear and 200 is the adder for the right ear. Outputs from 100 and 200 are sent to the respective left and right channels.

Applications utilizing Multiple Order HRTF can have both stereo and multichannel input signals. Multiple virtual sound sources can be created with Multiple Order HRTF. If the input signal is in an ordinary five channel surround sound format Multiple Order HRTF can be used to create five virtual speakers located in the usual positions of a five-channel surround sound setup i.e. front left and right, center and surround left and right. The discrete input channels are then played back by the corresponding virtual speaker. Similarly, more virtual speakers can be created for the latest surround sound formats involving more surround speakers and additional ceiling speakers. With a stereo input signal ordinary sound extraction and steering processes can be employed to extract the individual feeds to the virtual speakers. The stereo extraction and steering process would in this case be the same as in ordinary surround sound products.

The virtual sound sources created with Multiple Order HRTF works on both headphones and speakers. With headphones it is possible to create a surround sound field that approaches the experience using individually measured HRIR. On speakers it is possible to code virtual speakers into the direct sound from a pair of stereo speakers creating virtual center, surround and height speakers. With Multiple Order HRTF virtual speakers it is possible to create a surround sound field that is perceived to be similar to a setup with a multitude of speakers.

Playback using Multiple Order HRTF virtual sound sources is of course not limited to present day stereo or surround formats and their sound source locations. The examples above only illustrate possible Multiple Order HRTF applications and any number of virtual speakers in any position can of course be created as desired. Multiple Order HRTF can be applied at any stage from sound recording/- generation to playback, it is not limited to the playback stage. It is possible to use Multiple Order HRTF in design and/or production applying locations to sounds using Multiple Order HRTF that can later be played back on headphones, an ordinary stereo or multichannel playback system. Multiple Order HRTF can as an example be used within a gaming engine to locate sound within the generated sound field of a game. Another example is the use of Multiple Order HRTF within DAW software, either integrated or as a plugin, to locate sound within a sound field in sound production. In other words, the Multiple Order HRTF algorithm and sound processing can be applied at any stage providing the same end result.

Specific embodiments of the present invention

Below some specific embodiments of the present invention are presented.

According to one specific embodiment of the present invention, the method comprises at least a third order HRTF going from the left ear to the right ear in the same way as from the right ear to the left ear, preferably at least a fourth order HRTF going from the left ear to the right ear in the same way as from the right ear to the left ear.

Moreover, according to another embodiment, said method comprising creating one or more virtual sound sources by embedding coded position information into the sound.

According to yet another embodiment, each head related transfer function (HRTF) from a second order and upwards comprises the parameters time delay, frequency response and attenuation.

Furthermore, according to another embodiment, the method takes into account the difference for different sound paths, e.g. the difference of the sound path from one ear to the other ear in front of the head and the sound path in back of the head. In this regard it should be noted that the sound paths from one ear to the other ear may be any path around the head. Therefore, the method according to the present invention may involve several sound paths.

Moreover, according to yet another embodiment, the method comprises averaging. As disclosed above, according to the present invention, averaging is possible to perform across individuals. With time domain coding there is provided a more stable localization of sound sources that is clearly positioned outside and in front, if desired, of the listener’s head. Based on this, according to one embodiment of the present invention, the method comprises averaging being time domain focused. Furthermore, according to one embodiment of the present invention, the method comprises averaging of the parameters time delay, frequency response and attenuation independently of each other. This is yet a further difference when comparing to averaging performed in known methods used today.

The present invention is also directed to different types of systems, hardware and software implementations.

According to one embodiment, the present invention is directed to a headphone playback system arranged for using a method according to the present invention.

Furthermore, the present invention also refers to a speaker playback system arranged for using a method according to the present invention.

Moreover, the present invention is also directed to a playback system comprising a pair of stereo speakers, said system being arranged for using a method according to the present invention, for creating virtual surround sound sources around a listening room by embedding coded position information into the direct sound from the pair of stereo speakers.

Also other applications are possible according to the present invention, as clear from the description above. According to one such embodiment, the present invention refers to a gaming engine system arranged for using a method according to the present invention. According to another embodiment, the present invention provides a digital audio workstation (DAW) software system arranged for using a method according to the present invention.