Background of the Invention This invention relates to the fields of communications, audio and acoustics and more particularly to the registration and reproduction of sensory fields composed of sounds and images.

The concept of creating virtual sounds or images ^" has fascinated the human mind for many centuries. Thus far, the highest achievements in the audio field have been two channel stereo and multi¬ channel movie sound systems. In the visual field, holography and stereoscopic glasses have given many unfulfilled promises. In both fields, there is little room for significant improvement using the above-

described techniques in order to create more realistic, spatial sound and images.

Usually, sound is reproduced by the use of loudspeakers emitting audible sound waves, and the prevailing wisdom has been that the sound emitted by each loudspeaker should be the same as the sound emitted by the original source (guitar, voice, orchestra, etc.) . This leads to the unavoidable perception that the sound is coming from the loudspeaker, and does not allow for spatial positioning of the sound other than the position of the loudspeaker.

Sound registration is currently done by using microphones based on a diaphragm with relatively large size, statically suspended in a fluid such as air or water, capable of being moved in one dimension, by fluctuations predominantly m the audible range. The size of the diaphragm and its one dimensional movement have an averaging effect, limiting its ability to capture complex movements of the fluid (such as air or water) . The static positioning of the diaphragm causes it to be able to capture only movements of the fluid (such as air or water) sufficiently large to overcome its inertia. Visual reproduction is usually done by projecting a two dimensional image on a semi-flat surface (movie projection screen, TV screen, LCD, etc.) . Naturally, the image is restricted within the physical dimensions of the screen. So far, the few attempts to at least improve depth perception by the use of holography have been unsuccessful.

Visual registration is currently done by directly exposing, for a limited time, visible light sensitive material or elements to the reflected or

emitted source visible field. By its very nature, this technology is limited to capturing a two dimensional image and restricted by the size of its aperture. Additionally, this technology is capable of registering only visible with light intensities within a very limited range.

Tracking the position and movements of physical objects or sources in a fluid, such as air or water, is currently done by measuring the directly reflected visual (visual is used throughout this invention to connote light phenomena of any frequency, amplitude, phase or other parameters, regardless of its origin, position, direction or relations to other physical events) or sound tracking frequency (a frequency which either uses the Doppler effect to detect motion or uses the reflected chirp effect to determine distance) . This creates a number of limitations and problems related to direct visibility, sufficient amplitude of the measured reflection, resolution, etc.

Unless modified, "light" denotes any light phenomena, whether visible or invisible, of any frequency or amplitude.

"Registration" means capturing, processing, transmitting and storing of vibrational fields information. These fields are created by sources of sound and light energy.

"Scattering" is used to denote nonlinear interaction or interactions between sound, light or other wave beams or fields in a fluid such as air or water. In the case of such interaction, there are certain events that take place, such as generation of by-products (combinatorial and differential frequencies not existing in the originating sources) ; change m the

direction of sound, light or other phenomena; modification of the amplitude, phase, time delay, frequency and other parameters of sound, light and other waves; and many others. "Self demodulation" is used to denote nonlinear propagation through fluids of sound, light or other waves, whereas the wave shape and the wave contents change with the propagation distance.

"Multiple modulation" is a method for a complex modulation of sound, light or other waves including control of the source signal, temporal and spacial waveshapes, focusing and direction for the purpose of causing subsequent multiple demodulation. "Multiple demodulation" is a method for creating complex sound, light or other waves m fluids, in order to achieve particular scatterings and self demodulations, throughout the propagation distance and withm a certain spatial environment with certain directivity either with stationary or moving locations. "Lattice" is used to denote interaction fields of sound, light or other waves m fluids, whether or not a result of self-demodulation, multiple modulation, scattering, interference, linear or nonlinear phenomena, or any other wave phenomena or interaction.

A number of publications such as J. L. S. Bellin and R. T. Beyer, "Experimental Investigation of An End-Fire Array, " The Journal of the Acoustical Society of America (August 1962), L. Wallace Dean, III, The Journal of the Acoustical Society of America, (August 1962), Peter J. Westervelt, "Parametric Acoustic Array, " The Journal of the Acoustical Society of America (April 1963), Mark F. Hamilton and J. A. TenCate, "Sum and Difference Frequency Generation Due

to Noncollmear Wave Interaction m a Rectangular Duct, " The Journal of the Acoustical Society of America (June 1987), J. N. Tjotta and S. Tjotta, The Journal of the Acoustical Society of America (December 1990) , treat mostly the theoretical aspects of sound scattering of two elementary sound waves. The scientific community has generally accepted P. J. Westervelt's assumption that there is no scattered sound out of the interaction area, therefore there is not any known research on sound scattering of more than two sources in a fluid such as air or water.

Sound self demodulation in a fluid such as air is a direct result of the nonlinear characteristics of the medium. The nonlinear theory has dealt for many years with the theoretical aspects of self demodulation m different fluids in the publications of M. J. Lighthill, Proc. R. Soc. London (1952), H. 0. Berktay, J. Sound Vib. (1965), M. B. Moffett, P. J. Westervelt and R. T. Beyer, The Journal of the Acoustical Society of America (1970), M. F. Hamilton, J. N. Tjotta and S. Tjotta, The Journal of the Acoustical Society of America (1985) and many others. Masahide Yoneyama et al., "The Audio Spotlight," The Journal of the Acoustical Society of America (May 1983) describes the design of traditional loudspeaker utilizing ultrasonic self demodulation and elementary sound demodulation of a single array of sound reproducing elements, all fed with the same single modulated signal.

None of the prior art utilizes a combination of self demodulation, scattering and other nonlinear and linear phenomena m a fluid such as air for the purpose of creating stationary or moving localized sounds in the images with particular directivity. Additionally, there is no prior art on the subjects of

multiple sound and image modulation and demodulation, and on multiple sound and image scattering. None of the prior art provides means for creating or registering single or multiple sounds or images. In view of the foregoing, it would be desirable to provide a system and a method for dynamically changing a stationary or moving sensory field with particular location, composition or directivity. It would also be desirable to provide a sound system which utilizes multiple sound self demodulations and sound scatterings.

It would also be desirable to provide a method for individual sound demodulation and sound scattering.

It would also be desirable to provide a method for multiple sound demodulations and sound scatterings .

It would also be desirable to provide a method for a combination of individual and multiple sound demodulations and sound scatterings.

It would also be desirable to provide a sound system which utilizes sound demodulation and scattering. It would also be desirable to provide a method of multiple sound demodulation.

It would also be desirable to provide a sound system which utilizes the byproducts of multiple sound interactions m a fluid. It would also be desirable to provide a method of multiple sound scattering.

It would also be desirable to provide a sound system utilizing multiple sound scattering.

It would also be desirable to provide a method and a system for creating virtual sounds or sources utilizing nonlinear and linear events in a fluid. It would also be desirable to provide a method and a system for registration of sounds or sources utilizing nonlinear and linear events m a fluid.

It would also be desirable to provide a method and a system for creating virtual images or sources utilizing nonlinear and linear events m a fluid.

It would also be desirable to provide a method and a system for registration of images or sources utilizing nonlinear and linear events in a fluid.

It would also be desirable to provide a method and a system for tracking the position and movements of physical objects or sources utilizing nonlinear and linear events in a fluid.

It would also be desirable to provide a method and apparatus for producing and registering sound m proximity to an individual using the invention. It would also be desirable to reproduce sound for the user in one or more controllable locations m three-dimensional space.

It would also be desirable to register sound for the user from one or more controllable locations in three-dimensional space.

It would also be desirable to eliminate the need for the user to have one or more reproduction transducers near their ears.

It would also be desirable to reduce background, surrounding or other noise while registering without the need for a registering transducer to be near the source being registered. It would also be desirable to provide a device which can be either carried with the individual or placed in the environment near to the individual.

It would also be desirable to eliminate the need for the user to hold a communication device in their hand.

It would also be desirable to sense characteristics of the physical environment including the position and movement of individuals or other physical objects in the environment. It would also be desirable to reproduce and register sound m response to the position and movement of the user m the environment.

It would also be desirable to hold the location of sound sources m a fixed relationship with the user's head or body.

It would also be desirable to hold the location of sound sources in a fixed relationship with the physical environment.

It would also be desirable to reproduce sound m the proximity of the user's ears such that it can not be easily heard by anyone else.

It would also be desirable to reproduce sound in such a way that it can be heard similarly by other individuals in proximity to the individual with the device.

It would also be desirable to link two or more devices together such that the sound imagery perceived by the users is coordinated.

It would also be desirable to encode or decode spatial information about sound events being registered or reproduced.

Summary of the Invention It is an object of the present invention to provide a system and a method for dynamically changing a stationary or moving sensory field with particular location, composition or directivity.

It is also an object of the present invention to provide a sound system which utilizes multiple sound self demodulations and sound scatterings.

It is also an object of the present invention to provide a method for individual sound demodulation and sound scattering. It is also an object of the present invention to provide a method for multiple sound demodulations and sound scatterings.

It is also an object of the present invention to provide a method for a combination of individual and multiple sound demodulations and sound scatterings.

It is a further object of this invention to provide a sound system which utilizes sound demodulation and scattering.

It is also an object of this invention to provide a method of multiple sound demodulation.

It is also an object of this invention to provide a sound system which utilizes the byproducts of multiple sound interactions m a fluid.

It is also an object of this invention to provide a method of multiple sound scattering.

It is a further object of this invention to provide a sound system utilizing multiple sound scattering.

It is also an object of this invention to provide a method and a system for creating virtual sounds or sources utilizing nonlinear and linear events m a fluid. It is an object of this invention to provide a method and a system for registration of sounds or sources utilizing nonlinear and linear events in a fluid.

It is an object of this invention to provide a method and a system for creating virtual images or sources utilizing nonlinear and linear events m a fluid.

It is further an object of this invention to provide a method and a system for registration of images or sources utilizing nonlinear and linear events m a fluid.

It is also an object of this invention to provide a method and a system for tracking the position and movements of physical objects or sources utilizing nonlinear and linear events in a fluid.

It is an object of this invention to provide a method and apparatus for producing and registering sound m proximity to an individual using the invention. It is another object of this invention to reproduce sound with controllable directivity for the user in one or more stationary or moving controllable locations m three-dimensional space.

It is a further object of this invention to register sound for the user from one or more controllable locations m three-dimensional space.

It is a further object of this invention to eliminate the need for the user to have one or more reproduction transducers near their ears.

It is a further object of this invention to reduce background, surrounding or other noise while registering without the need for a registering transducer to be near the source being registered. It is a further object of this invention to provide a device which can be either carried with the individual or placed in the environment near to the individual.

It is a further object of this invention to eliminate the need for the user to hold a communication device in their hand.

It is a further object of this invention to sense characteristics of the physical environment including the position and movement of individuals or other physical objects in the environment.

It is a further object of this invention that sound reproduction and registering respond to the position and movement of the user m the environment.

It is a further object of this invention to hold the location of sound sources m a fixed relationship with the user's head or body.

It is a further object of this invention to hold the location of sound sources m a fixed relationship with the physical environment. It is a further object of this invention to reproduce sound in the proximity of the user's ears such that it can not be easily heard by anyone else.

It is a further object of this invention to reproduce sound m such a way that it can be heard similarly by other individuals m proximity to the individual with the device.

It is a further object of the invention to link two or more devices together such that the sound imagery perceived by the users is coordinated.

It is a further object of the invention to encode or decode spatial information about sound events being registered or reproduced.

This invention provides a system and method for creating and registering such previously impossible virtual sounds and images by utilizing separately or m combination various nonlinear as well as linear wave phenomena, such as self demodulation, scattering, multiple modulation, lattice, etc. The present invention is a sound system which utilizes ultrasonic sources. This sound system creates sound events m the audible range. These sound events occur at a location other than the location of the ultrasonic source. Thus, the sound produced does not come from a speaker. Rather, the produced sound event occurs at particular places in space and time with certain directivity or movement much like live sound.

The invention utilizes finite amplitude ultrasonic sources. These sources may be used alone or m combination with finite amplitude sonic sources. A controller manipulates the phase, time delay, frequency, temporal and spacial waveshapes, focusing, direction and amplitude parameters of the sources so that they interact with each other and create combinatorial and differential frequencies m particular locations of the acoustic field. These frequencies further interact with their byproducts, as well as with sonic frequencies to create a complex multi-dimensional acoustic field. The audible portion of this complex multi-dimensional acoustic field is what the human auditory system detects and perceives as sound. Thus, audible acoustic fields are achieved by changing the phase, time delay, frequency, temporal and spacial waveshapes, focusing, direction and amplitude

parameters of the sources. The sound events which occur in these complex multi-dimensional acoustic fields have particular location, directivity and/or movement and are perceived by the listener as live sound.

In one embodiment, a device is provided which enables an individual to, for example, hold a conversation with someone whose voice would appear to emanate from in front of the individual or listen to music which seemed to emanate from various locations around the individual. This device can create the sound of a whisper in the proximity of an individual's ear such that it would not likely be heard by anyone else. It can also reproduce sound m such a way that it can be heard by other individuals in proximity to the individual with the device. The device can register the individual's voice such that it can be sent and communicated elsewhere. The device can register sounds emanating from other specific locations around the individual such as from other people.

Because the device is capable of sensing the environment and the position of the individual with the device, it is also capable of adjusting its output in response to physical events. The device can, for example, hold the apparent location of a sound source, its directivity and/or movement fixed m relation to the environment even while the device is moving. It can, for example, hold the location of a sound source next to the listener's ear even while the head was moving. The device can also hold the location of a sound m a fixed direction relative to the environment and at a fixed distance even as the listener moves about. One application of the invention is to facilitate voice communication by subsuming into the

device the sound reproduction and registering functions typical of devices such as telephones, headsets, cordless phones, mobile phones, or any device for verbal communication. Thus, the device can be used to facilitate listening to music, speech, or any other sound by subsuming into the device the sound reproduction functions typical of devices such as home audio systems, portable audio systems, headphones, or any device for sound registration and reproduction.

Brief Description of the Drawings

The above and other objects and advantages of the invention will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings and m which:

FIG. 1 illustrates the evolution of a waveform distortion due to non-constant propagation speed; FIG. 2 illustrates a waveform distortion due to the addition of a quadratic term as a result of deviations from Hooke's law in a fluid;

FIG. 3 illustrates the effect of source wave¬ shape modification on the wave-shape evolution of ultrasound emitted in a fluid by a single, non-focused ultrasonic devices;

FIG. 4A illustrates a mechanical ultrasonic energy focusing device;

FIG. 4B illustrates an electronically controlled, array based, ultrasonic energy focusing device;

FIG. 5 is a diagram showing the wave-shape evolution of ultrasound emitted m a fluid by a single, non-focused ultrasonic device;

FIG. 6 is a diagram showing the wave-shape evolution of ultrasound emitted in a fluid by a single mechanical ultrasonic energy focusing device, as described in FIG. 4A;

FIG. 7 is a diagram describing the principles of sound registration according to the embodiment; FIG. 8 is a diagram describing the principles of visual reproduction according to the embodiment;

FIG. 9 is a diagram describing the geometry of non-collmear interaction of two primary sources and the mam direction of the difference frequency component;

FIG. 10 illustrates variations of the mam directivity of the difference frequency component as a function of the primary sources parameters;

FIG. 11 is a diagram describing the effort of non-collmear interaction of two primary sources on the difference frequency component radiation pattern;

FIGS. 12A and 12B show the spatial distribution of the pressure amplitude of one and two focusing sources, respectively; FIG. 13 is a diagram describing the pressure amplitude distribution along the axis of a focusing source;

FIG. 14 is a block diagram showing the construction of a device accordmg to this embodiment; and

FIG. 15 shows the modulation of a fluid by the field of a finite amplitude pump wave.

Detailed Description of the Invention

Generally, a fluid such as air possesses certain nonlmearities . When sound, light or any other wave travels through a fluid such as air, it changes its shape, direction or other parameters due to those nonlmearities. Additionally, when two or more waves interact in a nonlinear fashion, their shapes, directions or other parameters are further altered due to this interaction. For instance, an inaudible sound wave consisting of 50 kHz amplitude modulated by 1 kHz, due to the above mentioned nonlmearities and under the appropriate circumstances, will develop an audible 1 kHz component. Another example would be the nonlinear interaction between two otherwise inaudible sound waves with frequencies of 46 kHz and 48 kHz creating an audible 2 kHz component. Those and other nonlinear, as well as linear phenomena, occur withm a certain area of space and have certain directivity and/or movement. Once created, they become physical events on their own.

When a vibration of sufficiently large amplitude travels through a fluid, its wave shape changes due to the properties of the fluid. Some examples of wave shape changes due to fluid nonlmearities are given in FIGS. 1 and 2. This particular distortion phenomenon has been used to devise a narrow directional, low efficiency loudspeaker. See, e.g., Yoneyama et al., "The Audio Spotlight, " The Journal of the Acoustical Society of America (1983) , Kamakura et al . (10th ISNA, 1954) , Tannshi U.S. patent 5,357,578.

Source wave shape modification (illustrated m FIG. 3) can also be used to both statically and dynamically alter the resultant wave shape/s.

As nonlmearly induced wave shape changes are related mostly to the amplitude of the wave (e.g., sound pressure level in a fluid) , a desired wave-shape profiles at particular areas of space is achieved by utilizing single or multiple ultrasonic energy focusing devices m a fluid. Examples of these devices 10, 12 are shown in FIG. 4A and 4B along with a point source 14.

The use of multiple devices (illustrated m FIGS. 5 and 6) could be used to further alter the spatial amplitude distribution, resulting also in spatial alteration of the resulting wave shape/s. This control of the wave shape allows to produce spatially and dynamically varying scattering and self demodulation, resulting m various audible byproducts being produced, emanating outward from the area of non-collmear interaction, as described m FIG. 6.

FIG. 7 describes the principles of sound registration, by the use of array of transducers 20, signal source 21 and at least one pump transducer 22. The interaction of the pump wave and the signal wave, as registered by the array, provides spatial and dynamic information about the signal source. It should be noted that if biased transducers are used (as described in United States Patent Application No. 08/460,855) in a non-collinear (i.e., there is a non¬ zero angle between the two transducers) fashion, than the pump transducer may be discarded. The wave shapes of both the pump wave and the biased transducers could be modified to focus on a particular sound source or

extract additional information about the spatial wave energy distribution.

The spatially controlled waveshapes described above are essentially fluid compressions and rarefractions . As the refractive index of a fluid depends on the density, those compressions and rarefractions are used m FIG. 8 as acoustic refractive lenses to alter the direction of propagation and the properties of particular light waves. If used alone or m combination with multiple modulation and lattice, this method will produce visual images m a fluid without the need of screen.

When at least two, non-collinear waves interact nonlmearly, there is a differential wave radiated in a direction, determined by the vector equation, as described in FIG. 9. This direction can be changed by altering the parameters of the non- collmear interacting waves.

The non-collmear mteraction of two directional sources is illustrated m FIG. 10, where DI is the differential beam when kl > k2, when D2 is the differential beam when k2 > kl . It should be noted that the actual direction of the differential beam can be statically or dynamically altered spatially by using the method described above.

FIG. 11 describes the effect of non-collmear interaction of Gaussian primary beams on the differential wave farfield radiation pattern: φι=φ, φ2=0. The reduction of intensity associated with the increased φ can be altered by utilizing means for focusing the energy mto one or several areas m space, as illustrated in FIGS. 12A and 12B.

The normalized pressure amplitude along the axis of a focusing source is described in FIG. 13.

This provides means for shifting of the energy intensity maximum away from the transducer with the corresponding benefits for more efficient generation of differential wave with controllable direction, wave- shape and spatial profile. As the transducers are operating in continuous biased transducer mode, as described in United States Patent Application No. 08/460,855, an array based signal reception is simultaneously occurring with the transmission, allowing for receiving, sensing and tracking the physical environment.

FIG. 14 is a block diagram showing an example in which the personal audio communicator according to this invention is used in an electronic apparatus. The transducer array 100 is fed by signals from the amplification circuitry 200. The digital signal processor 300 is processing and modulating the signals received from the information processor 400, and is feeding the amplification circuitry 200. There are feedback loops 800 and 900 between the transducer array 100 and from the information processor 400, as well as m between the transducer array 100 and the digital signal processor 300. This provides the means for receiving, sensing and tracking the physical environment, as well as for steering and controlling the parameters and the physical location of the wave. The signal input block 500 receives the signal via universal signal input 600, and feeds the information processor 400. Devices 100, 200, 300, 400, 500 and 600 are controlled by a control device 700.

The use of ultrasound to create lower frequency sound by means of nonlinear interaction is also described m the scientific literature such as in: M. B. Moffett, P. J. Westervelt and R. T. Beyer,

- 2 Θ

"Large-amplitude pulse propagation -- A transient effect" Journal of the Acoustical Society of America (1970) and "Large-amplitude pulse propagation -- A transient effect II" by the same authors m Journal of the Acoustical Society of America (1971) . See also Michalakis A. Averkiou, Yang-Sub Lee, and Mark Hamilton, "Self-demodulation of amplitude- and frequency-modulated pulses in a thermoviscous fluid, " Journal of the Acoustical Society of America, (Nov. 1993) and Cormne M. Darvennes and Mark F. Hamilton,

"Scattering of sound by sound from two Gaussian beams, " Journal of the Acoustical Society of America (May, 1990) . These sources teach that the nonlinear interaction of two ultrasonic tones produce a difference tone which could fall withm the audible range. Subsequently, for every audible tone there must be a corresponding pair of ultrasonic tones separated by the same frequency.

Ultrasonic transducers for reproducing or registering ultrasound may actually include a number of ultrasonic transducers. These transducers may be functionally grouped together m an "array" which is a cluster of transducers having regular or irregular spacing. The direction and focus of the sound beam produced by an array is determined by the intensity, time delay, and phase of each transducer. See W.S.H. Munro and C. Wykes, "Arrays for airborne 100 kHz ultrasound," Ultrasonics, vol. 32, no. 1, and Bernard Steinberg, "Digital Beamformmg In Ultrasound" IEEE transactions on Ultrasonics, (Nov. 1992) . This device may include one or more arrays. Audible sound may arise at locations m space m which ultrasonic signals produced by one or more arrays is focused with

sufficient intensity to create non-linear by-products in a fluid such as air.

The invention is capable of reproducing a sound in a region of space which parameters, directivity and movement are determined by the signals sent to the transducers. The invention is capable of registering a sound in a region of space which is determined by the signals sent to the transducers. These arrays are capable of reproducing or registering multiple sound sources simultaneously.

While the invention is capable of creating direct sounds m the vicinity of the individual, the invention is also capable of creating the illusion of sound sources arising from any location m three- dimensional space. This is accomplished by creating acoustic signals at each ear that mimic the properties of signals arriving at the ears from a location m three-dimensional space as m everyday spatial hearing. These could be based on acoustic measurements of such signals that are often called head-related transfer functions, although it is recognized that other directional transfer functions can achieve the same effect.

The invention can incorporate means to sense the physical environment and to track the movement of individuals. This could be done in much the same manner as ultrasonic devices used in underwater radar applications, medical imaging, and non-destructive testing. For example, see the articles G. C. Gauhard and H. C. Strifors, "Time-frequency processing of underwater echoes generated by explosive sources, " Ultrasonics (1995), T. Niederdrank "Correlations of back scattered ultrasound from scattering suspensions in turbulent flow," Ultrasonics (1995), and others

which deal with reflected, backscattered or transmitted ultrasound and discuss different ways to interpret the results . One skilled in the art will recognize that the invention can incorporate controls which can be activated by any variety of means such as touch or voice, means to transmit and receive signals from other possibly different devices (m the case of a stationary device, this transmission can take place through a wire, for example), means to link together two or more devices, means to read from and write to storage devices such as sound recordings, means to encode spatial information about the sound events being registered, and means to decode spatial information about the sound events being reproduced. In addition, an encoding/decoding scheme can be incorporated mto a data protocol for transmitted or communicated information. This encoded information can be transmitted to or received from other devices and written to or read from data storage devices.

The introduction of a powerful ultrasonic pump wave in a fluid such as air alters the amplitudes of existing audible waves in a fluid such as air. The properties of a fluid such as air are modulated by the field of finite amplitude pump wave, as shown in

FIG. 15, and, if the conditions of phase synchronism are satisfied, the energy can be pumped in and out of the audible waves, as well as transferred from and to other frequencies. Some of the possible applications are selective noise suppression and signal amplification.

Thus it can be seen that a system and method for registration, reproduction and modification of sensory fields is provided whereby single or multiple

sounds or images can be created or registered have been described. One skilled in the art will appreciate that the present invention can be practiced by other than the described embodiments, which are presented for purposes of illustration and not of limitation, and the present invention is limited only by the claims which follow.