;. A portion of the disclosure of this patent document

^• ^ contains material which is subject to copyright

5 protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

10 BACKGROUND

The present invention relates to an apparatus and method for generating a control signal in response to movement, and more particularly, relates to a music generator that extracts information for "conducting" 15 music, in the same sense that a conductor would conduct an orchestra.

People enjoy music. Many especially enjoy classical music and appreciate the role of the conductor. Typically, a conductor will orchestrate music individually 20 played by over one hundred independent instruments into one united harmonious score. The styles and products of different conductors are as unique as the music scores that they conduct.

The conductor is viewed as the head of the

25 orchestra, its leader, and frequently, is the recipient of praise for his creativity and his ability to transform the work of a composer into a derived, artistically unique i musical product.

Many have fantasized being a conductor and being

30 able to create such a unique musical creation. However, for most, this fantasy will not be achieved, because of

the difficulty in learning and in mastering the conductor's unique "language".

The conductor must be fluent in expressing conducting information, used to produce the final musical product, to each member of the orchestra. This information is transmitted through the gestures and movements of the conductor and is based upon his knowledge and experience in music in general, his knowledge of the music score which is to be performed, his own style and taste, and the knowledge and experience and style of each member of the orchestra. Learning the conductor's musical experience and the other information necessary to conduct an orchestra is fairly difficult on its own, but in addition, the layman must also learn the language by which a conductor communicates his expression and taste to enable the members of the orchestra to play in unison at the correct tempo, volume, emphasis and presence.

Thus, the conductor's musical skills necessary to synchronize and direct the playing of music typically exceed those of the common person. However, this inability does not eliminate the desire that many have to express their own musical taste and style. To this end, equipment and methods have been developed over recent years which attempt to enable a common person to step into the shoes of the conductor, and to create a unique musical product, stylized by their own expression.

One such system, employed for many years at "EPCOT Center," at Walt Disney World, Florida, enables one to simulate the role of conductor by mixing instrument tracks that correspond to a prerecorded musical score. Using this system, one assumes a defined spot and moves and gestures as if he or she were the conductor. Four sonar devices are used to each derive the relative distance of a hand intersecting a sonar beam to a background object

normally struck by the beam. This relative distance is then used to raise or lower corresponding tracks of a prerecorded musical score.

Another method utilizes video processing to isolate 5 the outline of the form of a guest conductor against a distinguished background and to derive information from the orientation of the person's outline. According to this method, the repeatedly captured outline can be analyzed for movement, and direction or speed of movement 10 determined from recognized states can be extracted and used or analyzed to modify sound.

Still another system attaches motion sensors to a guest conductor. These sensors, which may include, for example, motion sensing gloves or the like, sense 15 acceleration or movement relative to other sensors, and thereby provide an electronic signal that can be used to generate music.

Each of these systems has disadvantages that offer room for improvement and modification. For example, in 20 the video system mentioned above, the guest conductor and a background must be specifically contrasted in order to allow the video equipment to provide a stark contrast to capture the guest conductor's outline.

Other difficulties are also present in a video 25 device. Video systems generally use image processing equipment that transforms the video signal into "pixels", i.e., numbers that each represent sampled visual characteristics at distinct locations scanned by a video

' / camera. These video systems produce many thousands of 30 "pixels" per second. Typically, the known methods of processing these pixels, such as by tracing only the outline, must rely on some shortcut in order to track movement. The huge number of pixels and complex

processing that is required make full image processing a practical impossibility for real time applications.

With respect to the sonar system mentioned above, its field of view is extremely limited since the sonar waves are either directed to, or are received from, a specific point in order to accurately locate the guest conductor's hands and to distinguish them against other sound reflecting backgrounds. Furthermore, the nature of the sonar system dictates that only a limited range of guest activities will produce results. This limitation in the range of activities detracts from the guest's freedom of expression and makes it more difficult for a guest to create music by imitating his or her mental impression of a conductor in action.

Accordingly, there has existed a definite need for an apparatus or method which can generate a control signal in response to movement and use that signal to alter the performance parameters of prerecorded music. Such a system would need to track movements which occur within a field of view. Additionally, it should provide a method for processing that employs a reference back to specific, previously identified pixels derived from a video image to enable a digital processing system to process the movement in real-time. This needed apparatus or method should be applicable in particular to a device that permits the controlled performance of music in response to tracked conductor parameters.

The present invention fulfills these needs, overcomes many of the aforementioned disadvantages and provides an improved and unique apparatus and method for playing music and for allowing a guest conductor to direct a musical score. In broader terms, however, the present invention provides a unique and novel apparatus and method for generating control signals which may be applied to a

wide variety of tasks by tracking movements that occur within a field of view.

SUMMARY OF THE INVENTION

The present invention provides a novel control 5 system that generates a control signal by tracking movement within a field of view, and which can track gestures and movements of a person occupying that field of view. In this manner, movement may be utilized in a wide variety of applications to orchestrate system control

10 through derived electronic signals. In particular, the specific contemplated application of the invention is to a Guest Controlled Orchestra that permits a guest to step into the shoes of an orchestra conductor and direct the performance of music that tracks his movements, much as a

15 real orchestra would a professional conductor.

The invention provides a digital motion processing system having an imaging device that repeatedly captures a field of view, which is then processed by image and data processing elements to yield relative movement between

20 frames. From the comparison of the two images, the processing elements determine a single value representative of positions of the current image that correspond to movement. In response to this single value a signal generator generates at least one control signal.

25 Since the preferred embodiment is a music system, the signal generator may be included in a sound generator that uses the generated control signals to generate and format sound.

The method of processing information from a field of

'(

30 view to generate this control signal includes generating and storing a first image representing the field of view at a first point in time, and generating a second image representing the field of view at a second different point

in time. Pixels from these two images are compared to determine a set of pixels that represent movement of the second image relative to the first image. From the locations of those pixels in the set that represent movement, a single value is computed. The control signal is generated in response to this single value.

More particularly, the apparatus utilizes an imaging device to provide an electronic signal representative of a field of view captured by the imaging system. This electronic signal, representing a sequence of scans of the field of view, is digested by an image digitizer that processes the electronic signal to yield a sequence of numbers. Each number corresponds to the intensity of a particular point in the field of view and thus represents a specific, addressable location, or "pixel", within that field of view. This numeric data is then processed in a series of steps to efficiently permit the tracking of movement within the field of view.

The intensity of pixels of an image are compared with the intensity of a previous image at the same pixel location to determine those pixels that represent movement relative to the previous image. A difference in corresponding pixel values that exceeds a selected threshold is taken as an indication of motion. The data processor computes a single value representative of all the pixels that represent movement for a given image.

At a more specific level of the invention, the data processor computes a single position, or "centroid", for each "window" or predefined subportion of the field of view which is to be specifically analyzed for movement. It then computes, from all such single positions, the single value, or "scalar" _r which it then analyzes to generate the control signal. For example, in the case of the preferred embodiment described below, x and y indices

of two such centroids for each image are simply summed to yield a single number, or "scalar", representing movement within the field of view.

Following a more particular form of the invention, after processing several image frames, the relative magnitude of several of the single values representing recent frames is correlated over a larger group of the single values to determine how fast or slow prerecorded music should be played. By recognizing specific guest actions, more particular forms of the invention also provide for selection of different instruments which can be alternatively used to play the same notes, and for volume control. The latter is dependent upon the number of pixels for a given image frame that represent movement.

Another feature of the motion processing system in accordance with the invention provides for control of a video display in response to tracked movement. For example, a prerecorded video image, such as a cartoon, may be visually displayed to the user or others at a rate that tracks the user's movements occurring within the field of view. This latter embodiment correctly emphasizes the current invention as providing a method of generation control signals in general, with many possible applications.

The invention may be better understood by referring to the following detailed description, which should be read in conjunction with the accompanying drawings. The detailed description of a particular preferred embodiment, set out below to enable one to build and use an example of the invention, are not intended to limit the claims but to serve as a particular example thereof.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an electronic guest controlled orchestra system in accordance with the invention;

FIG. 2 is a perspective view of the system illustrated as in FIG. 1, with the components of a console shown in greater detail;

FIG. 3 is a flow chart illustrating operation the main processing functions of a data processor used in the guest conducted orchestra system shown in FIG. 1;

FIG. 4 is a flow chart ^" of the pixel sampling function, referred to in FIG. 3, that illustrates in greater detail the steps used to accomplish pixel sampling and centroid computation;

FIG. 5 is a flow chart of a background schedule filling process for the MIB that operates in parallel with the main processing of the data processor and that controls the loading of MIDI note commands from a computer to the MIB used in the guest conducted orchestra system of FIG. 1;

FIG. 6 is a flow chart which illustrates in greater detail a correlation process referred to in the flow chart shown in FIG. 3;

FIG. 7 is a flow chart which illustrates maxima detection in accordance with FIG. 6 ; and,

FIG. 8 is a flow chart which illustrates minima detection in accordance with FIG. 6.

DETAILED DESCRIPTION

In accordance with the principles of the invention which have been summarized in the section above, the inventors have developed a preferred implementation of their invention which will be further described below. This preferred embodiment is called a "Guest Controlled Orchestra", and allows a guest to imitate the actions of a conductor, and to observe how those actions affect the generation of music and the display of animation. The "orchestra" may be an ensemble of at least one orchestral voice, which may be an instrument or voice, or any other means of generating sound. Thus, the preferred embodiment may be applied to allow the guest to conduct nearly any ensemble, from a single instrument to a choir, to a rock group, etc. However, it is emphasized that the invention, as described above, relates not just to a music system, but is more broadly a system for generating control signals from movement.

The preferred use of the Guest Controlled Orchestra is in a theme park, and for that reason, it has been designed with smooth and simple guest interaction and throughput in mind. Pursuant to these criteria, the system has been designed to provide intuitive operation to a guest, and so does not require an employee/operator or specific instruction before use.

Referring first to FIGS. 1 and 2, the Guest Controlled Orchestra 10 includes a conductor station 12, on which a guest 14 may stand to perform conducting movements with his or her arms, and a console 15 spaced a predetermined distance D from the conducting station and facing toward the person. The console 15 conceals from the person a monitoring means for optically scanning movements of the guest and which is aligned towards the

conductor station for this purpose and a computer processing means for digesting the scanned movements of the guest and generating and applying control signals. The console 15 also conceals a music storage means for storing electronic digital information that represents a musical piece to be played and audio playing means for playing the musical piece in response to the electronic digital information, as modified by the computer processing means.

More particularly, the system 10 includes a video camera 16, aligned to monitor the guest 14 through an infrared window 18, a computer 20 for performing image and data processing tasks, a MIDI synthesizer 22, which converts MIDI format music information supplied by the computer into a signal for mixing by a mixer 24 and audio amplification by an amplifier 26, and a speaker system 28 for ultimate rendition of a prerecorded musical score that has been varied in accordance with the guest's conducting actions. The computer 20 is fitted with an image digitizer add-on board for digitizing the camera's analog video output into a form that can be read and processed by the computer. In addition, the computer is also fitted with a musical/personal computer interface add-on board that controls the timing of music commands for the MIDI synthesizer. A video disk player 30 and monitor 32 are coupled to the computer to permit display of animation that is synchronized to the guest's tempo, as well as an additional monitor 34 that displays a special viewing image that represents detection of movement by the system 10.

The system 10 also includes a conductor statuette 36 and appropriate suggestive materials, illustrated in FIG.

1 as signs and notes 38, to indicate to the guest to imitate the actions of a conductor in order to achieve the desired result of generating music. It has been found

that such suggestive materials are necessary and sufficient to cause the guest to intuitively operate the Guest Controlled Orchestra.

Within the conductor station 12, the guest occupies a field of view which is scanned by the video camera 16. The preferred embodiment captures this field of view and analyzes "windows" where independent movement is expected, such as one window about the left arm, and one window about the right arm. Through image processing, the guest's image is separately processed for each window so that movement of each arm yields control information used to process instrument, volume and tempo information that will influence the rendition of the prerecorded music score. The field of view may optionally be divided into as many windows as necessary that will yield distinct physical movement, such as movement of a finger, for example. These "windows" may be dynamic, or made to change in position relative to the field of view, to track objects and movement. In the Guest Controlled Orchestra, however, only two static windows are used to separately process movement of the left and right arms.

Each window may be processed to yield one or more control signals. Alternatively, a single control signal may be the product of several independent windows. The Guest Controlled Orchestra yields four control signals which influence the playing of the musical score. These signals include tempo, left only volume, right only volume and volume for instruments to be applied to both speakers.

With reference to FIG. 2, the contents of the console 15 are shown as removed from their normal position of concealment from the guest. The video camera 16 repeatedly scans the field of view and thereby captures the guest's image through the infrared window 18. The computer 20 is coupled to the video camera 16, so as to

receive its thirty frame-per-second input and process it to generate MIDI format music commands, as well as commands to control the playing of a video disk. The console 15 also conceals the professional style video disk player 30, which receives these commands, i.e., 1-5 frame advances, and directs the display of prerecorded animation stored on the video disk that also is sequenced to the guest's actions. The computer 20 processes audio data, which is stored internally in its RAM, to format MIDI commands and set volume and tempo information for the ultimate rendition by the sound system. The synthesizer 22 receives the MIDI output of the computer and generates an analog electronic signal which may be used to drive the speakers 28. Selection of the acoustic elements, namely, the MIDI synthesizer 22, mixer 24, audio amplifier 26 and speakers 28, is well within the skill of one familiar with electronic music equipment and hence, will not be discussed in detail.

The console 15 conceals the video disk player 30, which is a professional type video disk player that may be used both to record and play, and which is controlled by a control signal output by the computer. Video frames are output by the video disk player in response to the tempo information generated by the computer 20 and coupled to the disk player 30 via an RS-232 connector. The disk player's output feeds the video monitor 32 to display visual data stored on the disk player to the guest conductor 12 and observers. In the case of the preferred embodiment, the video monitor displays animation, including animated sea creatures, that appear to dance or otherwise perform in synchronization with the music and the guest conductor's actions. Derived control of video display is yet another example of the application of the invention to a system for generating control signals in general.

The processing hardware, including the image processor, the data processor and their supporting memory, is embodied in the personal computer 20, as modified with the addition of add-on boards, including an image digitizer and a MIDl/personal computer interface. More specifically, the video frame store chosen for the system 10 is an "IVG-128" board which is available from Datacube, Inc., of Peabody, Massachusetts. The MIDI/personal computer interface is analogous to a "MPU401" board, sold under the designation "MQX-32M", available from Music Quest, Inc., of Piano, Texas, and will hereafter be referred to as the musical interface board ("MIB") . The personal computer used in the preferred embodiment is an IBM 386-25 personal computer. Graphics and data processing software, written by the inventors in the "C" language and partly in machine language, directs the CPU's performance of the various tasks and the CPU's interface with these two add-on boards. The software, which is illustrated in the flow charts shown in FIGS. 3-8, is described in functional terms in the paragraphs that follow.

The video camera 16 chosen is a model "TI-24A", available from the NEC Corporation of Tokyo, Japan. It generates a black and white electronic output of standard video format, elaborated upon below. While the preferred embodiment uses a black and white camera, which will typically capture portions of the infrared spectrum, any type of camera may be used in accordance with the principles of the invention. The infrared window 18, discussed above, is implemented not for the desirability of capturing the infrared spectrum, but primarily to hide the video camera from observance by the guest and yet allow the camera to capture the field of view.

The video camera 16 generates a standard video signal by scanning the field of view and producing sixty

interlaced fields per second, or thirty complete frames per second. An electronic signal is produced by the video camera which represents the monochromatic luminance of the field of view as it is line-scanned from left to right. Each scan is slightly offset vertically from other scans, such that five hundred and twenty-five horizontal lines of scanning are used to cover the entire field of view. The electronic video signal, which repeats itself for each frame or thirty times per second, represents a generally continuous trace of the picture of five hundred and twenty-five horizontal lines when placed adjacent to one another. More precisely, each frame is comprised of two interlaced fields of alternating lines: The camera scans twice for each frame, scanning every other line each time, and produces five hundred and twenty-five lines. By scanning a picture in discrete lines comprising interlaced fields, the video camera produces a video electronic signal from a picture in approximately the reverse manner that a television reproduces a picture from a video signal.

This video signal is coupled to the personal computer 20 for image and data processing. During receipt of the video signal from the video camera 16, the computer's "IVG-128" board digitizes the video signal and thereby generates a plurality of pixels that represent the video signal. A "pixel" is nothing more than a sample of the video signal from which luminance can be discerned for a particular location within the field of view. Since the system 10 is a digital system, each video signal is digitized to provide a number for each pixel. This number comprises 8 bits that identify a sampled monochromatic (black-and-white) luminance for that pixel.

Thus, the "IVG-128" board of the computer 20 converts the repeated electronic video signal from the imaging system into a sequence of numbers, each number

corresponding to the luminance of the field of view at a particular point, or pixel. As will be discussed further below, each pixel corresponds to a location and is represented by coordinates that represent its location within the field of view.

Thus, as described above, the video camera produces an electronic signal that scans left to right across the field of view producing approximately 262-1/2 lines of visual information, or a single interlaced field, sixty times per second. In addition, the electronic signal from the video camera contains synchronization information that is utilized by equipment receiving the electronic signal to reconstitute the visual image within the field of view.

In digitizing the video signal, the "IVG-128" board produces two types of digital information in response to this electronic signal for processing by the computer.

First, it produces 8 bit digital values that represent the sampled black-to-white luminance of a pixel, or of a particular position within the captured field of view. Second, it produces status flags, or bits for sampling by the computer, which indicate when the pixel generator has digitized each of the first and second interlaced fields of a given frame of visual data. It is necessary for the computer to monitor these status flags to wait for and synchronize the commencement of its main program loop with the digitizer's completion of the first interlaced field.

The visual pixel information is stored in four 64 k banks of random access memory (which can be looked at as a 512 by 512 single byte frame buffer) , resident on the "IVG-128" board. Like any other type of random access memory ("RAM"), the pixel values may be overwritten with digital information to modify the stored image. The system 10 makes use of this ability in providing an output to the second monitor 34.

The computer's CPU compares a defined subset, i.e., every fourth pixel every sixth line, of these pixels with corresponding information of a previous image that has been stored in the computer's random access memory. In other words, the computer compares the pixel's number representative of luminance with a number representing luminance at the same coordinates from a previous image.

These sampling steps, described below, are described as including fixed increments. However, these constants are defined at the beginning .of the program, and may be chosen in the discretion of the computer operator to be any practical value. These constants are described below as specific values, because it is these values which have been used in operation of the preferred system.

To sample and process the pixels, the computer first waits for the "IVG-128" to raise a status flag that indicates completion of the first of the two interlaced fields that make up each video frame. It is only this first field that is sampled by the computer's CPU for visual data. The CPU starts with column 25 (of 384 columns that contain visual data) and row 50 (of 485 rows that contain visual data) and reads every fourth pixel until forty pixels have been read. It then moves six lines below, i.e., column 25, row 56, and repeats this same procedure. When fifty rows of forty horizontal samples are read, i.e., a left window of the image corresponding to the location of the right arm, the CPU reads another 40 x 50 sample block, beginning with column 208, row 50, and proceeding every 4th pixel, every 6th row, to develop a right window of the image corresponding to the left arm.

The "IVG-128" board stores the digitized image in an address format with the least significant address bits containing the horizontal coordinates, i.e., 0-IFF Hex, or

0-511, and the most significant address bits, i.e., 200- 3FE00 Hex, or multiples of 512, containing the vertical coordinates, or row information. This information is organized into four banks of 64k random access memory and requires selection of a particular bank in accordance with the "IVG-128"'s specifications and a standard sixteen bit address and CPU read and write operations. As indicated, although there are 512 possible columns or horizontal positions for each row, only 384 pixels contain visual information and only 80 of these are looked at by the CPU. Similarly, while the video signal has 525 interlaced lines per frame, only 485 contain visual information. It is therefore seen that by sampling only odd rows, the CPU scans only the first interlaced video field, and must complete its processing tasks before the completion of digitization of the next first interlaced field.

As the number representing each pixel is read by the CPU from the "IVG-128"'s memory, it is compared with a number representing the same pixel of a previous frame, saved to the computer's random access memory. After performing the comparison steps, described below, the computer writes the new number over the ..old number representing the previous frame, such that the new frame samples are stored in the computer's RAM and serve as the "previous frame" pixel data the next time the computer performs a comparison, one-thirtieth of a second later, for the next frame. It does this for each pixel that was used in the comparison regardless of magnitude of the luminance change, and thus requires 2 x 40 x 50 bytes of memory, or 4 k RAM, for the task.

In this manner, the computer ascertains which of the sampled subset of pixels represent movement relative to the earlier image. "Movement" is represented by a change in intensity, or luminance, as referred to above.

Luminance will change somewhat for most pixels of the captured image. It is therefore necessary for the computer to identify only those pixels for which the change in luminance is sufficiently great that the new luminance represents actual movement, as opposed to a slight change in lighting, or the like. To this end, as indicated in the software block diagram of FIG. 4, the CPU subtracts the luminance of a particular pixel from the old number corresponding to that same pixel, i.e., the same location within the field of view, from the immediately previous image. If the absolute value of the difference exceeds a predetermined number, the computer adds special x-y indices corresponding to that pixel to a x index sum and a y index sum of all pixels that likewise are sufficiently different from the previous image for that window. In the Guest Controlled Orchestra, the +/- difference between numbers is compared with both 10 hex and F0 hex, rather than the above-mentioned use of the absolute value of the difference, as computation of the absolute value is equivalent but requires additional software steps.

The computer writes each new pixel into the pixel sample buffer corresponding to each window, until all such pixels are exhausted. Therefore, in the Guest Controlled Orchestra, the computer 20 performs these steps for each sampled pixel of each of the left window and the right window. When it finishes each window, it will have a x index sum and a y index sum corresponding to that window.

In addition, when the computer 20 determines that a pixel has changed its luminance sufficiently from the past image, it increments a pixel count corresponding to the analyzed window. In other words, the computer begins the pixel count at zero each time it begins analyzing a window for movement. It then keeps track of the number of pixels of each window that represent movement. Each of the x

index sum and y index sum corresponding to that window are subsequently divided by this "pixel count". The result is a single x and y index value that represents a "centroid", or single position within the window that is the average of all points which changed significantly in luminance, or which represent movement. As mentioned, centroids are computed for each window frame. As will be discussed further below, the system 10 uses the pixel counts for each window to also determine the audio volume instruments, or orchestral voices, of the music played.

The x and y index corresponding to each sampled pixel should not be confused with its video row (of 512) and column (of 512) coordinates which necessitate an 18- bit address. Rather, the CPU performs a simpler task of assigning a row number, commencing with 49 and decreasing to 0, and a column number, commencing with 39 and decreasing to 0, which it uses for its processing tasks.

As shown in FIG. 4 and the appended software listings, the CPU looks at the pixels in step wise fashion, as described above, in three stages. For each window, the CPU first searches for a valid pixel, comparing each new pixel sample with the corresponding old pixel sample until it detects a difference greater than 10 hex or less than F0 hex, writing each new pixel into the pixel sample buffer as it does so. When the CPU has found a first valid pixel, it samples 10 rows each stepped 6 lines apart, adding x and y index values of qualifying pixels to a corresponding cumulative sum and increasing the pixel count, as described above. Finally, after all ten rows have been sampled, the CPU ceases its comparison and testing functions and merely writes each remaining new pixel (in the 40 x 50 sample window) to the pixel sample buffer, overwriting the corresponding old pixel values as it does so. If at any time the CPU reaches the 2000th pixel for the window, i.e., row index = 0 and column index

= 0, it determines that it has reached the last sample, ceases all processing functions, and proceeds to the second window. As the CPU finishes each window, it computes a "centroid", or movement center point for that window. Again, although the sampling for the second window commences at column 208, line 50 for the second window, the CPU defines an index value of y = 49, x = 39, and decrements these coordinates as the second window is sampled, to compute a second centroid corresponding to the second window.

Thus, as the guest 14 waves each of his left and right arms, the computer 20 tracks these movements and generates a single point, thirty times per second for each of the left and right image windows, to represent movement relative to the immediately preceding frame. Ideal operation of the system makes some assumptions. The first is that the guest 14 remains at the conductor station 12 so that his movement is captured by the video camera. Second, it is theoretically possible for the guest to swing his arms across his body in such a manner that the sum of both centroids produces a constant sum, in which case the system will detect a minimum tempo. It is stressed, however, that the system as described is robust against the guest's arms crossing his body so that any motion, including body swaying, will enable detection of a tempo.

In the Guest Controlled Orchestra, the field of view, as digitized, is sampled in static left and right half windows, which capture the guest's movements as long as he remains in the conductor station 12. Thus, the same subset of pixels for each window are always used, and their values subsequently stored in computer's RAM, for use as the prior image in the subsequent comparison step for the next frame. Alternatively, the windows analyzed could be small areas of the field of view and could be

made to be dynamic, or centered around prior movement, and thus made to track independent movements within the field of view. In this latter case, the computer would need to preview each dynamic window within the field of view and anticipate and store-into memory those pixel values which it will need for the subsequent comparison, which may not be coextensive with those pixels used for other comparisons. As shown by the attached software appendix G, fixed pixel sample step sizes are defined at the outset of the subroutine and associated with variables. However, it would be well within the skill of one familiar with writing computer software to implement a subroutine which would adjust the value of the pixel step size variables during the program operation. Implementation of this or other alternate embodiments which make use of dynamic windows would be well within the ordinary level of skill in computer science or electronics.

In addition to overwriting the 2000 samples for each window one-by-one into the computer's RAM, the computer also provides the above-mentioned second monitor output. In the preferred embodiment, the computer is also coupled to a second display monitor 34 that permits observers to observe the operation of the Guest Controlled Orchestra. After the computer has compared the difference between pixel values with 10 hex and F0 hex, the computer also writes a predefined luminance value into the "IVG-128"'s memory associated with the sampled pixel. For example, if the result of the comparison is that the difference is either less than 10 hex or greater F0 hex, a grey value (80 hex) is written into the sampled pixel location. If the pixel change is greater than 10 hex or less than F0 hex, indicating movement, a white value (FF hex) is written into the sampled pixel location.

The "IVG-128" board is configured to independently provide an analog video output that may be used to

directly drive a video monitor. Thus, the second video monitor 34 is coupled to this output and requires no intervention by the computer's CPU to display information stored in the "IVG-128"'s memory. As a result of its comparison step, however, the computer changes the sampled pixel value to either white or grey. Thus, on the display monitor, the concentration of white pixels will readily be observed superposed on the black-to-white image as indicating movement, and concentrations of grey pixels observed as indicating regions where movement has not occurred.

Once the computer's CPU has finished a 2000 sample window and has overwritten the last sample into the pixel sample buffer, the CPU will compute the centroid coordinates for that window. The CPU computes a centroid for each window by dividing each of the x index sum and the y index sum for each window by the window's qualifying pixel count. It then stores this centroid coordinate in memory for subsequent use in computing a scalar, described further below. If no qualifying pixels have been found for the window, the CPU will utilize the centroid for the previous frame.

Once both windows have been processed, the CPU satisfies itself that at least four qualifying pixels have been found over both windows. If less than four such pixels have been found, the CPU abandons its centroid and correlation steps and loads a minimum tempo to the MIB, as illustrated in the flow chart of FIG. 4, and then proceeds to process any video frame advance.

After computing the centroid for each window, the

CPU will determine a current volume for each channel used of eight possible channels, each corresponding to an orchestral voice.

Each MIDI command either turns an electronic note on or off. The "note on" commands, which are segregated into channels that each correspond to an orchestral voice, also contain digital information as to the volume of the note to be produced. The Guest Controlled Orchestra uses pixel counts to define volume of instruments appearing from each speaker and updates this information as it has finished looking at both windows.

To update the volume corresponding to "note on" commands of each of the eight channels, the CPU looks at each window's pixel count to determine the volume that is to be associated with certain instruments. For example, in the Guest Controlled Orchestra, the left channel is used for bass, and accordingly, the CPU will store a volume level derived from the number of qualifying pixels for the left window into a channel variable corresponding to bass.

MIDI data is formatted to include data representing pitch (128 possible) , instrument type (16 possible) and a command that turns the note on or off. Thus, whenever the CPU is called upon to send a "note on" command,to the MIB, it simply replaces the bits representing volume with a set of bits derived from the pixel count, depending upon the type of instrument that the particular MIDI data represents. In current use of the Guest Controlled Orchestra, four instruments are used, including flute, bass, marimba, and drum. However, any type or number of instruments can be used. The flute is panned to the right channel and accordingly, the volume bits associated with MIDI flute commands will be derived entirely from the right window changed pixel count and put into a channel variable associated with the flute for use during the one- thirtieth second program cycle. Similarly, the volume of MIDI bass instrument commands will be substituted with a value derived from the changed pixel count in the left

window. The changed pixel counts for each of the left and right windows are averaged to yield volume information for marimba and drum note commands. Alternatively, rather than the preferred step of replacing the MIDI command bits associated with note volume, the computer's CPU could be instructed to amplify the command's default volume by an amount dependent upon the changed pixel count.

The system 10 uses two categories of interrupts, which request the CPU to cease performance of all program tasks and perform interim processing. The most significant of these CPU interruptions is utilized to direct filling the MIB's "schedule" of notes to be played. Operation and use of these interrupts may be understood with reference to FIG. 5.

FIG. 5, rather than describing a subroutine or expanded processing step otherwise referred to in FIG. 3, illustrates the flow of the MIB that is distinct and collateral to the main program loop of FIG. 3. When the system is initialized, the MIB has no commands in its buffers, or slots, and the CPU will format and load MIDI commands into eight buffer pairs, defined in, AM by the CPU. Each of these buffer pairs includes a "note on" buffer and a "note off" buffer and corresponds the eight scheduling slots of the MIB. Each of these eight "channels" will correspond to a particular instrument. When the CPU retrieves audio data from RAM and formats it to a MIDI command, the CPU detects the instrument type and places the command into either the "note on" or "note off" buffer corresponding to that instrument. Once the CPU has loaded at least twelve "note on" commands corresponding to each channel into its "note on" buffer (and corresponding "note off" commands into the corresponding "note off" buffer) , the CPU sends a command to the MIB directing the MIB to play music.

Operation of the MIB is the initiated by the CPU command, which triggers a generic interrupt from the MIB to tell the CPU that its slots are empty. The CPU feeds a command to the MIB for each channel used, along with a countdown time associated with the command. When the countdown time has expired and the command sent from the MIB to the synthesizer, the MIB generates a channel specific interrupt, which directs the CPU directly to the corresponding "note on" and "note off" channel buffer pair. The CPU looks at each of these buffers to ascertain which buffer holds the command to be issued the soonest. The CPU retrieves this command and formats and sends it to the empty MIB slot. Operation of the Guest Controlled Orchestra currently uses only four of these eight channels, assigned to flute, bass, marimba and drums.

The "slots" described are a set of buffers resident on the MIB that each output a "note on" or "note off" command after a "countdown" time associated with the notes has elapsed. This countdown time is defined in "ticks", with 192 ticks per musical beat. The tempo given to the MIB determines the rate at which the MIB counts "ticks," and hence determines the rate at which notes are played or turned off.

The "MQX-32M" used as the MIB is a standard musical interface board which receives MIDI commands formatted by the computer's CPU, and which puts those commands into one of eight scheduling buffers along with each command's associated countdown time. As stated, when the countdown time associated with the command has elapsed, the MIB removes the command from its associated buffer and feeds the command to the music synthesizer, triggering a channel specific interrupt to the CPU. The MIB thus interrupts the CPU at various intervals seeking a subsequent note command. Once note commands for a particular channel have expired, no new command is sent to the corresponding MIB

slot, which remains idle until reactivated by system initialization and a new command, prompted by the generic interrupt, described above.

As the main program loop cycles through every one- thirtieth of a second, the CPU will access its pointer that points to the next digital information in RAM and look at that information stored in the score sequence. Each piece of digital information is stored as a nine byte command, including channel (or orchestral voice) , velocity (volume) , pitch, note start time and duration. The CPU analyzes channel type and ascertains by looking at the corresponding channel "note on" buffer whether that buffer has its allotment of twelve "note on" commands. If not, the CPU retrieves the nine byte word and increases a pointer to point at the next piece of digital information in the score sequence.

The CPU must define "note on" and "note off" command times from each nine byte word and store respective commands in the channel's associated buffer pair. The constant chosen to define "note on" buffer length defines twelve slots that include an absolute time (four bytes) , a default volume, which may be altered as the "note on" command is output from the buffer, and pitch. The "note off" buffer is filled with a corresponding "note off" command, which must be properly inserted into a sequence of "note off" commands since note durations may vary. In other words, since successive note start times will be in sequence, but notes may vary in their duration, associated note end times may be out of sequence. The "note off" buffer for each channel is defined as twenty slots, which each store absolute command time and pitch. The absolute time for the "note off" commands is obtainedby summing the absolute time of the corresponding "note on" command with the duration time. Each of these buffers is a

circular buffer and has a pointer associated with it by the CPU.

Each command is processed in this manner to fill the CPU's buffers with sufficient notes to last for the next one-thirtieth second cycle.

When the CPU receives a channel specific interrupt from the MIB that tells the CPU to remove a note command from one of its corresponding channel buffer pair, the CPU first compares the next command for each of the "note on" and "note off" buffers corresponding to the channel, or orchestral voice. The command to be executed sooner is removed. If a "note on" command, the volume represented by one of the bytes of the command is overwritten with the current computed volume that has been associated with that channel. Also, the countdown time is computed by the CPU by subtracting the absolute time of the channel's previous command, which is stored by the CPU, from the current absolute time of the "note on" or "note off" command. This countdown time is sent as part of the command to the MIB and the absolute time of the command stored by the CPU for further use.

When the CPU is outputting a command to the MIB and computing the command's countdown time, the CPU compares the countdown time to 240 ticks. This is necessary, since the MIB can only hold a 240 tick countdown time. If the command's time is greater than 240 ticks, the CPU does not increment the note on/off pointer to the chosen command and instead sends an overflow command to the MIB. The CPU retains an absolute time of the overflow command for subsequent comparison with the next command. The overflow command is associated with a 240 tick countdown time which triggers no action by the synthesizer, but which causes another MIB interrupt corresponding to the same channel when the 240 ticks have elapsed.

A mentioned contemplated alternative embodiment also analyzes the guest's actions to select instrument, for example, by analyzing the vertical range of the guests's movement. Thus, the vertical range or location of a guest's action could be made to influence the changing of notes having bass as the default instrument to horn notes, etc. Such a procedure could be implemented, for example, when the CPU removes a command from a channel buffer pair and formats the command for the MIB. This contemplated option has not been implemented in the current system, but it is within the principles of the invention.

With the MIDI command appropriately formatted and stored for output to the MIB from the CPU's buffer, the MIB's interrupt is reset and the CPU returns to its normal program operation.

The MIB operates on supplied tempo information and will employ the most recently provided tempo information to schedule note output. Tempo information is defined as a number of beats per minute, which may vary with the particular music to be played. In the preferred mode of the Guest Controlled Orchestra, the audio data represents a song with 240 beats per minute.

MIDI format specified minimal temporal resolution is 192 ticks per beat. Thus, in the preferred embodiment the MIB will schedule notes with each tick and will adjust its internal clock according to the tempo of the guest's actions in relation to 240 beats per minute. For example, if the guest moves his arms at a fast rate, the tempo will be perceived as being greater than 240 beats per minute, etc. The MIB will read a tempo word fed to it at the end of the main program cycle, every one-thirtieth second, and will perceive changes in the guest's rate of movement, adjusting its own internal clock accordingly. Thus, the MIB counts ticks faster or slower in waiting to output

channel commands to the synthesizer, depending upon the guest's rate of motion.

To detect and ascertain this tempo information, the CPU must quantify movement occurring since the previous frame and must correlate this movement to ascertain a pattern of movement. To do this, the CPU looks at the mean location of perceived movement, which will tend to form a path over time. The CPU correlates the position of the current location of movement with the path over time, or history, to determine when it was last at a similar location,. Since each location is associated with a frame, or one-thirtieth of a second, the CPU can compute the rate of the guest's movement and a current tempo.

When the CPU has computed the centroids for each of the left and right windows, it computes a scalar, or single value, which is a measure of the magnitude of all movement occurring within both windows. In order to capture all of guest's movement, which may be both vertical and horizontal, the CPU sums all coordinates for all of the centroids within the field of view. Thus, the CPU will add the y index of both the left and right window centroids, as well as the x index. However, since the guest will typically move the left and right arms together vertically, but in opposite directions horizontally, the x index for the centroid for the right window is inverted by taking its l's complement before summation. In this manner, opposite movements of the left and right arms in the horizontal direction will not cancel each other out, and the resultant scalar will be a measure of both vertical and horizontal movement of both arms.

The CPU stores the scalar for each frame in a two hundred position circular buffer, defined in the computer's RAM. Thus, the computer's CPU will have a long-term memory of the most recent two hundred frames and

all older data will be overwritten by the most recent frame's scalar.

When the program is initiated, the CPU's pointer which accesses the MIDI audio data stored in the computer's RAM is initialized to point at the first note of the song. Similarly, the various memory allocations corresponding to buffers to be used with data processing are zeroed. When the computer first detects a guest's presence and subsequent movement, it will load the MIB with a predefined tempo word read from its RAM and corresponding to the ideal rate of play of the music in the computer's RAM. During this time, the CPU analyzes the guest's movements to develop a history and to store this information as scalars in the circular buffer. Once two bars have been played, the CPU is free to change the tempo, as stored in the MIB, and thereby change the rate of play of the song. This "two bar" feature implemented as an inhibit which, after the computer has performed the processing and correlation steps described above and below, will prevent the CPU's update of the sixteen bit tempo word to the MIB.

After the computer's CPU has written the newly processed scalar into the two hundred position circular buffer, it then correlates the most recent thirty scalars over the entire two hundred scalars in memory, as shown in FIG. 5. It does this by utilizing an index to point at the beginning sample of a thirty sample dynamic window over the two hundred scalars, and by performing the following process until one-hundred and seventy iterations have been performed and all two hundred samples correlated. The CPU:

(1) subtracts 30 scalars beginning with.the scalar defined by the index from the most recent 30 scalars, respectively;

(2) squares each of the 30 differences obtained and sums all 30 squares together;

(3) stores the sum in a 170 position buffer;

(4) increases the index to point to the next scalar for the next iteration; and

(5) continues with 170 iterations until the index again points to the most recent sample; to save time each iteration subsequent to the first is computed by taking the square of the difference of the oldest scalar within the 30 sample window with the 30th most recent sample, adding that to the most recent sum, and subtracting the square of the difference of the most recent scalar with the newest scalar in the thirty sample window.

This correlation process will thus produce one- hundred and seventy values which are each sums of the squares of the differences between two varying pair groups of thirty samples. Each of these two hundred numbers will vary from a value of zero, i.e., the two thirty sample windows are identical and thus when subtracted produce a value of zero, to a value which may be extremely high, and which is thus provided for by allocating 4 bytes of memory to each of the two hundred memory positions. Thus, the second buffer begins with the value of zero (both thirty sample windows coterminous) and will ascend to a very high number. When a scalar pattern similar to that represented by the most recent frame of data is detected, the sum of the squares of the differences will again be at a minimum. Thus, the positions are analyzed to obtain local minima corresponding to the guest's movements which were similar to that captured by the most recent series of video frames, and tempo extracted.

Since the first of the two hundred summed squares is always zero, the computer's CPU analyzes the increase in the squares by looking for a maximum and a subsequent minimum in two sequential program loops. When it detects a potential maximum, by determining that the next value is less than the preceding value, the CPU applies a bandgap to the maximum to make sure that it is followed by a downward trend differing by at least the number 512. It does this to guard against spurious maxima caused by noise. When it detects a potential minimum, it filters spurious minima by testing subsequent sums to ensure that the potential minimum is followed by an upward trend differing by at least the number 512. The CPU stops searching once it has obtained a second minimum.

Two minima are sought in the preferred embodiment, because it is expected that a guest's arms will be moved both vertically and horizontally. As a result of this movement, the sums of the squares of the differences may experience either one or two minima associated with each cycle of typical motion by the guest. To correctly identify the guest's tempo, the CPU compares the first two minima to ascertain which is smallest, which it presumes to identify the correct tempo. This minimum is compared with a cut-off (selected in the preferred system as the number 2048) to ensure that the detected minimum is small enough to represent a reasonable correlation. If it is not, the correlation result is ignored and the tempo is not updated.

With each correlation, the computer's CPU looks for the first two beats to generate the tempo information. Each squared sum corresponds to a location in the history buffer and is thus associated with time, a particular video frame and a particular point along the scalar's periodic magnitude. The CPU, by performing the correlation process, effectively matches the scalar's

change in magnitude at any particular point with a corresponding change of a different cycle, and thereby determines periodicity. The computer's CPU uses this information to compare the beat information derived from periodicity of the guest's movement with the ideal tempo for the particular musical piece. If the guest's tempo deviates from the most recent tempo information that the CPU has, the CPU writes a new 16-bit tempo to the MIB, so that MIDI notes may be appropriately scheduled for output.

The functions described above and embodied in the appended software require mention of several additional points. First, if a guest arrests all movement during the playing of audio data, the software must be capable of freezing the computer's continued correlation of scalars, to avoid skewing the two hundred position circular buffer, or history, described above. In the Guest Controlled Orchestra, the computer's CPU does not perform the correlation steps if the number of changed pixels for a given image frame is less than or equal to four. Thus, the circular buffer will not be updated until the guest once again continues movement. Also, the computer's CPU will arrest the playing of music by the MIB by loading the MIB's tempo register with a minimum tempo, indicating that music is to only be slowly output by the MIB to the MIDI synthesizer.

Also, it is noted that the algorithm utilized in the preferred embodiment for deriving the scalar from the centroids for each frame is susceptible to a minimum tempo if the guest engages in particular movement, i.e., movement of arms that produces a constant scalar as a result of the scalar computation algorithm. It is expected that those skilled in the art can implement an alternative algorithm that avoids this result without departing from the principles of the invention.

In addition, the computer's RAM must store the total number of beats of the chosen musical score. The system 10 preferably uses the song "Under The Sea" from the Disney movie "The Little Mermaid, " in a theme park setting. The computer's CPU determines from this information the number of MIDI commands in the audio sequence, and thereby determines when its pointer references the last note in the song. It formats and accordingly loads this final command to the MIB's register, and terminates the main program loop, as indicated in FIG. 3.

The last step of the main program loop, as shown in FIG. 3, is to synchronize the video disk player 30 with the music generated with by the audio system. The CPU determines, from comparison of the tempo to the score's ideal tempo, the rate at which the video disk player 30 is told to advance frames. As video frame advance is directed by the CPU once every thirtieth of a second, the computer will normally direct the video disk player to advance one frame. To the extent that the comparison of the guest's tempo and the ideal tempo are not the same, i.e., their ratio is not an integer, the CPU .accumulates any excess to be applied to the next program cycle. Thus, if the guest's tempo is slower than the ideal tempo, the computer will not instruct the video disk player 30 to advance, but will accumulate the guest's beats per minute for addition to the subsequent tempo calculation during the next video frame advance step. In the Guest Controlled Orchestra, the video disk player 30 does not communicate to the computer 20, and hence, the program must instruct the CPU to keep independent tally of the progress of the video frame display sequence.

The disk player used in the system is a "TQ-3032 F" optical disk player, available from Panasonic Industrial Company of Secaucus, New Jersey, and, as mentioned, is

coupled to a port of the computer via the RS-232 connector. Each time the video frame advance is called, the computer will either not instruct the video player or it will instruct the video disk player to advance from one-to-five frames. The disk player automatically provides a video rate output of the frame it is instructed to display, and continues displaying that frame, thirty times per second, until instructed to advance by the computer.

The second of the CPU's interrupts is utilized to load data to the RS-232 port for communication to the video disk player. Every one-thirtieth of a second, the CPU will either not command the optical disk player or will format the 1-5 frame advance commands, as discussed. The second interrupt is repeatedly called to load successive individual bytes of this command into an output buffer of the RS-232 port until there are no remaining bytes of the command. After a single byte is loaded, the interrupt is subsequently reset, and the loaded information transmitted bit-by-bit until the output buffer is once again empty. The interrupt will be triggered for loading a remaining byte into the output buffer as long as any bytes in the video command remain.

A further refinement of the system 10 within the scope of the invention would be to implement communication from the video disk player to the computer to indicate frame number. In this manner, the computer 20 could directly read the frame number, instead of keeping independent tally of the frame, as mentioned above. There are commercially available video disk players which have an output port to provide this connection.

Those skilled in the art will observe that numerous changes may be made to the Guest Controlled Orchestra without departing the principles of the current invention.

For example, implementations may be easily devised which correlate values separately for each window, or which simply determine periodicity by analyzing when the scalar's path of movement, or magnitude with respect to time, transgresses a predefined value.

Having thus described several exemplary embodiments of the invention, it will be apparent that various alterations, modifications and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements, though not expressly described above, are nonetheless intended and implied to be within the spirit and scope of the invention. Accordingly, the foregoing discussion is intended to illustrative only; the invention is limited and defined only by the following claims and equivalents thereto.