There are at least three well known representations of a signal: 1. The spectrum or frequency domain representation is a two-dimensional graph in which amplitude is plotted versus frequency. All points of the graph are normally plotted using the same color. Time information is not evident.

2. The spectrogram is a three-dimensional graph in which frequency is plotted versus time and the points of the graph are rendered using colors or shades of gray to convey amplitude information.

3. The waveform display or time domain representation is a two-dimensional graph in which amplitude is plotted versus time. All points of the graph are normally plotted using the same color. Frequency information is not evident.

FIG. 1 shows a portion 10 of a typical waveform display derived from an audio signal, i. e. an electrical signal containing audio frequency components. The waveform display shown in FIG. 1 is of the kind that might be presented by a display device having a display area composed of pixels 14 arranged in columns 18. For convenience, the pixels 14 are shown in FIG. 1 as having an aspect ratio (height: width) substantially less than one, but in practice the aspect ratio would normally be closer to one.

By way of example, the portion 10 of the waveform display represents an audio signal that has been sampled at intervals T and has been quantized to return the twenty values indicated at the bottom of FIG. 1. Each column 18 of pixels is used to display a representation of the evolution of the signal over an interval 5T, so that the twenty sample values are represented by four columns. Typically, the background color of the display is white and the amplitude information is plotted by rendering selected pixels 14 with a

contrasting foreground color or black. The pixels in the column that are rendered with the foreground color (or black) show a line segment connecting the maximum and minimum sample values within the interval represented by the column. Thus, in the leftmost column the pixels from-2 to +5 are rendered with the foreground color (or black), represented by solid diagonal shading.

Summarv of the Invention In accordance with a first aspect of the present invention there is provided an improved method of representing time evolution of a signal during a signal epoch using a rectangular array of pixels in which each column of pixels is assigned to a time interval within the signal epoch, and wherein foreground pixels in a column of pixels form a line segment connecting minimum and maximum amplitude values within the time interval to which the column is assigned, wherein the improvement resides in rendering the foreground pixels in a column with a color that depends on frequency-dependent information relating to a time segment that includes the time interval.

In accordance with a second aspect of the present invention there is provided an improved method of displaying information regarding a signal of which amplitude and frequency vary with time, in which the signal is used to generate a time domain representation of the amplitude of the signal as a foreground waveform against a background, wherein the improvement resides in varying the color with which the waveform is displayed as a function of frequency-dependent information obtained from the signal.

In accordance with a third aspect of the present invention there is provided a computer programmed to process a signal that evolves with time during a signal epoch by partitioning the signal epoch into a plurality of consecutive time segments, processing the signal to extract frequency- dependent information that characterizes each time segment, assigning a value to each time segment based on the frequency-dependent information that characterizes the time

segment, receiving a user input to select a time segment of the signal, determining whether the value assigned to another time segment matches the value assigned to the user selected time segment, and, if so, providing an output which emphasizes at least one such time segment.

In accordance with a third aspect of the present invention there is provided a method of navigating a signal record, comprising partitioning the signal record into a plurality of consecutive segments, processing the signal to extract frequency-dependent information that characterizes each time segment, assigning a value to each time segment based on the frequency-dependent information that characterizes the time segment, selecting a time segment of the signal, determining whether the value assigned to another time segment matches the value assigned to the user selected time segment, and, if so, emphasizing at least one such time segment.

In accordance with a fourth aspect of the present invention there is provided a computer programmed to generate a representation of time evolution of a signal during a signal epoch using a rectangular array of pixels in which each column of pixels is assigned to a time interval within the signal epoch, and wherein foreground pixels in a column of pixels form a line segment connecting minimum and maximum amplitude values within the time interval to which the column is assigned, wherein the computer is programmed to render the foreground pixels in a column with a color that depends on frequency-dependent information relating to a time segment that includes the time interval.

Brief Description of the Drawings For a better understanding of the invention, and to show how the same may be carried into effect, reference will now be made, by way of example, to the accompanying drawings, in which FIG. 1 is a graph illustrating a portion of a waveform display of an audio signal,

FIG. 2 is a block diagram of an apparatus for processing an acoustic signal to provide an enhanced waveform display, FIG. 3 is a diagram illustrating signal processing in the apparatus shown in FIG. 1, FIG. 4 is a graph illustrating a portion of an enhanced waveform display, and FIG. 5 is a flow chart illustrating a method in accordance with the invention.

Detailed Description Referring to FIG. 2, a microphone 22 receives an acoustic signal and converts it into an audio frequency analog electrical signal. An analog-to-digital converter (ADC) 26 samples the analog audio signal at 44.1 ks/sec and quantizes the samples to generate a digital audio signal composed of a succession of N1 sample values or words. The digital audio signal may be acquired from another source, such as playback of a compact disc or by digitizing an analog signal generated on playback of a tape recording. In any event, an audio data record in which each word represents the amplitude of the audio signal at a selected sampling time during a signal epoch is generated and is loaded into an acquisition memory 30. If, for example, the duration of the signal epoch is 200 seconds, the audio data record will contain 8.82x106 words.

A display controller 40 generates a video signal for controlling operation of a display device 32 to display pixels organized in a rectangular array composed of H rows and N2 columns. The display device 32 is able to display a palette of colors, and each pixel can be painted with any of the colors in the palette.

To render the waveform display of the signal as a bitmapped image on the display device 32, each column of the array of pixels represents a distinct interval of the signal epoch containing C = N1/N2 sample values.

A processor 34 reads the audio data record and generates a pair of minimum and maximum amplitude values for each set of C consecutive sample values, corresponding to one of the

N2 intervals of the signal epoch. The processor 34 supplies the pairs of minimum and maximum amplitude values to the display controller 40. The pairs of maximum and minimum values allow the display controller to generate a video signal specifying the foreground pixels of each column, forming a line segment connecting the minimum and maximum values within the interval represented by the column.

The processor 34 partitions the audio data record read from the acquisition memory 30 (FIG. 3, step 100) into N3 consecutive, contiguous, non-overlapping segments, where each segment consists of kC sample values, where k is an integer between 1 and N2. Thus, N3 is equal to N2/k, and each segment is represented by a distinct group of k consecutive columns of pixels. In this specification, the term"super column"is used to refer to the k columns of pixels associated with a given segment of the data record. In this manner, the audio signal epoch is divided into N3 consecutive, contiguous, non-overlapping time segments of equal duration. In a typical application, there might be one thousand segments and in the case of the example, each segment represents a signal interval of 0.2 s and contains 8,820 words.

The processor 34 analyzes each segment (step 104) to extract frequency-dependent information. For example, the processor might execute a Fast Fourier Transform or a Linear Prediction algorithm, in which case each segment of the audio data record is decomposed into multiple frequency components.

The audio frequency spectrum is divided into several bands and a unique index value is associated with each band. For example, the audio spectrum may be divided into six bands and one might associate the index value 1 with the lowest frequency band, 2 with the second lowest frequency band, 3 with the third lowest frequency band, and so on, associating the index value 6 with the highest frequency band. The processor 34 determines which frequency band is dominant (step 108A), for example by determining which band contains the frequency components having the greatest amplitude, and assigns an index value P to that segment based on the

location of the dominant frequency band in the audio spectrum (step 112A). The processor 34 also determines an amplitude variance value V for each segment (step 108B), for example based on the variance of the sample values used to form the segment. The data elements P and V for each segment are combined to create a signature (P, V) which contains both frequency information and amplitude variance information and characterizes the segment (step 116). The processor loads the signature in a signature memory 38 having N3 separate addressable memory locations (step 120).

The display controller 40 reads the signatures from the signature memory 38 and generates a video signal for controlling operation of the display device 32 such that the appearance of the foreground pixels in the ith super column of the display depends on the signature stored at the ith memory location in the signature memory. Specifically, the display controller generates the video signal such that the foreground pixels in the ith super column are painted with a color that depends on the element Pi of the signature in the corresponding memory location of the signature memory. If red is considered a more active or urgent color than blue and correspondingly higher frequency sounds are considered more active or urgent than lower frequency sounds, it may be appropriate to associate shades of red with higher frequencies and shades of blue with lower frequencies. Thus, one might associate the colors red, orange, yellow, green, blue and violet with the index values 6,5,4,3,2 and I respectively.

It will therefore be seen that the display device provides an display which is similar to the waveform display shown in FIG. 1 but is enhanced by rendering frequency information evident through variation in the foreground color. For example, in the case of the signal whose waveform is shown in FIG. 1, if k were equal to two, so that two columns of pixels form a super column, and the index value 6 were assigned to the segment corresponding to the columns at the left of FIG. 1 and the index value 2 were assigned to the segment corresponding to the columns at the right of FIG. 1,

the enhanced display shown in FIG. 4 would be obtained, where the two orientations of diagonal shading represent red and blue respectively.

Although the foregoing discussion is directed to all the foreground pixels in each super column being displayed with the same color, it would be possible to display the foreground pixels in a given super column with more than one color. By way of example only, a given index value could be assigned to two colors, in which case some of the foreground pixels in the super column may be painted with one color and other foreground pixels in the super column may be painted with another color. The two groups of pixels may be interspersed, thus creating a third color, or the two groups of pixels may be discrete and continuous or the group of pixels of one color may be located between two subgroups of pixels of the other color. The number of possible ways of utilizing the frequency-dependent information to control the color of the foreground pixels is very great, and no attempt has been made to be exhaustive.

A printer 36 is associated with the display device 32 and can provide a printout, e. g. on paper or a transparency medium, of the image shown on the display device 32. Thus, an image of only transient importance may be only displayed temporarily on the display device, but if an image is perceived to have enduring significance, it can be printed out using the printer to form a lasting record.

In the event that the display device 32 is able to display only W columns of pixels, where W is less than N2, the user may operate a horizontal scroll bar to bring into view any W consecutive columns of the image.

The signatures stored in the signature memory can also be used to navigate the audio data record. Referring to FIG.

5, the user can select (step 200) a particular segment (segment M) of the audio data record by using a cursor to identify a super column of the display while viewing the waveform. The processor 34 retrieves the signature for segment M of the audio data record from the signature memory and compares other signatures with the signature for segment

M, using both the frequency information and the amplitude variance information, in order to identify signatures that match the signature for segment M within a desired tolerance.

As shown in FIG. 5, this may be accomplished by setting a variable N equal to M (step 204), reading the signature for segment N+1 from the signature memory (step 208) and comparing it with the signature for segment M (step 212). If the two signatures match within the desired tolerance, the processor 34 instructs the display controller to alter the display by emphasizing or drawing attention to the super column corresponding to segment N+1 of the audio data record (step 216), for example by moving the cursor of the display to point to this super column. If the signatures do not match, the process determines whether segment N+1 is the last segment, i. e. N+1 = N3 (step 220) and, if so, the process terminates. Otherwise, the variable is incremented (step 224) and the process returns to step 208.

In the process described with reference to FIG. 5, the segments having signatures that match the signature for segment M are located by reading forward from segment M.

Alternatively, the matching segments may be located by examining all the signatures from the start of the signature memory or by reading backward from the selected segment, or in another convenient fashion, depending on user commands.

The signatures may be compared until a selected number of matches have been accumulated, including only a single match, or until all signatures have been compared. The user can move the cursor forward or backward from one matching segment to another, for example by pressing direction arrow keys or invoking other standard cursor movement commands.

The weight to be accorded to amplitude variance in the matching process can be selected by the user. For example, the user may wish to base the matching only on the color component of the signature, in which case no weight would be accorded to amplitude variance.

It would also be possible to select a segment of the audio data record by invoking a select command while listening to the acoustic signal concurrently with processing

the acoustic signal in the manner described with reference to FIGS. 2 and 3 to generate the signatures, or while listening to a replica of the acoustic signal created by playing back the audio data record stored in the acquisition memory, through a digital-to-analog converter and a loudspeaker. In the former case, it may be necessary to wait briefly while the digital audio signal is processed before it will be possible to navigate the audio data record.

The process described with reference to FIG. 5 is not restricted to matching a single segment with other segments, and the user may select a sequence of consecutive segments and have this sequence compared with every other sequence of segments to locate matches.

Although it is normally preferred that the waveform should be displayed in color, as described with reference to FIGS. 2 and 3, it is not necessary to do so when using the signatures to navigate the audio data record as described with reference to FIG. 5, and in this case the display could be monochrome.

It will be appreciated that the invention is not restricted to the particular embodiment that has been described, and that variations may be made therein without departing from the scope of the invention as defined in the appended claims and equivalents thereof. For example, although the foregoing description refers to a method in which the segments are non-overlapping and each corresponds to the same length of time, and the segmentation is accomplished automatically, the segmentation could be performed in response to user input, wholly or in part, and the segments may correspond to time intervals of non-uniform length and intervals that are overlapping. Particularly, if the signal remains uniform (within specified limits) in amplitude and frequency for considerable periods of time, a single segment may correspond to a longer time interval than if the signal changes relatively rapidly in amplitude or frequency. Further, although the foregoing description is concerned with extracting information regarding the dominant frequency, information regarding subordinate frequencies may

also be used to determine the color or colors used to paint a super column. Moreover, the functions and arrangement of the blocks shown in FIG. 2 were chosen in order to facilitate discussion of signal flow and the invention is not restricted to use of the specific functional blocks shown in FIG. 2, arranged and connected in the manner shown in FIG. 2. For example, although the display controller 40 is shown in FIG.

2 as a distinct block from the processor 34, its function may be performed by the processor 34.