Technical Field

[0001] The embodiments disclosed herein relate to audio signal processing including natural language processing, and, in particular to a system and method for analog feature extraction and subsequent neural network processing, such as natural language processing using an audio sensor such as an analog microphone and all analog processing elements.

Introduction

[0002] There has been significant progress in audio processing for sound detection and categorization, most notably for natural language processing for voice detection, keyword spotting and full sentence translation. Most existing systems for natural language processing employ digital solutions.

[0003] FIG. 1A shows a typical keyword spotting digital system 100. The system 100 may be employed for “always on” audio/voice detection. The digital system 100 includes an analog microphone 102 for receiving audio/voice input. The raw signal from the microphone 102 must be converted from analog to digital (for simplicity, audio activity detection, voice activity detection, analog to digital converter blocks and noise filtering components are not shown), for processing by a logarithm of filter bank energy (LFBE) or mel-frequency cepstrum coefficient (MFCC) feature extractor 106. The feature extractor 106 translates the time-domain speech signal 104 into a set of frequency domain spectral coefficients which enables dimensionality compression of the input signal 104 to extract voice or sound “features”. This type of system can be used to identify many classes of audio signals, keywords being one example.

[0004] The extracted feature matrix is fed to a neural network classifier 108 for classification of the voice or sound which generates the probabilities for the output classes 110. The neural network 108 can vary (e.g., deep learning neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), convolutional recurrent neural network (CRNN)) depending upon the system 100 requirements as can the complexity of the LFBE/MFCC feature extractor 106. Typically, neural network classifiers 108 for natural language processing can include 220 thousand to 1 .4 million 8- bit parameters. In a real-world scenario where keywords need to be identified from a continuous audio stream, a posterior handling module (not shown) averages the output probabilities of each output class over a period of time, improving the overall confidence of the prediction.

[0005] A limitation of existing digital systems 100 include high power consumption, which is problematic for battery powered or other severely power constrained applications. The total power consumption of the complete system 100 is the summation of the power consumption of all the components including the microphone 102, analog to digital converter, audio and voice activity detection, noise filtering, the LFBE/MFCC feature extractor 106 and the neural network classifier 108. In particular, significant power consumption is required for converting the analog voice signal to a digital signal for processing by the LFBE/MFCC feature extractor 106 and neural network classifier 108. For extremely low power digital solutions, the total power consumption of the system 100 can be as low as 250-500 pW.

[0006] Referring to FIG. 1 B, another limitation for a digital always-on voice system 120 is an audio buffer 126, typically in the range of 500 ms up to 10 s, is required for replay of the audio signal to further process keywords. This buffering function can typically consume excess power because the audio input signal is converted from analog to digital, then stored in a standard memory 128, such as an SRAM, and the read back out through a digital port. This typical implementation requires a precise analog-to-digital converter 124 to maintain signal integrity in the system 120. The input data to the buffer 126 is digital, requiring N-input and N-output bits, which can be driven either serially or parallel, each of which consumes significant power. Accordingly, there is a need for new and improved systems and methods for natural language processing that conserve power by performing the entire language processing function in the analog domain.

Summary

According to some embodiments, an all-analog natural language processing system is provided. Analog audio input is processed directly by an all-analog signal pathway wherein the audio activity detection, voice activity detection, feature extraction and neural network processing are all performed in the analog domain. Audio/voice detection and feature extraction is performed by a bandpass filter bank having a plurality of individual bandpass filters. Each bandpass filter includes an array of individual capacitively coupled current conveyor second order sections having a charge-trapped transistor as a programmable element for tuning the passband of the filter. Compared to typical digital systems for natural language processing, the present all-analog system can perform natural language processing with comparable accuracy but greatly reduced energy consumption of up to two orders of magnitude less.

[0007] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.

Brief Description of the Drawings

[0008] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0009] FIG. 1 A is a diagram of a typical digital keyword spotting system;

[0010] FIG. 1 B is a diagram of a typical digital audio buffer;

[0011] FIG. 2 is a diagram of an all-analog natural language processing system, according to an embodiment;

[0012] FIG. 3 is a diagram of an analog bandpass filter, according to an embodiment;

[0013] FIG. 4 is a diagram of a bandpass channel for feature extraction, according to an embodiment;

[0014] FIG. 5 is a block diagram of an always-on analog natural language processing system, according to an embodiment; and

[0015] FIG. 6 is a phase-based audio buffer, according to an embodiment.

Detailed Description

[0016] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0017] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0018] Each program is preferably implemented in a high-level procedural or object- oriented programming and/or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0019] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0020] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0021] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0022] Referring to FIG. 2, shown therein is an all-analog natural language processing system 200, according to an embodiment. The system 200 includes an audio sensor, for example, analog microphone 202 for receiving audio/voice input. The microphone’s 202 analog signal is processed directly by an all-analog signal pathway wherein the audio activity detection, voice activity detection, feature extraction for natural language processing and neural network processing are all performed in the analog domain. By providing an all-analog processing pathway, the system 200 has an energy consumption on the order of 1 -10 pW - nearly two orders of magnitude less than the digital system 100 in FIG. 1A.

[0023] The system 200 includes a programmable analog bandpass filter bank 204 for audio/voice detection and feature extraction. The filter bank 204 is configurable to have between 4 and 20 programmable analog bandpass filters configured for voice or sound detection and feature extraction 206. The analog bandpass filters are described in more detail with reference to Figure 3, below. The filter bank 204 may include an analog peak detector circuit (not shown) for thresholding and detecting peak energy in a frequency band or channel of interest. The filter bank 204 may further include an integrator (see FIG. 4) for outputting the accumulated energy within a particular frequency band over a given period of time.

[0024] The filter bank 204 may approximate the response of the basilar membrane (in a human ear) to receiving audio/voice input. For example, the filter bank 204 may be configured to model the velocity coupling of the cochlear fluid with the cilia of the inner hair cell by a time derivative. [0025] Beneficially, the analog filter bank 204 eliminates the need to convert the raw analog signal to a digital signal, thereby conserving significant power consumption, and further performs the feature extraction function of the LFBC/MFCC feature extractor 106 in FIG. 1A. Furthermore, the analog filter bank 204 may be more robust to audio signal noise compared to digital systems (i.e. , MFCC feature extraction techniques).

[0026] The analog filter bank 204 feeds directly into a neural network 208, which can be of several different types with a recursive neural network (RNN) 208 shown in Figure 2. The uniqueness of this solution is utilizing an all-analog signal processing chain from the microphone 202 through the classification output 212. The RNN 208 may include ~ 50K parameters for audio/voice feature classification. The RNN 208 may be configured to process up to 100 inferences in a 1 second query. The RNN 208 may be configured to support voice activity detection (VAD) and/or audio activity detection (AAD). In certain embodiments, the bank of band pass filters may be eliminated and the RNN can be enhanced to extract the features as well.

[0027] The all-analog neural network 208 can be built in many ways but the preferred embodiment utilizes a neural signal processor in which charge trap transistors (CTT) perform parameter storage and multiplication and a conversion block performs accumulation and activation generation, for example, as disclosed in U.S. Patent Publication No. 2022/0374698, wholly incorporated by reference herein.

[0028] Referring to FIG. 3, shown there is a diagram of an analog bandpass filter 300, according to an embodiment. The analog bandpass filter 300 utilizes CTTs 302a, 302b, 302c as the programmable or adjustable element in a second-order charge-coupled conveyor filter structure. A conveyor filter structure is typically implemented using floatinggate analog integrated circuit (VLSI) technology and may be adapted for use in the allanalog system 200 by inclusion of CTTs 302a, 302b, 302c into the conveyor filter structure of the analog bandpass filter 300 to program the filter characteristics.

[0029] The bandpass filter 300 is a continuous-time bandpass filter and may include an array of individual capacitively coupled current conveyor second order sections 304a, 304b, 304c. Each second order section 304a, 304b, 304c includes a CTT 302a, 302b, 302c as a programmable element for tuning the corner frequencies of the bandpass filter 300. The corner frequencies may be set independently of each other, thus the bandwidth of the filter 300 may be tuned as required to isolate a given frequency or passband. For example, the array of second order sections 304a, 304b, 304c with exponentially spaced center frequencies forms a good model of the frequency response of the human cochlea.

[0030] The analog bandpass filter 300 may correspond to one channel in a multichannel analog bandpass filter bank 204 having a plurality of 4 to 20 individual analog bandpass filters 300. In such case, each analog bandpass filter 300 (and each channel) may be spaced to cover a segment of the frequency range from 100 Hz to 4000 Hz. Having multiple analog bandpass filters 300 in the filter bank 204 provides for parallel computation ability that consumes orders of magnitude less power than similar digital systems.

[0031] Referring to FIG. 4, shown therein is a diagram of a (single) bandpass channel 400 for feature extraction, according to an embodiment. The bandpass channel 400 may be one of several bandpass channels for feature extraction implemented in the all-analog natural language processing system shown in FIG. 2.

[0032] Audio signal 402 from a microphone passes through a bandpass filter 403 in an analog filter bank 404. Each output of the bandpass filter 403 is followed by an integrator 405 to provide an output that represents the accumulated energy within the particular frequency band of the bandpass filter 403 over a given period of time, the integration time. The integrator 405 may be a variety of known implementations for signal integration.

[0033] The outputs of the integrators 405 in the analog filter bank 404 provides a frequency spectrograph 406 of the audio signal 402 input and is thus capable of replacing the typical MFCC function in digital systems. The frequency spectrograph 406 is processed by a neural network configured for audio activity detection, voice activity detection and natural language processing.

[0034] Referring to FIG. 5, shown there is a diagram of an always-on all-analog natural language processing system 500, according to an embodiment. The always-on audio system 500 includes an analog microphone 502. According to some embodiments, the microphone 502 is an analog micro electro-mechanical system (MEMS) microphone 502. [0035] Audio/voice input to the microphone 502 is received by an always-on all-analog audio processing system 504. The audio processing system 504 may be a system-on-a- chip, that includes analog audio processing components. According to an embodiment, the audio processing system 504 may be combined with the microphone 502, for example, to form a system in module.

[0036] The all-analog audio processing system 504 includes an audio activity detector 506. The audio activity detector 506 may include an analog bandpass filter bank (e.g., bandpass filter bank 202 in FIG. 2) configured for noise cancellation and audio band energy thresholding. According to an embodiment, the audio activity detector 506 may be added to the microphone 502 itself.

[0037] The all-analog audio processing system 504 includes a voice activity detector 508. The voice activity detector 506 may include an analog bandpass filter bank (e.g., bandpass filter bank 202 in FIG. 2) configured for voice detection and feature extraction.

[0038] In digital systems (FIG. 1A) audio activity detection and voice activity detection are done as separate digital signal processing functions. Advantageously, according to various embodiments, the audio activity detector 506 and voice activity detection 508 may be the same analog filter bank that feeds into a neural network configured for audio activity detection, voice activity detection and/or natural language processing.

[0039] The all-analog audio processing system 504 includes one or more neural networks, for example, a wake work and keyword spotting RNN 510 configured for classifying features extracted by the voice activity detector 506 as wake words and keywords, respectively. Wake words may be voice commands for rousing a device from a sleep or a standby mode to a full operational mode. Keywords may be voice commands for a device to perform a particular act or function.

[0040] A simpler voice detect RNN 512 functions in parallel to the wake word and keyword spotting RNN 510 to allow the audio processing system 504 to look for wake words and keywords constantly. The output of wake word and keyword spotting RNN 510 is gated by output of the voice detect RNN 512, according to a gating function 514, to ensure that wake words/keywords are detected only when a voice is present to filter erroneous keyword detections in a noisy environment. [0041] In a typical digital always-on audio system, to save power, a wake word neural network is only turned on to detect/classify words when a voice detecting neural network detects the presence of a voice. By comparison, in the analog always-on natural language processing system 500, the benefit of constantly running the wake word and keyword spotting RNN 510 in parallel with the voice detection NN 512 is that the latency to respond to a positive detection of a keyword or wake word is reduced significantly.

[0042] According to an embodiment, the neural networks 510, 512 may include ~50K parameters for audio/voice feature classification. The neural networks 510, 512 may each be configured as 3 layers of neurons. The first layer is configured as a recurrent neural network for receiving up to 20 inputs of the extracted features from the audio activity detector 506 and/or voice activity detector 508 filter banks. A second hidden layer is configured to reduce the output of the first layer to a reduced number of activations. A third layer is configured to generate output classes of the keywords and the wake words. According to an embodiment, the voice activity detector 508 may be merged with the wake word RNN 510, wherein the RNN 510 is trained to provide an output that indicates that a voice is present.

[0043] The neural networks 510, 512 may be trained for wake work and keyword classification using a keyword and/or wake word classification dataset, for example, Google® Speech Commands or Mozilla™ Common Voice datasets. As an example, when trained with these specific datasets, the system 500 can detect the provided 20 wake words or keywords with >95% accuracy, comparable to traditional digital natural language processing systems. However, compared to digital systems, the energy consumption of the all-analog system 500 is on the order of two orders of magnitude lower.

[0044] Still referring to FIG. 5, the input analog signal from the analog microphone 502 is converted to time pulse widths or a phase representation utilizing a dual slope voltage to phase converter including an N-bit counter to generate the audio samples (see FIG. 6) which uses a gated ring oscillator as the time base. The audio samples are stored in a highly optimized, first-in-first-out (FIFO) sub-threshold memory structure 516 to minimize power consumption. The sub-threshold memory structure 516 may be an ephemeral memory structure disclosed in U.S. Patent Publication No. 2022/0374698.

[0045] The audio is replayed in several possible manners, including, but not limited to, outputting the digital phase representations directly, or regenerating the analog form of the pulse widths. During regeneration, the gated ring oscillator is used as the time base such that the acquisition of input voltages and regeneration of output signals is all based upon the same time reference. The audio buffer 516 is limited in the amount of time in which any storage element needs to hold any value in any particular memory location, thus the memory elements used to store the phase information can be dynamic, further reducing the power consumption. The audio phase information and detected keywords/wake words output from the audio processing system 504 are received and processed by a microcontroller 518 or processor.

[0046] Referring to FIG. 6, shown therein is a diagram of a phase-based audio buffer 600, according to an embodiment. The audio buffer 600 may be the audio buffer 516 in FIG. 5. The audio buffer 600 includes a timing element 604. Advantageously, the timing element 604 used to capture the phase information of each audio sample does not need to be absolutely accurate relative to any other timing standard in the system. Since the same time base is used to capture the audio signal phase and regenerate the audio signal, the time base can be asynchronous. This allows for significant simplification of the timing element 604, which can be as simple as the gated ring oscillator 604 running in subthreshold.

[0047] The audio buffer 600 converts the audio voltage to a phase representation utilizing a dual-slope voltage to phase converter 605 including an N-bit counter 606 to digitize the audio phase representation for storage in an N-bit FIFO memory structure 608. The audio buffer 600 also converts the phase information to time pulse widths utilizing an N-bit counter 610 and a comparator 612. The comparator 612 compares the time pulses against zero or an arbitrary threshold as the counter 610 counts down and outputs the audio phase as an analog pulse using 1 pin for reduced power. By comparison, a typical digital solution (FIG. 1 B) with N-bit bit resolution will require ~N times as much power as the audio buffer 600. [0048] The systems 200, 400, 500 and 600 shown in FIGS. 2, 4, 5 and 6, respectively, may be adapted for applications beyond voice/keyword detection and classification and natural language processing. While the systems 200, 400, 500 and 600 include an analog microphone, the microphone may generally be any analog sensor for detecting vibrations or vibrating particles (e.g., sound waves, ultrasound waves, seismic waves, etc.). Accordingly, the systems 200, 400, 500 and 600 may be adapted for detection and classification of non-voice sounds (e.g., glass breaking), ultrasound features and seismic activity by appropriate tuning of the analog bandpass filters to extract relevant features and training a neural network to classify the relevant features.

[0049] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.