PRIORITY

[0001] This application claims the benefit of priority to U.S. Patent Application No. 16/048,235 filed on July 28, 2018 and U.S. Provisional Patent Application No. 62/539,194, filed July 31, 2017, titled "Microcontroller Interface for Audio Signal Processing," which is hereby incorporated by reference into this application in its entirety.

FIELD

[0002] Embodiments of the disclosure relate to the field of neuromorphic computing. More specifically, embodiments of the disclosure relate to, but are not limited to, a microcontroller interface for audio signal processing and methods thereof.

BACKGROUND

[0003] Traditional central processing units ("CPUs") process instructions based on "clocked time." Specifically, CPUs operate such that information is transmitted at regular time intervals. Based on complementary metal-oxide-semiconductor ("CMOS") technology, silicon-based chips may be manufactured with more than 5 billion transistors per die with features as small as 10 nm. Advances in CMOS technology have been parlayed into advances in parallel computing, which is used ubiquitously in cell phones and personal computers containing multiple processors.

[0004] However, as machine learning is becoming commonplace for numerous applications including bioinformatics, computer vision, video games, marketing, medical diagnostics, online search engines, etc., traditional CPUs are often not able to supply a sufficient amount of processing capability while keeping power consumption low. In particular, machine learning is a subsection of computer science directed to software having the ability to learn from and make predictions on data. Furthermore, one branch of machine learning includes deep learning, which is directed at utilizing deep (multilayer) neural networks.

[0005] Currently, research is being done to develop direct hardware implementations of deep neural networks, which may include systems that attempt to simulate "silicon" neurons (e.g., "neuromorphic computing"). Neuromorphic chips (e.g., silicon computing chips designed for neuromorphic computing) operate by processing instructions in parallel (e.g., in contrast to traditional sequential computers) using bursts of electric current transmitted at non-uniform intervals. As a result, neuromorphic chips require far less power to process information, specifically, artificial intelligence ("AI") algorithms. To accomplish this, neuromorphic chips may contain as much as five times as many transistors as a traditional processor while consuming up to 2000 times less power. Thus, the development of neuromorphic chips is directed to provide a chip with vast processing capabilities that consumes far less power than conventional processors. Further, neuromorphic chips are designed to support dynamic learning in the context of complex and unstructured data.

[0006] There is an ongoing need for the development and enhancement of special-purpose processing capabilities such as that found in, for example, hearing aids. Provided herein are systems and methods for enhancing special-purpose processing capabilities for special- purpose processors operable as stand-alone processors utilizing a microcontroller interface.

SUMMARY

[0007] Disclosed herein is a neuromorphic-processing system including, in some embodiments, a special-purpose host processor operable as a stand-alone host processor; a neuromorphic co-processor including an artificial neural network; and a communications interface between the host processor and the co-processor configured to transmit information therebetween. The co-processor is configured to enhance special-purpose processing of the host processor with the artificial neural network.

[0008] In some embodiments, the host processor is a hearing-aid processor configured to transmit frequency elements or signal spectrum information to the co-processor in the form of Fourier transforms over a serial communications interface as the communications interface. The co-processor further includes a demultiplexer configured to demultiplex serial signals from the serial communications interface into parallel signals for a plurality of inputs of the artificial neural network.

[0009] In some embodiments, the communications interface between the host processor and the co-processor is a serial peripheral interface ("SPI") bus or inter-integrated circuit ("I ² C") bus.

[0010] In some embodiments, the co-processor is configured to enhance the special-purpose processing of the hearing-aid processor by providing information to the hearing-aid processor over the communications interface, thereby enabling the hearing-aid processor to selectively suppress noise and enhance desired signals.

[0011] In some embodiments, the co-processor is configured to enhance the special -purpose processing of the hearing-aid processor by providing a frequency mask to the hearing-aid processor over the communications interface, thereby indicating noise frequencies to suppress and signal frequencies to boost.

[0012] In some embodiments, the frequency mask is a set of attenuation or amplification factors corresponding to a set of frequencies to be suppressed or boosted for each of a number of time steps in an audio sample. [0013] In some embodiments, the artificial neural network is disposed in an analog multiplier array of a plurality of two-quadrant multipliers in a memory sector of the multi-chip neuromorphic-processing system.

[0014] In some embodiments, synaptic weights of the artificial neural network are stored in firmware of the neuromorphic co-processor. The firmware is configured for cloud-based updates to update the synaptic weights of the artificial neural network.

[0015] In some embodiments, the host processor and the co-processor are embodied in a single monolithic integrated circuit, a stacked die assembly, a multi-chip module, or separate integrated circuits of separate modules. The neuromorphic-processing system is configured to operate on battery power.

[0016] Also disclosed herein is a neuromorphic processor including, in some embodiments, a plurality of interface circuits including a demultiplexer configured to demultiplex serial signals into parallel signals; and a multi-layered artificial neural network configured to receive the parallel signals from the interface circuits. The serial signals are received by the interface circuits from a serial communications interface between the neuromorphic processor and a special-purpose host processor. The neuromorphic processor is configured to enhance special- purpose processing of the host processor with the artificial neural network.

[0017] In some embodiments, the host processor is a hearing-aid processor. The neuromorphic processor is configured to receive frequency elements or signal spectrum information from the hearing-aid processor in the form of Fourier transforms over the serial communications interface.

[0018] In some embodiments, the neuromorphic processor is configured to enhance the special-purpose processing of the hearing-aid processor by providing information to the hearing-aid processor over the serial communications interface, thereby enabling the hearing- aid processor to selectively suppress noise and enhance desired signals.

[0019] In some embodiments, the neuromorphic processor is configured to enhance the special-purpose processing of the hearing-aid processor by providing a frequency mask to the hearing-aid processor over the serial communications interface, thereby indicating noise frequencies to suppress and signal frequencies to boost.

[0020] In some embodiments, the frequency mask is a set of attenuation or amplification factors corresponding to a set of frequencies to be suppressed or boosted for each of a number of time steps of an audio sample.

[0021] In some embodiments, the artificial neural network is disposed in an analog multiplier array of a plurality of two-quadrant multipliers in a memory sector of the neuromorphic processor.

[0022] Also disclosed is a method of a neuromorphic-processing system having a special- purpose host processor and a neuromorphic co-processor including, in some embodiments, enhancing special-purpose processing of the host processor with an artificial neural network of the co-processor. The host processor is operable as a stand-alone host processor.

[0023] In some embodiments, the method further includes transmitting frequency elements or signal spectrum information from the host processor configured as a hearing-aid processor to the co-processor in the form of Fourier transforms over a serial communications interface; and demultiplexing serial signals from the serial communications interface with a demultiplexer of the co-processor into parallel signals for a plurality of inputs of the artificial neural network.

[0024] In some embodiments, enhancing the special-purpose processing of the hearing-aid processor includes providing information to the hearing-aid processor over the serial communications interface, thereby enabling the hearing-aid processor to selectively suppress noise and enhance desired signals.

[0025] In some embodiments, enhancing the special-purpose processing of the hearing-aid processor includes providing a frequency mask to the hearing-aid processor over the serial communications interface, thereby indicating noise frequencies to suppress and signal frequencies to boost. [0026] In some embodiments, the method further includes updating synaptic weights of the artificial neural network. The synaptic weights of the artificial neural network are stored in firmware of the neuromorphic co-processor configured for cloud-based updates.

DRAWINGS

[0027] Embodiments of this disclosure are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

[0028] FIG. 1 provides a schematic illustrating a system 100 for designing and updating neuromorphic integrated circuits ("ICs") in accordance with some embodiments.

[0029] FIG. 2 provides a schematic illustrating an analog multiplier array in accordance with some embodiments.

[0030] FIG. 3 provides a schematic illustrating an analog multiplier array in accordance with some embodiments.

[0031] FIG. 4 provides a schematic illustrating a bias-free, two-quadrant multiplier of an analog multiplier array in accordance with some embodiments.

[0032] FIG. 5 provides a schematic illustrating a microcontroller interface between a coprocessor of a neuromorphic IC and a host processor of a special-purpose IC in accordance with some embodiments.

[0033] FIG. 6 provides a schematic illustrating a method of a neuromorphic IC for enhancing a signal-to-noise ratio in accordance with some embodiments.

DESCRIPTION

TERMINOLOGY

[0034] In the following description, certain terminology is used to describe features of the invention. For example, in certain situations, the term "logic" may be representative of hardware, firmware and/or software that is configured to perform one or more functions. As hardware, logic may include circuitry having data processing or storage functionality. Examples of such circuitry may include, but are not limited or restricted to a microprocessor, one or more processor cores, a programmable gate array, a microcontroller, a controller, an application specific integrated circuit, wireless receiver, transmitter and/or transceiver circuitry, semiconductor memory, or combinatorial logic.

[0035] The term "process" may include an instance of a computer program (e.g., a collection of instructions, also referred to herein as an application). In one embodiment, the process may be included of one or more threads executing concurrently (e.g., each thread may be executing the same or a different instruction concurrently).

[0036] The term "processing" may include executing a binary or script or launching an application in which an object is processed, wherein launching should be interpreted as placing the application in an open state and, in some implementations, performing simulations of actions typical of human interactions with the application.

[0037] The term "object" generally refers to a collection of data, whether in transit (e.g., over a network) or at rest (e.g., stored), often having a logical structure or organization that enables it to be categorized or typed. Herein, the terms "binary file" and "binary" will be used interchangeably.

[0038] The term "file" is used in a broad sense to refer to a set or collection of data, information or other content used with a computer program. A file may be accessed, opened, stored, manipulated or otherwise processed as a single entity, object or unit. A file may contain other files and may contain related or unrelated contents or no contents at all. A file may also have a logical format or be part of a file system having a logical structure or organization of plural files. Files may have a name, sometimes called simply the "filename," and often appended properties or other metadata. There are many types of files, such as data files, text files, program files, and directory files. A file may be generated by a user of a computing device or generated by the computing device. Access and/or operations on a file may be mediated by one or more applications and/or the operating system of a computing device. A filesystem may organize the files of the computing device of a storage device. The filesystem may enable tracking of files and enable access of those files. A filesystem may also enable operations on a file. In some embodiments the operations on the file may include file creation, file modification, file opening, file reading, file writing, file closing, and file deletion.

[0039] The term "host processor" includes a primary processor such as a CPU or digital signal processor (DSP) of an IC in a system. The host processor is one that can stand alone but benefits from addition of neuromorphic computing provided by a neuromorphic IC or processor thereof through the microcontroller interface.

[0040] The term "co-processor" includes an additional processor interfaced with the host processor through a microcontroller interface. The additional processor can be configured to perform functions not possible with the host processor alone or functions the co-processor can perform faster or at lower power.

[0041] The term "enhancement filter" includes a filter configured to suppress undesired noise in a signal by selectively attenuating or boosting certain components of the signal on a time- varying basis. Likewise, "enhancement filtering" includes filtering to suppress undesired noise in a signal by selectively attenuating or boosting certain components of the signal on a time- varying basis.

[0042] Lastly, the terms "or" and "and/or" as used herein are to be interpreted as inclusive or meaning any one or any combination. Therefore, "A, B or C" or "A, B and/or C" mean "any of the following: A; B; C; A and B; A and C; B and C; A, B and C." An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive. [0043] Referring now to FIG. 1, a schematic illustrating a system 100 for designing and updating neuromorphic ICs is provided in accordance with some embodiments. As shown, the system 100 can include a simulator 110, a neuromorphic synthesizer 120, and a cloud 130 configured for designing and updating neuromorphic ICs such as neuromorphic IC 102. As further shown, designing and updating neuromorphic ICs can include creating a machine learning architecture with the simulator 110 based on a particular problem. The neuromorphic synthesizer 120 can subsequently transform the machine learning architecture into a netlist directed to the electronic components of the neuromorphic IC 102 and the nodes to which the electronic components are connected. In addition, the neuromorphic synthesizer 120 can transform the machine learning architecture into a graphic database system ("GDS") file detailing the IC layout for the neuromorphic IC 102. From the netlist and the GDS file for the neuromorphic IC 102, the neuromorphic IC 102, itself, can be fabricated in accordance with current IC fabrication technology. Once the neuromorphic IC 102 is fabricated, it can be deployed to work on the particular problem for which it was designed. While the initially fabricated neuromorphic IC 102 can include an initial firmware with custom synaptic weights between the nodes, the initial firmware can be updated as needed by the cloud 130 to adjust the weights. Being as the cloud 130 is configured to update the firmware of the neuromorphic IC 102, the cloud 130 is not needed for everyday use.

[0044] Neuromorphic ICs such as the neuromorphic IC 102 can be up to lOOx or more energy efficient than graphics processing unit ("GPU") solutions and up to 280x or more energy efficient than digital CMOS solutions with accuracies meeting or exceeding comparable software solutions. This makes such neuromorphic ICs suitable for battery powered applications.

[0045] Neuromorphic ICs such as the neuromorphic IC 102 can be configured for application specific standard products ("ASSPs") including, but not limited to, keyword spotting, voice recognition, one or more audio filters, speech enhancement, gesture recognition, image recognition, video object classification and segmentation, or autonomous vehicles including drones. For example, if the particular problem is one of speech enhancement, the simulator 110 can create a machine learning architecture with respect to one or more aspects of speech enhancement. The neuromorphic synthesizer 120 can subsequently transform the machine learning architecture into a netlist and a GDS file corresponding to a neuromorphic IC for speech enhancement, which can be fabricated in accordance with current IC fabrication technology. Once the neuromorphic IC for speech enhancement is fabricated, it can be deployed to work on speech enhancement in or with, for example, a hearing system or device as detailed further herein.

[0046] Neuromorphic ICs such as the neuromorphic IC 102 can be deployed in toys, sensors, wearables, augmented reality ("AR") systems or devices, virtual reality ("VR") systems or devices, mobile systems or devices, appliances, Internet-of-things ("IoT") devices, or hearing systems or devices.

[0047] Referring now to FIG. 2, a schematic illustrating an analog multiplier array 200 is provided in accordance with some embodiments. Such an analog multiplier array can be based on a digital NOR flash array in that a core of the analog multiplier array can be similar to a core of the digital NOR flash array or the same as a core of the digital NOR flash array. That said, at least select and read-out circuitry of the analog multiplier array are different than a digital NOR array. For example, output current is routed as an analog signal to a next layer rather than over bit lines going to a sense-amp/comparator to be converted to a bit. Word-line analogs are driven by analog input signals rather than a digital address decoder. Furthermore, the analog multiplier array 200 can be used in neuromorphic ICs such as the neuromorphic IC 102. For example, a neural network can be disposed in the analog multiplier array 200 in a memory sector of a neuromorphic IC.

[0048] Since the analog multiplier array 200 is an analog circuit, input and output current values can vary in a continuous range instead of simply on or off. This is useful for storing weights (aka coefficients) of a neural network as opposed to digital bits. In operation, the weights are multiplied by input current values to provide output current values that are combined to arrive at a decision of the neural network.

[0049] The analog multiplier array 200 can utilize standard programming and erase circuitry to generate tunneling and erase voltages. [0050] Referring now to FIG. 3, a schematic illustrating an analog multiplier array 300 is provided in accordance with some embodiments. The analog multiplier array 300 can use two transistors (e.g., a positive metal-oxide-semiconductor field-effect transistor ["MOSFET"] and a negative MOSFET) to perform a two-quadrant multiplication of a signed weight (e.g., a positive weight or a negative weight) and a non-negative input current value. If an input current value is multiplied by a positive or negative weight, the product or output current value can respectively be either positive or negative. A positively weighted product can be stored in a first column (e.g., column corresponding to Iouto+ in the analog multiplier array 300), and a negatively weighted product can be stored in a second column (e.g., column corresponding to Iou _t o- in the analog multiplier array 300). The foregoing positively and negatively weighted products or output signal values can be taken as a differential current value to provide useful information for making a decision.

[0051] Referring now to FIG. 4, a schematic illustrating a bias-free two-quadrant multiplier 400 of an analog multiplier array such as the analog multiplier array 300 is provided in accordance with some embodiments. Because each output current from the positive (e.g., Ml of the two-quadrant multiplier 400) or negative transistor (e.g., M2 of the two-quadrant multiplier 400) is wired to ground and proportional to the product of the input current value and the positive or negative weight, respectively, the power consumption of the positive or negative transistor is at or near zero when the input current values or weights are at or near zero. That is, if the input signal values are Ό,' or if the weights are Ό,' then no power will be consumed by the corresponding transistors of the analog multiplier array 300. This is significant because in many neural networks, often a large fraction of the values or the weights are Ό,' especially after training. Therefore, energy is saved when there is nothing to do or going on. This is unlike differential pair-based multipliers, which consume a constant current (e.g., by means of a tail bias current IB) regardless of the input signal.

[0052] Referring now to FIG. 5, a schematic is provided illustrating a microcontroller interface 500 between a co-processor of a neuromorphic IC 502 and a host processor of a special-purpose IC 504 in accordance with some embodiments. While the neuromorphic IC 502 and the special-purpose IC 504 are shown as two distinct ICs in FIG. 5, such as two distinct ICs of a multi-chip module or stacked die assembly, it should be understood the foregoing ICs can alternatively be embodied in a single monolithic IC. As such, description regarding the microcontroller interface 500 between the co-processor of the neuromorphic IC 502 and the host processor of the special-purpose IC 504 should also be taken to describe the microcontroller interface 500 between a neuromorphic co-processor and special-purpose host processor of a single monolithic IC; that is, unless context clearly indicates otherwise.

[0053] As shown in FIG. 5, the special-purpose IC 504 can include, but is not limited to, an IC for hearing systems or devices including hearing aids. An IC for a hearing aid can include a hearing-aid host processor (see "Core Proprietary Processing" in FIG. 5) as the special- purpose host processor, as well as a microphone, a filter bank, a speaker driver, and a speaker. However, it should be understood the neuromorphic co-processor can be interfaced with any one or more of a number of special-purpose host processors. The example embodiment of the neuromorphic co-processor interfaced with the hearing-aid host processor is merely provided for exposition. It should be understood that extension to other ASSPs might need certain modifications in line with the specific functions of the other ASSPs.

[0054] Adverting back to the particular problem of speech enhancement, there are efforts to solve the so-called cocktail party problem in hearing aids. The cocktail party problem occurs in settings such as a restaurant or party, where a hearing-impaired person with hearing aids is trying to hold a conversation with one person while there are many other people holding simultaneous conversations within earshot. It can be nearly impossible for the hearing- impaired person to pick out the conversation with the one person (desired signals) among all the many other people holding conversations (noise signals) within earshot when the hearing aids amplify all signals.

[0055] Deep learning networks can learn to look at speech in noisy environments such as that of the cocktail party problem and identify a pattern of frequency masks that can suppress the noise signals and pass through the desired signals. In step with the description for FIG. 1, the simulator 110 can create a machine learning architecture with respect to one or more aspects of speech enhancement including the conversation with the one person. The neuromorphic synthesizer 120 can subsequently transform the machine learning architecture into a netlist and a GDS file corresponding to a neuromorphic IC for speech enhancement, which can be fabricated in accordance with current IC fabrication technology - optionally as the above- referenced single monolithic IC - and deployed such that the neuromorphic co-processor can work as an audio co-processor on speech enhancement in or with, for example, a hearing system or device.

[0056] As shown in FIG. 5, the neuromorphic IC 502 or a portion of the above-referenced single monolithic IC corresponding to the neuromorphic IC 502 can include, but is not limited to, i) one or more interface circuits including a demultiplexer configured to demultiplex serial signals into parallel signals and ii) a deep-speech enhancement network with cloud-updateable synaptic weights configured to generate from the parallel signals ratio masks that separate desired speech signals from noise signals with reasonable sound quality. Again, the hearing- aid IC 504 or a portion of the above-referenced single monolithic IC corresponding to the hearing-aid IC 504 can include, but is not limited to, the hearing-aid host processor, the microphone, the filter bank, the speaker driver, and the speaker. The neuromorphic IC 502 and the hearing-aid IC 504 can be configured to communicate over a digital inter-chip interface such as a serial communications interface includes a SPI bus or I ² C bus.

[0057] The hearing-aid host processor can be configured to transmit frequency elements or signal spectrum information in the form of Fourier transforms or similar frequency decompositions over the SPI bus to the neuromorphic co-processor for audio signal-processing tasks such as audio enhancement or de-noising. The neuromorphic co-processor can be configured to subsequently perform the audio-signal processing tasks and transmit results back over the SPI bus to the hearing-aid host processor, the results including optimum weighting coefficients for frequencies in the form of a frequency or ratio mask. The hearing-aid host processor can be configured to subsequently aggregate the signals and deliver the aggregated signals to the hearing-impaired person with the hearing aid in accordance with the frequency mask. For example, the frequency mask could indicate the hearing aid should suppress frequencies in the 200-230 Hz range (mostly noise signals) and boost frequencies between 300-360 Hz (mostly desired signals).

[0058] With respect to generating the frequency or ratio mask, the neuromorphic co-processor can be configured to break up the frequency elements or signal spectrum information transmitted by the hearing-aid host processor into different frequency components at every time step. At every time step, the deep-speech-enhancement network can be configured to determine whether to attenuate or amplify certain frequency components based upon the extent to which they are mostly noise signals or mostly desired signals. The frequency mask then is the set of attenuation or amplification factors at each given time step that indicates the set of frequencies that should suppressed or boosted.

<■

Methods

[0059] A method of an integrated circuit having a special-purpose host processor and a neuromorphic co-processor includes, in some embodiments, enhancing special-purpose processing of the otherwise stand-alone host processor with an artificial neural network of the co-processor. As set forth herein, the host processor can be a hearing-aid processor. Enhancing the special-purpose processing of the hearing-aid processor includes providing information to the hearing-aid processor over the serial communications interface, thereby enabling the hearing-aid processor to selectively suppress noise and enhance desired signals. Alternatively, or additionally, enhancing the special-purpose processing of the hearing-aid processor includes providing a frequency mask to the hearing-aid processor over the serial communications interface, thereby indicating noise frequencies to suppress and signal frequencies to boost.

[0060] The method of the integrated circuit can further include transmitting frequency elements or signal spectrum information from the host processor (e.g., the hearing-aid processor) to the co-processor in the form of Fourier transforms over the serial communications interface. In a next operation, the method can further include demultiplexing serial signals from the serial communications interface with the demultiplexer of the co-processor into parallel signals for the plurality of inputs of the artificial neural network.

[0061] The method can further include updating the synaptic weights of the artificial neural network. The synaptic weights of the artificial neural network can be stored in the firmware of the integrated circuit configured for cloud-based updates.

[0062] Referring now to FIG. 6, a schematic is provided illustrating a method 600 of a neuromorphic IC for enhancing a signal-to-noise ratio in accordance with some embodiments. As shown, the method includes 1) a first step 610 where the hearing-aid host processor computes frequency components; 2) a second step 620 where the hearing-aid host processor transmits the frequency components to the neuromorphic co-processor; 3) a step 630 where the neuromorphic co-processor performs inference and computes enhancement filter coefficients from the frequency components; 4) a step 640 where the neuromorphic co-processor transmits the enhancement filter coefficients to the hearing-aid host processor; and 5) a step 650 where the neuromorphic co-processor waits until the next time step.

[0063] In view of the foregoing, the power of deep learning networks can be used to enhance at least speech in noisy environments such as that of the cocktail party problem and identify a pattern of frequency masks that can suppress the noise signals and pass through the desired signals. Thereby, the neuromorphic co-processors provided herein can enhance special- purpose host processors using algorithms the host processors cannot do alone, all while the special-purpose host processors perform the functions (e.g., noise cancellation, impulse noise detection, control functions [e.g., volume, battery power tracking, etc.], and the like for hearing-aid host processors) they are designed to do.

[0064] In the foregoing description, the invention is described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.