RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of United States Provisional Patent Application Number 62/826,258, which is hereby incorporated by reference in its entirety.

FIELD

[0002] The present disclosure relates generally to machine learning. More particularly, the present disclosure relates to direct and text-free speech-to-speech translation by machine- learned models such as, for example, sequence-to-sequence models.

BACKGROUND

[0003] Speech-to-speech translation (S2ST) refers to the process of translating speech in one language (e.g., as represented by a first speech waveform) into speech in another, different language (e.g., as represented by a second, different speech waveform).

Conventional S2ST systems rely on a cascaded approach in which multiple disparate systems are combined to perform the translation. In particular, conventional S2ST system are often broken into three components which operate separately and in sequence: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis.

[0004] Cascaded systems have the potential of compounding errors between components such as, for example, when recognition errors lead to larger translation errors. Cascaded systems also have other disadvantages, primarily in terms of increased computational requirements and higher inference latency since multiple distinct computational steps are necessary. In addition, cascaded systems struggle to retain paralinguistic and non-linguistic information during translation such as, for example, maintaining the source speaker’s voice, intonation, emotion, and prosody, in the synthesized translated speech. Finally, cascaded systems are unable to learn to generate fluent pronunciations of words which do not need be translated, such as names or other proper nouns.

SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments. [0006] One example aspect of the present disclosure is directed to a computing system. The computing system includes one or more processors. The computing system includes a machine-learned translation model configured to perform direct and text-free translation. The machine-learned translation model includes an encoder model configured to receive and process a series of input acoustic feature representations that are representative of a first speech in a first language to generate a series of hidden state representations. The machine- learned translation model includes an attention model configured to receive and process the series of hidden state representations to generate a series of attention outputs. The machine- learned translation model includes a decoder model configured to receive and process the series of attention outputs to generate a series of output acoustic feature representations that are representative of a second speech in a second language that is different than the first language. The computing system includes one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include obtaining the series of input acoustic feature representations that are representative of the first speech in the first language. The operations include inputting the series of input acoustic feature

representations into the encoder model of the machine-learned translation model. The operations include receiving the series of output acoustic feature representations that are representative of the second speech in the second language as an output of the decoder model.

[0007] The operations may further comprise backpropagating a loss function through the machine-learned translation model to train the machine-learned translation model. The loss function may evaluate a respective difference between each of the series of output acoustic feature representations and a corresponding ground truth acoustic feature representation derived from the second speech in the second language. One or more auxiliary speech recognition models may be configure to receive and process information from the encoder model to predict one or more items of textual content associated with the first speech in the first language or the second speech in the second language. The operations may further comprise backpropagating one or more auxiliary loss functions respectively associated with the one or more auxiliary speech recognition models through at least a portion of the encoder model to train at least the portion of the encoder model. Each auxiliary loss function evaluates a respective difference between the one or more items of textual content output by the corresponding auxiliary speech recognition model and a corresponding ground truth item of textual content associated with the first speech in the first language or the second speech in the second language. The one or more auxiliary speech recognition models may comprise one or more first speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the first speech in the first language. The one or more auxiliary speech recognition models may comprise one or more second speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the second speech in the second language.

[0008] The encoder model may comprise a multi-layer model. The information from the encoder model that is received and processed by the one or more first speech recognition models may be extracted from a relatively earlier layer of the encoder model relative to the information from the encoder model that is received and processed by the one or more second speech recognition models.

[0009] A speaker encoder model may be configured to receive and process data descriptive of reference speech uttered by a target speaker to generate a speaker embedding. The operations may further comprise inputting the speaker embedding into the attention model alongside the series of hidden state representations. During training of the machine- learned translation model, the target speaker may also be a second speaker that spoke the second speech in the second language. During inference performed by the machine-learned translation model, the target speaker may also be a first speaker that spoke the first speech in the first language.

[0010] The attention model may comprise a multi-headed attention model. The attention model may comprise a multi-headed attention model that performs additive attention. The machine-learned translation model may comprise a sequence-to-sequence model. One or both of the encoder model and the decoder model may comprise neural networks. One or both of the encoder model and the decoder model may comprise recurrent neural networks. One or both of the encoder model and the decoder model may comprise long short term memory recurrent neural networks. One or both of the encoder model and the decoder model may comprise bidirectional recurrent neural networks.

[0011] One or both of the series of input acoustic feature representations may comprise a series of input spectrogram. The series of output acoustic feature representations may comprise a series of output spectrograms. One or both of the input spectrograms may comprise logarithmic spectrograms or linear spectrograms. The output spectrograms may comprise logarithmic spectrograms or linear spectrograms. [0012] A vocoder may be configured to receive and process the series of output acoustic feature representations to generate a waveform of the second speech in the second language. The operations may further comprise inputting the series of output acoustic feature representations directly into the vocoder and receiving the waveform of the second speech as an output of the vocoder. The vocoder may comprise a Griffin-Lim vocoder or an autoregressive neural network.

[0013] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices configured to perform, or to enable to be performed, one or more of the methods described herein.

[0014] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0016] Figure 1 A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[0017] Figure IB depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0018] Figure 1C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0019] Figure 2 depicts a block diagram of an example machine-learned translation model according to example embodiments of the present disclosure.

[0020] Figure 3 depicts a block diagram of an example machine-learned translation model that includes encoder, attention, and decoder models according to example embodiments of the present disclosure.

[0021] Figure 4 depicts a block diagram of an example machine-learned translation model that is trained in a multi-task training scheme according to example embodiments of the present disclosure. [0022] Figure 5 depicts a block diagram of an example machine-learned translation model able to generate translations that sound similar to a target speaker according to example embodiments of the present disclosure.

[0023] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.

DETAILED DESCRIPTION

Overview

[0024] Generally, the present disclosure is directed to systems and methods that train and use machine-learned models such as, for example, sequence-to-sequence models, to perform direct and text-free speech-to-speech translation. In particular, aspects of the present disclosure provide an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. According to one aspect of the present disclosure, the machine-learned models described herein can be trained end-to-end, learning to map acoustic feature representations (e.g., spectrograms) of speech in a first language (e.g., Spanish) directly into acoustic feature representations (e.g., spectrograms) of speech in a second language (e.g., English). For example, the speech in the second language can correspond to translated content of the speech in the first language (e.g., which may also be in a different voice). According to another aspect of the present disclosure, additional techniques are provided which enable the machine-learned models to synthesize the translated speech using the voice of a target speaker (e.g., which may, in some instances, be the source speaker that is uttering the speech to be translated). According to yet another aspect of the present disclosure, multitask training can be performed by incorporating auxiliary decoder networks to predict the source or target transcripts, thereby improving performance of the machine- learned translation models. Finally, United States Provisional Patent Application Number 62/826,258, which is incorporated into and forms a portion of this disclosure, describes example experiments performed on example implementations of the systems and methods described herein which demonstrate the feasibility of the proposed approaches.

[0025] More particularly, example aspects of the present disclosure address the task of speech-to-speech translation (S2ST): translating speech in one language into speech in another. This application is highly beneficial for breaking down communication barriers between people who do not share a common language. Specifically, example systems are provided that can train an end-to-end model to accomplish this task directly, without relying on an intermediate text representation. This is in contrast to conventional S2ST systems which are often broken into three components: automatic speech recognition (ASR), text-to- text machine translation (MT), and text-to-speech (TTS) synthesis.

[0026] Cascaded systems have the potential of compounding errors between

components, e.g. recognition errors leading to larger translation errors. By training to solve the task end-to-end, direct S2ST models avoid this issue. They also have other advantages over cascaded systems, primarily in terms of reduced computational requirements and lower inference latency since only one decoding step is necessary, instead of three. In addition, end- to-end training naturally enables models to retain paralinguistic and non-linguistic information during translation, e.g. maintaining the source speaker’s voice, intonation, emotion, and prosody, in the synthesized translated speech. Finally, end-to-end approaches enable the models to learn to generate fluent pronunciations of words which do not need be translated, such as names or other proper nouns.

[0027] Thus, aspects of the present disclosure are directed to a machine-learned translation model that can perform direct and text-free translation and which can be trained in an end-to-end fashion. As one example, the machine-learned translation model can be a sequence-to-sequence model that includes an encoder model, an attention model, and a decoder model. The encoder model can be configured to receive and process a series of input acoustic feature representations that are representative of first speech in a first language to generate a series of hidden state representations. The attention model can be configured to receive and process the series of hidden state representations to generate a series of attention outputs. The decoder model can be configured to receive and process the series of attention outputs to generate a series of output acoustic feature representations that are representative of second speech in a second language that is different than the first language.

[0028] To train the machine-learned translation model, a computing system can obtain a set of training data that includes a number of pairs of training examples. Each training example can include data descriptive of first speech in a first language and data descriptive of second speech in a second language, wherein the second speech includes the same content (e.g., underlying expression or statement) as the first speech, but in a different language. As an example, the data descriptive of the first speech and second speech can be digital recordings of the first speech and second speech which may take the form of digitized speech waveforms. The computing system can generate a respective series of acoustic feature representations for both the first speech and the second speech. Example acoustic feature representations include various forms of spectrograms such as, as examples, linear frequency spectrograms or logarithmic frequency spectrograms (e.g., log-mel spectrograms). In particular, the series of acoustic feature representations generated based on the data descriptive of the second speech can be treated as ground truth acoustic feature

representations which the translation model will attempt to predict.

[0029] Specifically, the computing system can input the series of acoustic feature representations generated based on the data descriptive of the first speech in the first language into the machine-learned translation model. In response, the translation model can predict a series of output acoustic feature representations for the second speech in the second language. The computing system can evaluate a loss function that evaluates a respective difference between each of the series of output acoustic feature representations and a corresponding ground truth acoustic feature representation derived from the second speech in the second language. The computing system can backpropagate the loss function through the translation model (e.g., first through the decoder model, then through the attention model, and then through the encoder model) to train the translation model. For example, values of parameters of the translation model can be updated using a gradient descent approach. Thus, the translation model can learn to predict output acoustic feature representations that match the ground truth representations associated with the ground truth translated speech.

[0030] After training, the machine-learned translation model can operate to enable direct translation of speech from the first language to the second language. In particular, at inference time, a computing system (which may be the same or different than the system that performed training of the model) can obtain data descriptive of input speech in the first language (e.g., a digital waveform of the input speech in the first language). The computing system can generate a series of input acoustic feature representations (e.g., spectrograms) descriptive of the input speech in the first language and can provide the series of input acoustic feature representations as an input to the machine-learned translation model (e.g., as a direct input into the encoder model). The machine-learned translation model can generate a series of output acoustic feature representations associated with predicted speech in the second language. The computing system can generate, for example through use of a vocoder, a predicted speech waveform for the speech in the second language. In some

implementations, the computing system can cause playback of the speech waveform for the predicted speech in the second language in real-time, such that the input speech in the first language is quickly and efficiently translated directly into audible speech in the second language. [0031] In some respects, the S2ST models proposed herein resemble recent work on sequence-to-sequence models for voice conversion, which is the task of recreating an utterance in the same language but in a different person’s voice. For example, certain existing voice conversion approaches include an attention-based model to generate spectrograms in the target voice based on input features (e.g., spectrogram concatenated with bottleneck features from an speech recognition model) from the source voice. However, in contrast to the model proposed in the present disclosure, the alignment of input and output in the voice conversion task is much simpler. In addition, the existing voice conversion approaches only train models which are specific to a single input-output speaker pair (i.e. one-to-one voice conversion), whereas the models of the present disclosure are capable of many-to-one and many-to-many speaker configurations.

[0032] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.

Example Devices and Systems

[0033] Figure 1 A depicts a block diagram of an example computing system 100 according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0034] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0035] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0036] In some implementations, the user computing device 102 can store or include one or more machine-learned translation models 120. In some implementations, a machine- learned translation model 120 can totally run on the device-side (e.g., on the user computing device 102) without involving the server computing system 130. For example, the machine- learned translation models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine- learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Example machine-learned translation models 120 are discussed with reference to Figures 2-5.

[0037] In some implementations, the one or more machine-learned translation models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine-learned translation model 120 (e.g., to perform parallel speech-to-speech translation across multiple instances of input speech).

[0038] Additionally or alternatively, one or more machine-learned translation models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned translation models 140 can be implemented by the server computing system 140 as a portion of a web service (e.g., a speech-to-speech translation service). Thus, one or more models 120 can be stored and implemented at the user computing device 102 and/or one or more models 140 can be stored and implemented at the server computing system 130.

[0039] The user computing device 102 can also include one or more user input component 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0040] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0041] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0042] As described above, the server computing system 130 can store or otherwise include one or more machine-learned translation models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example models 140 are discussed with reference to Figures 2-5.

[0043] The user computing device 102 and/or the server computing system 130 can train the models 120 and/or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0044] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0045] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and/or 140 stored at the user computing device 102 and/or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0046] In particular, the model trainer 160 can train the machine-learned translation models 120 and/or 140 based on a set of training data 162. The training data 162 can include, for example, a number of pairs of training examples. Each training example pair can include data descriptive of first speech in a first language and data descriptive of second speech in a second language, wherein the second speech includes the same content (e.g., underlying expression or statement) as the first speech, but in a different language. As an example, the data descriptive of the first speech and the second speech can be digital recordings of the first speech and second speech which may take the form of digitized speech waveforms. Thus, to provide an example, a particular training example pair can include a first speech waveform that includes utterance of the Spanish sentence“Me gusta comer la cocina de mi abuela” and a second speech waveform that includes utterance of same content in English:“I like to eat my grandmother’s cooking”. The respective speakers of the Spanish and English utterances can be the same person or can be different people. Any different pair of languages can be used.

[0047] In some implementations, the training data 162 can further include textual transcriptions of the utterances (e.g., in both languages). For example, the textual

transcriptions can be manually generated by listening to the utterances and/or the utterances can be generated by reading pre-existing textual transcriptions. In yet further examples, some or all of the utterances included in training example pairs can be synthesized utterances generated using a text-to-speech system such as, for example, a Parallel WaveNet system.

[0048] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0049] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

[0050] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

[0051] Figure 1 A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.

[0052] Figure IB depicts a block diagram of an example computing device 10 according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0053] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0054] As illustrated in Figure IB, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0055] Figure 1C depicts a block diagram of an example computing device 50 according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0056] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some

implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0057] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 1C, a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0058] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

Example Model Arrangements

[0059] Figure 2 depicts a block diagram of an example machine-learned translation model 200 according to example embodiments of the present disclosure. In some

implementations, the machine-learned translation model 200 is trained to receive a series of input acoustic feature representations 204 descriptive of speech in a first language and, as a result of receipt of the series of input acoustic feature representations 204, provide a series of output acoustic feature representations 206 descriptive of speech in a second language. Thus, the machine-learned translation model 200 can directly translate from acoustic feature representations of speech in one language to acoustic feature representations of speech in another language.

[0060] The series of input acoustic feature representations 204 can be derived from data descriptive of first speech in a first language. For example, the data descriptive of the first speech can be a digital recording of the first speech which may, for example, take the form of a digitized speech waveform. Example acoustic feature representations include various forms of spectrograms such as, as examples, linear frequency spectrograms or logarithmic frequency spectrograms (e.g., log-mel spectrograms). In some implementations, one input spectrogram can be generated for each of a number of time steps into which the input waveform is discretized.

[0061] To train the machine-learned translation model 200, a computing system can obtain a set of training data that includes a number of pairs of training examples. Each training example pair can include data descriptive of first speech in a first language and data descriptive of second speech in a second language, wherein the second speech includes the same content (e.g., underlying expression or statement) as the first speech, but in a different language. As an example, the data descriptive of the first speech and second speech can be digital recordings of the first speech and second speech which may take the form of digitized speech waveforms. The computing system can generate a respective series of acoustic feature representations for both the first speech and the second speech. In particular, the series of acoustic feature representations generated based on the data descriptive of the second speech can be treated as ground truth acoustic feature representations which the translation model will attempt to predict.

[0062] Specifically, the computing system can input the series of input acoustic feature representations 204 generated based on the data descriptive of the first speech in the first language into the machine-learned translation model 200. In response, the translation model can predict the series of output acoustic feature representations 206 for the second speech in the second language. The computing system can evaluate a loss function that evaluates a respective difference between each of the series of output acoustic feature representations 206 and a corresponding ground truth acoustic feature representation derived from the second speech in the second language. The computing system can backpropagate the loss function through the translation model 200 to train the translation model 200. For example, values of parameters of the translation model 200 can be updated using a gradient descent approach. Thus, the translation model 20 can learn to predict output acoustic feature representations 206 that match the ground truth representations associated with the ground truth translated speech.

[0063] After training, the machine-learned translation model 200 can operate to enable direct translation of speech from the first language to the second language. In particular, at inference time, a computing system (which may be the same or different than the system that performed training of the model) can obtain data descriptive of input speech in the first language (e.g., a digital waveform of the input speech in the first language). The computing system can generate a series of input acoustic feature representations 204 (e.g., spectrograms) descriptive of the input speech in the first language and can provide the series of input acoustic feature representations 204 as an input to the machine-learned translation model 200. The machine-learned translation model can generate the series of output acoustic feature representations 206 associated with predicted speech in the second language. The number of output acoustic feature representations 206 can be the same as or different than the number of input acoustic feature representations 204. For example, in some implementations, two spectrogram frames can be predicted for each time step.

[0064] The computing system can generate, for example through use of a vocoder 208, a predicted speech waveform 210 for the speech in the second language. Thus, the vocoder 208 can convert the spectrograms into time-domain waveforms. Example vocoders include a Griffm-Lim vocoder and/or an autoregressive neural network such as the WaveRNN neural vocoder.

[0065] In some implementations, the computing system can cause playback of the predicted speech waveform 210 for the speech in the second language in real-time, such that the input speech in the first language is quickly and efficiently translated directly into audible speech in the second language.

[0066] Figure 3 depicts a block diagram of an example machine-learned translation model 300 according to example embodiments of the present disclosure. The machine- learned translation model 300 is similar to translation model 200 of Figure 2 except that translation model 300 is illustrated as including an encoder model 302, an attention model 306, and a decoder model 310.

[0067] The encoder model 302 can be configured to receive and process a series of input acoustic feature representations 204 that are representative of first speech in a first language to generate a series of hidden state representations 304. The encoder model 302 can be configured to operate directly on the feature representations 204 (e.g., spectrograms). Thus, in some implementations, the feature representations 204 (e.g., spectrograms) can be directly input into the encoder model 302 without any intermediate temporal reduction.

[0068] In some implementations, the encoder model 302 can have a significantly increased capacity in order to handle the challenging speech-to-speech translation task. For example, the encoder model 302 can be relatively deeper than encoders used for other tasks. As one example, the encoder model 302 can include a stack of 8 bidirectional LSTM layers.

[0069] The attention model can be configured to receive and process the series of hidden state representations 304 to generate a series of attention outputs 308. The decoder model can be configured to receive and process the series of attention outputs 308 to generate a series of output acoustic feature representations 206 that are representative of second speech in a second language that is different than the first language.

[0070] In some implementations, the decoder model 310 can have an architecture similar to the models described in United States Patent Application No. 16/058,640 titled

“Synthesizing Speech from Text Using Neural Networks”, which is incorporated herein by reference in its entirety. In particular, similar to the models described in United States Patent Application No. 16/058,640, the decoder model 310 can include pre-net, autoregressive LSTM stack, and post-net components. However, in some implementations, the architecture described in United States Patent Application No. 16/058,640 can be modified to use the multi -head additive attention (e.g., with 4 heads) for attention model 306, rather than a location-sensitive attention.

[0071] In some implementations, a significantly narrower 32 dimensional pre-net bottleneck can be used rather than a 256-dimension. In some implementations, a reduction factor of 2 can be used, such that two spectrogram frames are predicted for each decoder time step. In some implementations, the decoder model 310 can include 4, 6, or other numbers of LSTM layers. In some implementations, the decoder model 310 can use attention dropout, LSTM zone-out regularization, all with probability 0.1 For example, training can use the Adafactor optimizer with a batch size of 1024.

[0072] In one particular example, the encoder model 302 can map input acoustic feature representations 204 which take the form of 80-channel log-mel spectrogram features into the hidden state representations 304 which are passed to the attention model 306 which can perform an attention-based alignment mechanism to condition the decoder model 310 (e.g., which can be an autoregressive decoder) which, in turn, predicts the output acoustic feature representations 206 in the form of 1025-dim log-compressed spectrogram frames

corresponding to the translated speech. In some implementations, the input feature frames can be created by stacking (e.g., with deltas and accelerations) a number (e.g., 3) of adjacent frames of an 80-channel log-mel spectrogram.

[0073] Figure 4 depicts a block diagram of the example machine-learned translation model 300 of Figure 3 in which the translation model 300 is being trained in a multi-task training scheme according to example embodiments of the present disclosure. In particular, in the illustrated training scheme, the machine-learned translation model 300 is being trained alongside a plurality of auxiliary speech recognition models 402-412. [0074] Although six auxiliary speech recognition models 402-412 are illustrated, any number of different speech recognition models can be used (e.g., one, two, three, etc.). Thus, each of the auxiliary speech recognition models 402-412 is optional. Any combination of them may be used, and, in some embodiments, none of them are used. Furthermore, in some implementations, the auxiliary speech recognition models 402-412 are used only during training to assist in training the machine-learned translation model 300. In such

implementations, the auxiliary speech recognition models 402-412 are not used or implemented during inference.

[0075] Each of the auxiliary speech recognition models 402-412 is configured to receive and process information from the encoder model 302 to predict one or more items of textual content associated with the first speech in the first language or the second speech in the second language. As examples, the model 402 is configured to predict phonemes 452 of the speech in the second language; the model 404 is configured to predict graphemes 454 of the speech in the second language; the model 406 is configured to predict n-grams 456 of the speech in the second language; the model 408 is configured to predict phonemes 458 of the speech in the first language; the model 410 is configured to predict graphemes 460 of the speech in the first language; and the model 412 is configured to predict n-grams 462 of the speech in the first language.

[0076] To perform the multi-task learning scheme, in addition to the loss function that is based on the output acoustic feature representations 206, the computing system can further backpropagate one or more additional loss functions through at least a portion of the encoder model 302 to train at least the portion of the encoder model 302. In particular, each additional loss function can correspond to one of the models 402-412 and can evaluates a respective difference between each of the one or more items of textual content predicted by such model and a corresponding ground truth item of textual content associated with the first speech in the first language or the second speech in the second language. Thus, as one example, an additional loss function associated with model 402 might evaluate a difference between the predicted phonemes 452 of the speech in the second language to the ground truth phonemes actually contained in the speech of the second language. Thus, the additional loss function(s) can serve to regularize the encoder model 302 and assist the encoder model 302 in capturing information relevant to phonemes, graphemes, n-grams, and/or the like.

[0077] Although each of the auxiliary speech recognition models 402-412 is illustrated as including a respective attention model and respective decoder model, the models 402-412 may have different architectures than illustrated. In some implementations, the attention model in each of the auxiliary speech recognition models 402-412 is a single-head additive attention model. In some implementations, the decoder model in each of the auxiliary speech recognition models 402-412 is a 2-layer LSTM. In some implementations, the decoder model in each of the auxiliary speech recognition models 402-412 uses attention dropout, LSTM zone-out regularization, all with probability 0.1 For example, training can use the Adafactor optimizer with a batch size of 1024.

[0078] As illustrated in Figure 4, the output of the final layer of the encoder model 302 is passed to the primary attention and decoder models 306 and 310, whereas intermediate activations of the encoder model 302 are passed to auxiliary models 402-412, which predict textual sequences corresponding to the source and target speech. As illustrated, the information from the encoder model 302 that is received and processed by the auxiliary speech recognition models 408-412 that predict aspects of the first language is extracted from a relatively earlier layer of the encoder model 302 relative to the information from the encoder model 302 that is received and processed by the auxiliary speech recognition models 402-406 that predict aspects of the second language. This is because the early layers of the encoder model 302 are more likely to represent the first language content well, whereas deeper layers might learn to encode more information about the second language content.

[0079] Figure 5 depicts a block diagram of an example machine-learned translation model 300 able to generate translations that sound similar to a target speaker according to example embodiments of the present disclosure. In particular, as illustrated in Figure 5, the machine-learned translation model 300 can be trained and/or used in conjunction with an additional speaker encoder model 502.

[0080] The speaker encoder model 502 can obtain reference speech 504 from a target speaker and can generate a speaker embedding 506 descriptive of the target speaker’s speech. The speaker embedding 506 can be input into the attention model 306 alongside the series of hidden state representations 304. For example, a concatenation operation 508 can be performed to concatenate the speaker embedding 506 to (e.g., each of) the series of hidden state representations 304 prior to input into the attention model 306. The reference speech 504 can be in the first language or the second language.

[0081] In some implementations, during training of the machine-learned translation model 300, the target speaker that speaks the reference speech 504 is also a second speaker that spoke the second speech in the second language used to generate the ground truth acoustic feature representations. However, in some implementations, during inference performed by the machine-learned translation model 300, the target speaker that speaks the reference speech 504 is also a first speaker that spoke the first speech in the first language that is the object of the translation effort.

[0082] Thus, in order to control the output speaker identity (e.g., sound of the speech waveform produced from the predicted output acoustic feature representations 206), an optional speaker encoder model 502 can be incorporated into the system. For example, the speaker encoder model 502 can be a neural network. In some implementations, the speaker encoder model 502 can be discriminatively pretrained on a speaker verification task and used to condition the decoder model 310 by computing the speaker embedding 506 from the reference utterance 504. For example, the speaker embedding 506 can be 256 dimensions.

For example, the speaker encoder model 502 can be trained on a set of 85 IK speakers across 8 languages including English and Spanish.

[0083] In some implementations, use of the speaker encoding model 502 and the speaker embedding 506 can enable cross language voice conversion simultaneously with translation.

[0084] Example embodiments shown in the Figures can be combined. For example, the speaker encoding model 502 and the speaker embedding 506 can be used simultaneous with use of one or more of the auxiliary speech recognition models 402-412.

Additional Disclosure

[0085] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and

functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0086] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.