BACKGROUND

1. Field

[1] The present disclosure relates to methods and apparatus for performing speech recognition, particularly, adaptive permutation invariant training for multi-talker speech recognition.

2. Description of the Related Art

[2] In related art, many attempts have been made to address a "label ambiguity" or a "label permutation" problem in a speech recognition scenario where multiple talkers speak at the same time and only a single channel of mixed speech is available, but the results have been unsatisfactory. For instance, Non-patent Literature (NPL) document 1 describes a deep learning model to recognize the mixed speech directly by assigning the senone labels of the talkers according to the energy of the speech. To deal with the speaker switch problem, NPL document 1 describes using a two-talker joint-decoder with a speaker switching penalty was used to trace speakers. Further, NPL document 2 describes a technique called deep clustering (DPCL) to separate the speech streams by mapping a speaker's time- frequency bins into an embedding space where the bins belong to the same speakers are close and that of different speakers are far away from each other. Moreover, NPL document 3 describes a technique called deep attractor network (DANet) which learns a high-dimensional embedding of the acoustic signals and clustered embeddings with attractor points. [3] Despite the progresses made in monaural multi-talker speech recognition, the word error rates (WER) reported in previous works are still much higher than that in single-talker cases. In single-talker speech recognition, speaker adaptation reduces the mismatch between the training and the test speakers and improves the WER for the test speakers.

[4] NPL Documents:

NPL document 1 : Chao Weng, Dong Yu, Michael L. Seltzer, and Jasha Droppo, "Deep neural networks for single-channel multi-talker speech recognition,” TASLP, vol. 23, no. 10, pp. 1670-1679, 2015.

NPL document 2: J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation,” in ICASSP, 2016, pp. 31-35.

NPL document 3: Zhuo Chen, Yi Luo, and Nima Mesgarani,“Deep attractor network for single-microphone speaker separation,” in ICASSP, 2017, pp. 246-250.

SUMMARY

[5] Provided are methods of performing Teacher-Student (TS) permutation invariant training (PIT) framework for multi-talker speech recognition, and apparatuses for performing the same.

[6] One or more embodiments of the present disclosure discloses novel teach-student techniques which transfer knowledge from a single-talker ASR model to a multi-talker ASR model in the PIT framework. For instance, according to one or more embodiments, the inputs to the teacher and student networks are the single- talker clean speech and the multi-talker mixed speech, respectively, and the knowledge is transferred to the student through the soft labels estimated by the teacher. Furthermore, according to one or more embodiments, a newly designed model training pipeline may be constructed, which includes several features such as using the ensemble of multiple teachers, and applying a progressive training scheme to further improve the system. Moreover, according to one or more embodiments, a new framework can be utilized to perform data augmentation and domain adaptation for multi-talker speech recognition using only un-transcribed data, which will further improve the system performance.

[7] According to an aspect of an embodiment, there is provided a method of performing speech recognition training, the method comprising: acquiring a multi- talker mixed speech signal from a plurality of speakers; performing permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and updating a multi- talker speech recognition model based on a result of the PIT model training.

[8] The single-talker speech recognition model maybe a teacher model, and the multi-talker speech recognition model maybe a student model.

[9] The PIT model training may use labels from the single-talker speech recognition model, the labels being posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model.

[10] The method may further comprise: performing PIT model training on a single talker feature corresponding to one or more of the plurality of speakers; and transferring posteriors from the performing the PIT model training on the single talker feature as soft label input for the multi-talker speech recognition model.

[1 1] The performing PIT model training may comprise: performing a bidirectional long-short term memory (BLSTM) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from inputting a single- talker data corresponding to one or more of the plurality of speakers into the single- talker speech recognition model and generating a plurality of estimated output segments for multi-talker mixed speech signal; and minimizing a minimal average cross entropy (CE) for utterances of all possible assignments between the plurality of estimated output segments and soft labels.

[12] The minimal average cross entropy (CE) maybe determined based on the following equations:

where is posterior of stream s estimated by the multi-talker speech recognition model, is a reference distribution, and is posterior estimated by the single-talker speech recognition model using original single-talker speech stream

[13] The PIT model training may comprise: performing a bidirectional long- short term memory recurrent neural networks (BLSTM-RNN) operation on the multi- talker mixed speech signal by assigning soft labels that are posteriors from the single-talker speech recognition model.

[14] The soft label may be generated using an ensemble of a plurality of different single-talker speech recognition models such that

. .

wherein are interpolation weights and is the posterior estimated by the different plurality of single-talker speech recognition models.

[15] The soft label is generated using an ensemble of a plurality of different single-talker speech recognition models one by one, in ascending order of recognition performance on corresponding single-talker tasks.

[16] According to an aspect of another embodiment, there is provided a speech recognition training apparatus comprising: a memory; and a processor configured to: acquire a multi-talker mixed speech signal from a plurality of speakers; perform permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and update a multi-talker speech recognition model based on a result of the PIT model training.

[17] The single-talker speech recognition model maybe a teacher model, and the multi-talker speech recognition model maybe a student model.

[18] The PIT model training may use labels from the single-talker speech recognition model, the labels being posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model. [19] The processor maybe further configured to: perform PIT model training on a single talker feature corresponding to one or more of the plurality of speakers; and transfer posteriors from the performing the PIT model training on the single talker feature as soft label input for the multi-talker speech recognition model.

[20] The processor maybe further configured to: perform a bidirectional long-short term memory (BLSTM) operation on the multi-talker mixed speech signal by assigning soft labels that are posteriors from inputting a single-talker data corresponding to one or more of the plurality of speakers into the single-talker speech recognition model and generating a plurality of estimated output segments for multi-talker mixed speech signal; and minimize a minimal average cross entropy (CE) for utterances of all possible assignments between the plurality of estimated output segments and soft labels.

[21] The minimal average cross entropy (CE) maybe determined based on the following equations:

where is posterior of stream s estimated by the multi-talker speech recognition model, is a reference distribution, and is posterior estimated by the single-talker speech recognition model using original single-talker speech stream [22] The PIT model training may comprise: performing a bidirectional long- short term memory recurrent neural networks (BLSTM-RNN) operation on the multi- talker mixed speech signal by assigning soft labels that are posteriors from the single-talker speech recognition model.

[23] The soft label maybe generated using an ensemble of a plurality of different single-talker speech recognition models such that wherein are interpolation weights and is the posterior estimated by the different plurality of single-talker speech recognition models.

[24] The soft label may be generated using an ensemble of a plurality of different single-talker speech recognition models one by one, in ascending order of recognition performance on corresponding single-talker tasks.

[25] According to an aspect of another embodiment, there is provided a non-transitory computer readable medium having stored thereon a program for a method of performing speech recognition, the method comprising: acquiring a multi- talker mixed speech signal from a plurality of speakers; performing permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model; and updating a multi- talker speech recognition model based on a result of the PIT model training. BRIEF DESCRIPTION OF THE DRAWINGS

[26] These and/or other aspects will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:

[27] FIG. 1A is a block diagram illustrating a PIT architecture for speech recognition training.

[28] FIG. 1 B is flowchart illustrating operations of a PIT architecture for speech recognition training.

[29] FIG. 2 is a block diagram illustrating a TS permutation invariant training architecture according to an embodiment.

[30] FIG. 3 is an apparatus implementing the TS permutation invariant training architecture in FIG. 2 according to an embodiment.

[31] FIG. 4 is flowchart illustrating operations of the TS permutation invariant training architecture in FIG. 2 according to an embodiment.

DETAILED DESCRIPTION

[32] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.

[33] Hereinafter, TS PIT processing methods and apparatuses according to embodiments will be described in detail with reference to the accompanying drawings. The term such as “comprise” or“include” used herein should not be construed as necessarily including all of the elements or operations (or steps) described herein, and should be construed as not including some of the described elements or operations (or steps) or as further including additional elements or operations (or steps).

[34] Although terms such as“first” and“second” may be used herein to describe various elements or components, the elements or components should not be limited by the terms. These terms are only used to distinguish one element or component from another element or component.

[35] One or more embodiments of the present disclosure may address the multi-talker speech recognition, which recognize the multi-talker mixed overlap speech, in the scenario, such as cocktail party environment or other group environment.

[36] Permutation invariant training (PIT) is a technique for solving the label ambiguity problem in deep learning based multi-talker speech recognition. The basic architecture of PIT for multi-talker automatic speech recognition (PIT-ASR) is depicted in FIG. 1A. In this model, the mixed speech O is sent to the deep learning model to estimate the state-level posterior for each talker. For better ability of modeling long-range dependency, which can improve speaker tracing, recurrent neural networks (RNNs) are usually used. In FIGA. 1A and 1 B, bidirectional long- short term memory RNNs (BLSTM-RNNs) are applied in all models.

[37] According to PIT technique illustrated in FIG. 1A, layer 1 represents an input feature Y for the model. In layer 2, the input features are fed into a recurrent neural network (RNN) in layer 2. The RNN layer 2 may further include bidirectional long-short term memory RNNs (BLSTM-RNNs). Next, in linear layer 3, the output of the RNN layer 2 may go through linearl and Iinear2 for each speaker individually.

For instance, a first output of the RNN layer 2 corresponding to a first speaker may be input through a linearl and a second output of the RNN layer 2 corresponding to a second speaker may be input through a Iinear2. Thereafter, the outputs from the linear layer 3, i.e., linearl and Iinear2, are fed into softmax1 and softmax 2, respectively, of the softmax layer 4, which computes a softmax activation. The softmax activation may be referred to as the senone posteriori in speech recognition tasks. The output of the softmax activation in softmaxl and softmax2 may be considered as predictions1 and predictions2 in prediction layer 5.

[38] Next, in input layer 7, label inputs for the input features Y are input as labels1 and Iabels2. In pairwise scoring layer 6, pairwise scoring is performed, in which the cross entropy loss is computed between any pair of (prediction[i], label[j]), where i and j may be 1 , ... , n. . For instance, in illustrative FIG. 1A, the cross entropy loss may be computed between (predictions1 , labelsl ), (predictions1 , Iabels2), (prediction2, labels1 ) and (prediction2, Iabels2). In assignment layer 8, the cross entropy losses (error) are accumulated together for every possible assignment of labels to each prediction, which is the permutations of labels’ order. For instance, in assignment , the cross entropy losses (error) are accumulated together for every possible assignment of labels to predictions1 and in assignment2, the cross entropy losses (error) are accumulated together for every possible assignment of labels to prediction2. In layer 9, assignment, among assignment and assignment2, with the minimum cross entropy loss is selected as the final error. Accordingly, the model is updated based on the determined final error. Also, the prediction result from the updated model may be used for decoding an input speech data in decoding layer 10.

[39] According to an embodiment, one or more of the layers may be implemented by one or more units or modules. Moreover, each or the one or more units or modules may be implemented by a hardware processor. [40] FIG. 1 B illustrates a method of performing PIT model training for speech recognition training. In operation S1 10, input feature Y for the model for is received. The input feature may be input feature maps. In operation S120, the input features are fed into a recurrent neural network (RNN) and RNN operations are performed. The RNN operations may further include BLSTM-RNN operations. Next, in operation S130, the output of the RNN operations fed through a linear layer for each speaker individually. Thereafter, in operation S140, softmax activations are separately computed on the outputs from the linear layer.

[41] In operation S150, pairwise scoring is performed, in which, cross entropy loss is computed between any pair of (prediction[i], label[j]), label inputs for the input features Y are input as labelsl and Iabels2 and the output of the softmax activation in softmax1 and softmax2 may be considered as predictions1 and predictions2 . For instance, in illustrative FIG. 1A, the cross entropy loss may be computed between (predictions1 , labels1 ), (predictions1 , Iabels2), (prediction2, labelsl ) and (prediction2, Iabels2).

[42] In operation S160. the cross entropy losses (error) are accumulated together for every possible assignment of labels to each prediction, which is the permutations of labels’ orders. For instance, in assignmentl , the cross entropy losses (error) are accumulated together for every possible assignment of labels to predictionsl and in assignment2, the cross entropy losses (error) are accumulated together for every possible assignment of labels to prediction2.

[43] In operation S170, assignment, among assignment1 and assignment2, with the minimum cross entropy loss is selected as the final error, and in operation S180, the model is updated based on the determined final error. [44] The key ingredient in PIT is its training criterion and the objective function in PIT is defined as

where s is a permutation of [ 1 , 2, S], I ^s be the ground truth alignment, y ^s be estimated state-posterior of stream s, lt ^s' s is the ground truth label of stream s' _s at frame t. PIT aims to minimize the minimal average cross entropy (CE) of the whole utterance among all possible assignments of the reference to the estimated posterior.

[45] With this criterion, the network can automatically estimate the assignment. The CE is computed over the whole utterance so that all frames that belong to the same speaker are forced to be aligned with the same output segment (or branch). Moreover, compared to Deep clustering (DPLC) or DANet, the PIT structure in FIG. 1A is much simpler since it allows direct multi-talker mixed speech recognition without explicit separation. After the PIT model training, the individual output posterior stream can be used for decoding as normal to obtain the final recognition result.

[46] In some speech recognition systems, a TS training may use the posterior probability generated by the teacher as the supervision, instead of using only hard labels in traditional machine learning tasks. The objective function in the TS training is defined as

where is the posterior generated by the student model, is the reference distribution and is represented by a Kronecker delta function is the ground truth label, which is usually referred to as hard label. is the posterior probability estimated by the teacher model, which is also referred to as soft label because it is not a one-hot vector. A is the interpolation weight, which is a hyper parameter.

[47] In the related art TS training, knowledge is transferred from a large and complicated teacher network to a small student network to help reduce model footprint. The student tries to mimic the teacher by using the soft labels estimated by the teacher. The soft label is referred to as dark knowledge and is considered more important than hard labels for deep learning.

[48] Although the TS training in PIT framework has been used for single- channel multi-talker speech recognition, the accuracy of speech recognition is still unsatisfactory and is far from the accuracy of the normal single-talker ASR scenario. To further improve the system performance, so it can be enabled to address the multi-talker mixed speech in real scenario, a new architecture based PIT with a refined training pipeline is constructed.

[49] One or more embodiments of the present disclosure may overcome the deficiencies of the conventional and related art techniques, by providing new techniques and technologies that improve system performance. According to one or more embodiments,

[50] FIG. 2 is a block diagram illustrating the teacher-student (TS) permutation invariant training (PIT) architecture 200 according to an embodiment. The TS PIT architecture 200 may include a PIT feature 212 and a teacher model features 213a and 213b. According to another embodiment, the TS PIT may not include all of the PIT feature 212 and the teacher model features 213a and 213b.

[51] According to an embodiment, the PIT feature 212 and the teacher model features 213a and 213b may integrate the knowledge distillation withTS learning into the PIT framework. For instance, the teacher model features 213a and 213 transfer knowledge from a single-talker speech recognition model (the teacher) to a multi-talker PIT feature 212 (the student) to improve the recognition accuracy of the PIT feature 212.

[52] According to an embodiment, parallel data of original individual single- talker speech and the corresponding multi-talker mixed speech, are used by the teacher model features 213a and 213 and the multi-talker PIT feature 212. As illustrated in FIG. 2, the inputs to the teacher model are original single-talker speech streams and the inputs to the student model are the corresponding multi-talker mixed speech The training criterion in this architecture is

where is posterior of stream s estimated by the multi-talker speech recognition model, is a reference distribution, and is posterior estimated by the single-talker speech recognition model using original single-talker speech stream [53] According to the embodiment in FIG. 2, the TS PIT technique minimizes the minimal average CE among all possible assignment between model outputs and soft labels, as compared to the PIT illustrated in FIGS. 1A and 1 B.

[54] According to another embodiment the soft labels may be generated by using an ensemble of teachers, instead of using a single teacher. For instance, an ensemble of teachers may be generated such that wherein are interpolation weights and is the posterior estimated by the different plurality of single-talker speech recognition models.

[55] According to yet another embodiment, the soft label is generated using an ensemble of a plurality of different single-talker speech recognition models one by one, in ascending order of recognition performance on corresponding single-talker tasks. For instance, according to an embodiment, the soft label may be generated using Algorithm 1 as follows:

[56] According to an embodiment, the TS training may be further extended by exploiting large amount of unlabeled data. In the related art TS training, the target distribution is a weighted sum of the hard and soft labels. However, according to an embodiment, when unlabeled data are used, only the posterior estimated by the teacher model is used as the supervision, i.e.,

[57] Moreover, parallel data used as inputs to the model may be generated by varying the relative energy of the involved talkers without transcribing the source streams.

[58] According to this approach, fast domain adaptation with only untranscribed target domain speech may be conducted according to an embodiment. More specifically, a general PIT-based multi-talker ASR model is first trained and then in-domain speech waveform without transcription is collected and used to adapt a general model to generate the domain-dependent model.

[59] FIG. 3 shows an apparatus 300 implementing the TS PIT architecture 200 according to an embodiment. The apparatus 300 may include a processor 310, model storage 314, memory 315 and input/output unit 316. According to an embodiment, the processor 310 may be further configured to implement a decoder 31 1 , a student model training unit 312 and a teacher model training unit 313. According to another embodiment, the processor 310 may not implement all of the decoder 31 1 , the student model training unit 312 and the teacher model training unit 313. Instead, the apparatus 300 may include some of the decoder 31 1 , the student model training unit 312 and the teacher model training unit 313. [60] According to an embodiment, the model storage 316 may be configured to store an acoustic model, bidirectional long-short term memory, recurrent neural networks, convolutional neural network, and other models and networks associated with PIT.

[61] According to an embodiment, the processor 310 may be configured to receive a stream of speech data through the I/O unit 316. According to an embodiment, the stream of speech data may include speech data from multiple speakers speaking at the same time and may further include only a single channel of mixed speech data. Also, the speech data may have structures along both time and frequency axes.

[62] According to an embodiment, the student model training unit 312 may be configured to perform the PIT feature 212 illustrated in FIG. 2, the teachings model training unit 313 may be configured to perform the teacher model features 213a and 213b illustrated in FIG. 2 and the decoder 31 1 may decode speech data based on the PIT model training.

[63] FIG. 4 is flowchart illustrating operations of the TS PIT architecture of FIG. 2 according to an embodiment.

[64] At S410, a processor may receive a stream of speech data. According to an embodiment, the stream of speech data may include speech data from multiple speakers speaking at the same time and may further include only a single channel of mixed speech data. Also, the speech data may have structures along both time and frequency axes.

[65] At S420, the processor may perform PIT model training on a single talker feature corresponding to one or more of the plurality of speakers and update a single talker speech recognition model. According to an embodiment, the single talker speech recognition model is a teacher model.

[66] At S430, the processor may transfer posteriors from the single talker speech recognition model as soft label to be input for the multi-talker speech recognition model. According to an embodiment, the multi-talker speech recognition model is a student model.

[67] At S440, the processor may perform PIT model training on the multi- talker mixed speech signal based on the soft label input from the single talker speech recognition model and generate a plurality of estimated output segments.

[68] At S450, the processor may minimize a minimal average cross entropy (CE) for utterances of all possible assignments between the plurality of estimated output segments and soft labels.

[69] At S460, the processor may update a multi-talker speech recognition model based on minimizing the minimal average cross entropy (CE).

[70] According to one or more embodiments, the TS learning is integrated into the PIT framework to form a new architecture for single-channel multi-talker speech recognition, which further improves the system performance.

[71] According to one or more embodiments, the PIT model training uses a PIT model with 6 BLSTM layers. As such, the model is smaller and more compact that a progressive model with totally 10 BLSTM layers (6+4=10 BLSTM layers) in conventional system.

[72] According to one or more embodiments, different structure including CNN, BLSTM and the ensemble of CNN and BLSTM may be implemented as the teacher model, which further improves the system performance. [73] According to one or more embodiments, in the ensemble teacher, the teacher models may be sorted in ascending order of the performance on single- talker task, which further improves the system performance.

[74] According to one or more embodiments, an unsupervised learning scheme to do data augmentation for multi-talker speech recognition may be performed. Moreover, the unsupervised learning scheme can also be utilized to fast adapt the PIT model into a new domain without transcribed data.

[75] The units described herein may be implemented using hardware components and software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, non-transitory computer memory and processing devices. A processing device may be implemented using one or more general-purpose or special purpose computers, such as, for example, a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor, hardware circuitry or any other device capable of responding to and executing instructions in a defined manner. The processing device also may access, store, manipulate, process, and create data in response to execution of the software.

[76] The software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct or configure the processing device to operate as desired. The method according to the above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations of the above-described example embodiments. Examples of non- transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM discs, DVDs, and/or Blue-ray discs; magneto-optical media such as optical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.), and the like.

[77] Experimental Results

[78] Some results comparisons are illustrated as the following tables:

[79] PIT+TS vs. PIT according to an embodiment.

[80] Ensemble vs. non-ensemble

[81] Unsupervised learning to do data augmentation

[82] Unsupervised learning for fast domain adaptation

[83] The present disclosure has been described above with reference to embodiments. However, those of ordinary skill in the art will understand that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims. Therefore, the above embodiments should be considered in descriptive sense only and not for purposes of limitation. Thus, the scope of the present disclosure may be defined not by the above detailed descriptions but by the appended claims, and all differences within the scope will be construed as being included in the present disclosure. [84] It should be understood that embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in other embodiments.

[85] While one or more embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.