BACKGROUND

[0001] Neural networks are frequently employed in the field of machine learning to approximate unknown functions that can depend on a large number of inputs. Typically, a neural network (or simply "network") comprises a number of interconnected units that send messages to one another. Each network unit has associated weight values that can be tuned based on labeled training data, which is input data for which corresponding outputs

("labels") are known. Thereby, the network may be adapted (or "trained") to produce a suitable output when subsequently presented with unlabeled input data.

[0002] In some cases, training of a neural network based on labeled training data may result in a network that is so closely tuned to the training data that it does not produce meaningful results on unlabeled data that is different, but similar to, the labeled training data. This can be due to the network "overfitting" its tunable parameters to apparent characteristics of the training data that are, in actuality, sampling noise. Thereby, by overfitting to the training data the network may not have applicability to subsequent unlabeled data.

SUMMARY

[0003] According to some aspects, a method of classifying speech recognition results using a neural network is provided, the neural network comprising a plurality of

interconnected network units, each network unit having one or more weight values, the method comprising using at least one computer, performing acts of providing a first vector as input to a first network layer comprising one or more network units of the neural network, transforming, by a first network unit of the one or more network units, the input vector to produce a plurality of values, the transformation being based at least in part on a plurality of weight values of the first network unit, sorting the plurality of values to produce a sorted plurality of values; and providing the sorted plurality of values as input to a second network layer of the neural network.

[0004] According to some aspects, at least one computer readable storage medium is provided storing instructions that, when executed by at least one processor, perform a method of classifying speech recognition results using a neural network comprising a plurality of interconnected network units, each network unit having one or more weight values, the method comprising providing a first vector as input to a first network layer comprising one or more network units of the neural network, transforming, by a first network unit of the one or more network units, the input vector to produce a plurality of values, the transformation being based at least in part on a plurality of weight values of the first network unit, sorting the plurality of values to produce a sorted plurality of values, and providing the sorted plurality of values as input to a second network layer of the neural network.

BRIEF DESCRIPTION OF DRAWINGS

[0005] Various aspects and embodiments will be described with reference to the following figures. It should be appreciated that the figures are not necessarily drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing.

[0006] FIG. 1A depicts a conventional network unit;

[0007] FIG. IB depicts a maxout network unit;

[0008] FIG. 2 depicts an illustrative sortout network unit, according to some embodiments;

[0009] FIG. 3 is a flowchart depicting a method of using a sortout network unit to produce sorted output values, according to some embodiments;

[0010] FIG. 4A depicts an illustrative neural network including a layer of sortout units, according to some embodiments; [0011] FIGs. 4B-4C depict illustrative portions of the neural network shown in FIG. 4 A, according to some embodiments; and

[0012] FIG. 5 illustrates an example of a computing system environment on which aspects of the invention may be implemented.

DETAILED DESCRIPTION

[0013] As discussed above, some neural networks when trained may overfit to the labeled training data. Neural networks with multiple and often numerous layers (sometimes called "deep" neural networks) typically contain many network units and therefore have a large number of tunable parameters, which makes them able to learn very complex relationships between their inputs and outputs. However, these same characteristics can also lead to overfitting on labeled training data, particularly if the size of the training data set is limited. Due to the large number of tunable parameters, many of the parameters may not be able to converge on values having general applicability to unlabeled data unless the labeled training data set is very large. Accordingly, when a deep network trained on labeled training data is applied to unlabeled data or test data, the network's performance may deviate from the desired performance. This may be the case even if the training data and the test data are drawn from the same distribution. Test data refers generally to input data provided to a trained neural network for which the correct output is not known, for example, input data provided to a neural network deployed in a system to perform the operations the neural network was trained to perform.

[0014] Some techniques to address overfitting attempt to perform averaging over a number of possible network configurations, thus averaging out effects like transient sampling noise in a set of training data. One such technique utilizes a network unit that outputs the maximum value from a set of values produced within the unit. This so-called "maxout" unit is depicted in FIG. IB. For comparison, FIG. 1A illustrates a conventional network unit that receives an input vector x and transforms the input vector using a weight vector w to produce an output value a. The transformation may be a linear transformation, such as an inner product (e.g., a = xiwi + x ₂ w ₂ + ... + x _n w _n ). The value a may have a non- linear function f() applied to it, such as a logistic function, thereby producing f(a) as an output from the network unit. During training of a neural network, the values of the weight vector w are tuned by passing labeled training data (e.g., training data for which the desired output is known) through the network and comparing the network' s behavior to the intended behavior. By providing the neural network with sufficient training data as input and modifying the neural network based on the comparison, the neural network can be trained to produce the desired output from the corresponding input.

[0015] The so-called "maxout" unit shown in FIG. IB includes multiple weight vectors that are each applied to an input vector x to produce a number of respective output values ai, a ₂ and a ₃ . The maxout unit then takes the maximum of these values and outputs the maximum value from the network unit.

[0016] The inventors have recognized and appreciated that, by selecting only the maximum value within a maxout unit, information about other models (the models each being represented by a weight vector) is thereby lost. For instance, in a maxout unit, only the weight vector that produces the largest output value has any effect on the output of the network unit. In the example of FIG. IB, weight vectors wi and w ₂ do not contribute to the output since a ₃ , which was produced using w _3i corresponds to the maximum value. The inventors have recognized that subsequent units in the network may be able to utilize information about other models, even though these models may not have produced the highest value output when applied to an input vector.

[0017] Based on this insight, the inventors have developed a network unit configured to evaluate multiple models and generate output indicative of the performance of these models. According to some embodiments, a sorted list of output values from a network unit is produced, wherein each output value represents the performance of a respective model upon an input to the unit. Subsequent units in the network may then utilize values produced by each model. The significance of each value may be effectively encoded in its position in the sorted list of values. For example, the maximum value may always appear first in the list of sorted values. Information about the relative significance of each output value may thereby be implicitly provided to subsequent network units. For instance, weight vectors in one of the network units that perform sorting (which may be termed "sortout units") may each represent a model that is applied to the unit's input (e.g., applied through a linear combination of weight values w and input values x). Values produced from each weight vector' s application to the input vector are sorted and output from the sortout unit as a sorted list of values.

[0018] In some embodiments, a neural network may comprise a layer of "sortout" network units. Each of the sortout units may output a sorted list of values produced by operating on an input vector (e.g., by applying a plurality of weight vectors). The resulting plurality of sorted lists may then be provided to network units in a subsequent layer of the network. In some implementations, a single output vector (or other suitable ordered list of values) in which the plurality of sorted lists are combined in such a way to preserve the sorted order of each list may be supplied to a second network layer. For instance, the sorted lists may be concatenated together to produce a single ordered list. The single ordered list provides an output vector of the layer of sortout units, which may in turn be provided as the input vector to a subsequent layer in the network. For example, each unit of the subsequent network layer may receive an input vector that includes the combination of sorted lists output from the layer of sortout units. As noted above, this vector may provide information about the ordering produced by each of the preceding layer's sortout units via the position of each value in the output vector, which implicitly conveys the value's significance. In other implementations, values output by each sortout unit in a layer of sortout units may be assembled in an output vector (or other suitable ordered list of values) and then sorted as a whole before being provided to a second network layer.

[0019] In some embodiments, a subset of a sorted list output from a sortout unit may be provided to one or more network units in a subsequent layer of a neural network. The subset may include any suitable subset that may be generated from the ordered values, such as, but not limited to, the N largest values (where N is an integer value), the N smallest values, a median value or N values adjacent to each other in the ordered list that include the median. [0020] FIG. 2 depicts an illustrative sortout network unit, according to some embodiments. Exemplary sortout unit 200 includes three weight vectors wi, w ₂ and w ₃ that are each applied to input vector x, thereby producing values ai, a ₂ and a ₃ . The sortout unit sorts values ai, a ₂ and a ₃ in either ascending or descending order to produce output values from the sortout unit. In the example of FIG. 2, the sorted order produced is a ₃ , ai then a ₂ . For instance, a ₃ may be the largest value and a ₂ the smallest value; alternatively, a ₂ may be the largest value and a ₃ the smallest value, depending on whether the sorting is performed in descending or ascending order, respectively. In either case, the values output from the application of each weight vector to an input vector are ordered in ascending or descending order of value.

[0021] According to some embodiments, the weight vectors w of sortout unit 200 transform an input vector x using a linear transfer function to produce an output value a. In some implementations, the linear transfer function may be a linear combination of the weight values and corresponding input vector values (e.g., an inner product). For instance, x may have values {xi, x ₂ , ..., x _n }, w may have values {wi, w ₂ , ..., w _n } and a may be equal to xiwi + x ₂ w ₂ + ... + x _n w _n . In some cases, the linear combination may include a bias term that is added to the combination of the weight values and input values.

[0022] According to some embodiments, sortout unit 200 may apply a non-linear function to values ai, a ₂ and a ₃ , which may be performed before or after the values are sorted. Suitable non-linear functions may include, but are not limited to, a sigmoid function such as the logistic function, a softmax function, a softplus function, a hyperbolic function, or combinations thereof.

[0023] In some embodiments, a neural network including one or more sortout units, such as sortout unit 200, may be applied to perform automated speech recognition (ASR). For instance, an input vector to the neural network may comprise a plurality of coefficients obtained from speech, such as log-mel coefficients and/or mel-frequency cepstral coefficients. The neural network may be configured to produce an output vector having elements that each correspond to a particular phoneme and whose values indicate the likelihood that the input coefficients correspond to the respective phoneme. As such, the output of the neural network may be a hypothesis for the phoneme sequence corresponding to the input vector. The neural network may have been trained, and weights of the network thereby tuned, on labeled training data such that providing the input vector to the network produces the output vector based on the weight values of the network. In some

embodiments, such a neural network may be configured to receive hundreds, thousands or more input values, produce hundreds or thousands or more output values in response, have one million or more tunable parameters (e.g., weight values) and have multiple hidden layers (e.g., two, three, four, five, ten or more hidden layers). In some embodiments, output values from such a neural network may be used to construct a word hypothesis. It should be appreciated that a neural network may be of any size and include any number of hidden layers, as the aspects are not limited in this respect.

[0024] FIG. 3 is a flowchart depicting a method of using a sortout network unit to produce sorted output values, according to some embodiments. The sortout unit of method 300 is a network unit within a neural network that includes a plurality of units (some of which also may be sortout units). In method 300, a sortout unit receives an input vector in act 310. In act 320, the input vector is transformed based on weight values of the sortout unit. In some embodiments, the sortout unit includes a plurality of weight vectors that are each separately applied to the input vector using a selected transfer function, thereby producing a plurality of values. In some implementations, the transfer function may be a linear combination of the weight vector elements and the input vector elements. In some embodiments, the weight vectors of the sortout unit may be configured to have the same number of elements as the input vector.

[0025] In act 330, values output by transforming the input vector using the weight values are sorted, which rearranges the order of the values to reflect their relative magnitudes. In some implementations, a sortout unit may be configured to produce transformed values that may, dependent on the input values and the weight values, be positive or negative, in which case an ordering of such values may either be based on relative magnitude or may instead be based on relative absolute magnitudes. The ordering of the values produced in act 330 may be represented in any suitable way. In some embodiments, the sorted values are stored in a vector wherein the vector elements have an implicit ordering. However, values may in general be stored in any suitable way that indicates their relative order. In act 340, the sorted values are output to another network unit of the neural network.

[0026] FIG. 4A depicts an illustrative neural network including a layer of sortout units, according to some embodiments. To illustrate a neural network utilizing sortout units, neural network 400 is depicted, which includes six network units arranged into three layers. An input value 411 is provided to each of sortout units 421, 422 and 423. Values output from these sortout units are provided to each of network units 431 and 432. Output values from network units 431 and 432 are provided to network unit 441, which produces an output value 451 from the network. Each of network units 421, 422, 423, 431, 432 and 441 comprise a plurality of weight values that may have been tuned during previous training of network 400. For instance, labeled training data that includes input values and their corresponding output values may have been supplied to the network 400 and the network trained to determine suitable values for the weights of each of the network units.

[0027] FIG. 4B depicts a first portion of neural network 400, according to some embodiments. In the example of FIG. 4B, each of the network units 421, 422 and 423 include two weight vectors that are each applied to the input vector x. For instance, sortout unit 421 transforms x using a vector wn to produce a value an, and also transforms x using a vector wi ₂ to produce a value ai ₂ . As discussed above, application of a weight vector to an input vector may, in some embodiments, comprise generating a linear combination of the elements of the two vectors. In the example of FIG. 4B, sortout unit 421 sorts values an and ai ₂ to produce the ordering ai ₂ , an. Similarly, sortout unit 422 sorts values ai ₃ and ai ₄ to produce an ordering ai ₃ , ai ₄ and sortout unit 423 sorts values ais and ai ₆ to produce an ordering ai ₆ , ai ₅ .

[0028] Outputs from the three sortout units 421, 422 and 423 are combined to produce a single vector ai, which combines the outputs of each respective sortout unit whilst retaining the relative ordering determining by each sort operation. Accordingly, subsequent network units receiving ai as input may perform processing that is based on the fact that the first element of ai is greater than the second element of ai (assuming that the sort operation performed by sortout unit 421 was to sort in descending order). That is, the weight values of subsequent network units may have been tuned during training such that the weight values produce desired outputs from the network by responding to the relative sizes of the values in ai.

[0029] For instance, weight vector wn may have been tuned to produce a large output value when a first feature is present in input vector x and weight vector wi ₂ may have been tuned to produce a large output value when a second feature is present in input vector x. In this case, the weight vectors wn and wi ₂ may be viewed as "detectors" for these two features. The first two elements of vector ai will thereby indicate values of detection results of these two "detectors," which allows a subsequent layer of network units receiving ai as input to interpolate over these results. In at least some networks, during training the weight values in the subsequent layer may thereby be tuned to utilize this aspect of their input vector ai, which may reduce the network's tendency to overfit to training data compared with a layer of conventional network units or a layer of maxout units.

[0030] FIG. 4C depicts a second portion of neural network 400, according to some embodiments. The vector ai output from the layer of sortout units is provided as input to each of conventional network units 431 and 432, which each transform the input using a single weight vector w ₂ i and w ₂₂ , respectively, to produce outputs a ₂ i and a ₂₂ , respectively. These output values are combined to produce a vector a ₂ , which is provided as input to network unit 441. Network unit 441 transforms a ₂ using weight vector w ₃ i to produce an output value y, which is output from the network.

[0031] It should be appreciated that the neural networks illustrated herein are merely exemplary to demonstrate how sorting techniques may be utilized. Any type of neural network of any size, configuration and connectivity may be used, as techniques regarding sorting output values discussed herein are not limited for use with any particular network and can be applied to networks of any type. Moreover, the neural networks illustrated herein may be applied to any suitable machine learning task or tasks, including but not limited to automated speech recognition (ASR) or other types of pattern recognition. [0032] An illustrative implementation of a computer system 500 that may be used to implement a neural network including one or more sortout units as described herein is shown in FIG. 5. The computer system 500 may include one or more processors 510 and one or more non-transitory computer-readable storage media or storage devices (e.g., memory 520 and one or more non-volatile storage media 530). The processor 510 may control writing data to and reading data from the memory 520 and the non- volatile storage device 530 in any suitable manner, as the aspects of the invention described herein are not limited in this respect. To perform the functionality and/or techniques described herein, the processor 510 may execute one or more instructions stored in one or more computer- readable storage media (e.g., the memory 520, storage media, etc.), which may serve as non- transitory computer-readable storage media storing instructions for execution by the processor 510.

[0033] In connection with techniques for training and/or operating a neural network including one or more sortout units as described herein, code used to, for example, store weight values, perform training algorithms, store labeled and/or unlabeled data, produce output from the network using the network's units, etc. may be stored on one or more computer-readable storage media of computer system 500. Processor 510 may execute any such code to provide any techniques for training and/or operating a neural network as described herein. Any other software, programs or instructions described herein may also be stored and executed by computer system 500. It will be appreciated that computer code may be applied to any aspects of methods and techniques described herein. For example, computer code may be applied to determine weight values of a plurality of network units in a neural network based on labeled training data and/or may be applied to determine an output of a neural network based on previously determined weight values.

[0034] Various inventive concepts may be embodied as at least one non-transitory computer readable storage medium (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, etc.) or a computer readable storage device encoded with one or more programs that, when executed on one or more computers or other processors, implement some of the various embodiments of the present invention. The non-transitory computer-readable medium or media may be transportable, such that the program or programs stored thereon may be loaded onto any computer resource to implement various aspects of the present invention as discussed above.

[0035] The terms "program," "software," and/or "application" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of one or more embodiments described herein need not reside on a single computer or processor, but may be distributed in a modular fashion among different computers or processors to implement various aspects of the present invention.

[0036] While particular techniques for storing and arranging numbers within a computer system have been discussed herein, it will be appreciated that the techniques described herein are not limited to any particular technique(s) for storing data within a computer system. For instance, examples described herein utilize vectors to store and order a set of values. However, the techniques described herein are not limited to the use of vectors, as any technique or techniques for storing values and an ordering of those values may be utilized. This non-limiting description applies to any of: output of a sortout unit, weight values, input values, etc.

[0037] Techniques utilized herein may be utilized with any type of neural network, such as, but not limited to any feed forward network (e.g., a multi-layer perceptron network), a radial basis function (RBF) network, autoencoders, denoising autoencoders, recurrent neural networks (RNNS), (Bi-directional) long- short-term memory networks (LSTMS), neural turing machines (NTMs), or combinations thereof. Any number of network units, including any number of hidden layers, inputs and outputs may be utilized in such a neural network. In addition, the neural network may be trained (and weight values thereby determined) using any suitable algorithm(s), which may include a fixed dropout scheme and/or an annealed dropout scheme. [0038] Having thus described several aspects of at least one embodiment of this invention, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0039] Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the invention. Further, though advantages of the present invention are indicated, it should be appreciated that not every embodiment of the technology described herein will include every described advantage. Some embodiments may not implement any features described as advantageous herein and in some instances one or more of the described features may be implemented to achieve further embodiments. Accordingly, the foregoing description and drawings are by way of example only.

[0040] Also, various inventive concepts may be embodied as one or more methods, of which examples have been provided. The acts performed as part of a method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0041] Use of ordinal terms such as "first," "second," "third," etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.

[0042] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having," "containing," "involving," and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.