CROSS REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of and claims priority to U.S. Provisional Patent Application No. 62/971,786, filed on February 7, 2020, the entire contents of which are hereby incorporated by reference.

BACKGROUND

This specification relates to determining a machine learning algorithm to perform a machine learning task.

The machine learning algorithm can be, for example, a trained neural network that has been trained to perform the task.

Neural networks that are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an output machine learning algorithm to perform a particular machine learning task.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. Instead of constructing a neural network by combining sophisticated hand-designed components (e.g., convolutions, batch-norm, and dropout) as previous neural architecture search approaches do, the techniques described in this specification allows a system to automatically search for whole machine learning algorithms (e.g., from scratch) with little restriction on the form of the algorithms and using only basic mathematical operations as building blocks. Therefore, the system requires much less human-design, saves human research time and allows for the discovery of non-neural network algorithms (because the system does not assume the existence of a neural network or of gradients when defining the search space) in comparison to conventional techniques for automating machine learning research, which have largely focused on the architecture of neural networks and relied on using sophisticated expert-designed layers as building blocks or similarly constrained search spaces that are heavily reliant on human design. Further, by moving away from expert-designed search spaces, the described techniques can reduce human bias and can eventually lead to creative new machine learning concepts. Additionally, the use of the subject matter described herein to construct machine learning algorithms can lead to an improved performance of a machine learning model on a particular machine learning task. For example, where the machine learning task is an image/audio classification task, the accuracy and/or efficiency of the classification may be improved.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example machine learning algorithm search system.

FIG. 2 is a flow diagram of an example process for searching for one or more candidate machine learning algorithms.

FIG. 3 is a flow diagram of an example process for searching for an output machine learning algorithm to perform a machine learning task.

FIG. 4 shows mutation examples for modifying a parent candidate machine learning algorithm to generate one or more child candidate machine learning algorithms.

FIG. 5 illustrates an example of an output machine learning algorithm.

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an output machine learning algorithm to perform a particular machine learning task (or a set of machine learning tasks).

The machine learning algorithm defines a model architecture for a machine learning model (a neural network) for performing the task, hyperparameters for the training of the model to perform the task, and preprocessing techniques (e.g., data augmentation policies) applied to inputs during, after, or both the training.

The machine learning model can be configured to perform any kind of machine learning tasks, i.e., can be configured to receive any kind of digital data input and to generate any kind of score, classification, or regression output based on the input.

In some cases, the machine learning model is a neural network that is configured to perform an image processing task, i.e., receive an input image and to process the input image to generate a network output for the input image. For example, the task may be image classification and the output generated by the neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category. As another example, the task can be image embedding generation and the output generated by the neural network can be a numeric embedding of the input image. As yet another example, the task can be object detection and the output generated by the neural network can identify locations in the input image at which particular types of objects are depicted. As yet another example, the task can be image segmentation and the output generated by the neural network can assign each pixel of the input image to a category from a set of categories.

As another example, if the inputs to the neural network are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the task can be to classify the resource or document, i.e., the output generated by the neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

As another example, if the inputs to the neural network are features of an impression context for a particular advertisement, the output generated by the neural network may be a score that represents an estimated likelihood that the particular advertisement will be clicked on.

As another example, if the inputs to the neural network are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item.

As another example, if the input to the neural network is a sequence of text in one language, the output generated by the neural network may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

As another example, the task may be an audio processing task. For example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance. As another example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network can indicate whether a particular word or phrase (“hotword”) was spoken in the utterance. As another example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network can identify the natural language in which the utterance was spoken.

As another example, the task can be a natural language processing or understanding task, e.g., an entailment task, a paraphrase task, a textual similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on, that operates on a sequence of text in some natural language.

As another example, the task can be a text to speech task, where the input is text in a natural language or features of text in a natural language and the network output is a spectrogram or other data defining audio of the text being spoken in the natural language.

As another example, the task can be a health prediction task, where the input is electronic health record data for a patient and the output is a prediction that is relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient.

As another example, the task can be an agent control task, where the input is an observation characterizing the state of an environment and the output defines an action to be performed by the agent in response to the observation. The agent can be, e.g., a real- world or simulated robot, a control system for an industrial facility, or a control system that controls a different kind of agent. FIG. 1 shows an example machine learning algorithm search system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

To determine an output machine learning algorithm (e.g., output machine learning algorithm 150) to perform a particular machine learning task, the system 100 includes an algorithm search subsystem 120 that receives a training dataset 102. The training dataset 102 includes a set of training examples. Each training example in the set of training examples includes an example input for the particular machine learning task and a corresponding example output for the particular machine learning task.

The algorithm search subsystem 120 further receives a validation dataset 104, which includes a set of validation examples. Each validation example in the set of validation examples includes a validation input for the particular machine learning task and a corresponding validation output for the particular machine learning task. For example, a larger set of training data may have been randomly partitioned to generate the training data 102 and the validation set 104.

The system 100 can receive the training data 102 and the validation set 104 in any of a variety of ways. For example, the system 100 can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system 100, and randomly divide the uploaded data into the training data 102 and the validation set 104.

As another example, the system 100 can receive an input from a user specifying which data that is already maintained by the system 100 should be used for training the trainee neural network, and then divide the specified data into the training data 102 and the validation set 104.

After receiving the training dataset 102 and validation dataset 104, the subsystem 120 generates a sequence of candidate machine learning algorithms (e.g., candidate algorithms 106, 108, 110) to perform the particular machine learning task. Each candidate machine learning algorithm in the sequence can be represented as a computer program with three component functions including (i) a respective candidate setup function, denoted as Setup(), (ii) a respective candidate leam function, denoted as Leam(), and (iii) a respective candidate predict function, denoted as Predict().

The respective candidate setup function initializes one or more training parameters of the candidate machine learning algorithm. The training parameters are the parameters of the operations in the machine learning model that are learned or adjusted through training of the machine learning model. For example, the training parameters may be weights of neural network layers of the machine learning model. Optionally, the candidate setup function initializes hyperparameters (e.g., learning rate) for the training of the machine learning model to perform the particular machine learning task.

The respective candidate leam function adjusts training parameters of the candidate machine learning algorithm. The respective candidate predict function predicts an output for the particular machine learning task for a given input for the particular machine learning task using the adjusted training parameters.

More specifically, the candidate predict function and the candidate learn function are executed alternatively during the training of the machine learning model using the training data 102. The candidate predict function is executed on a batch of multiple training examples and the candidate leam function is executed on the outputs generated by the candidate predict function for the batch of multiple training examples. In particular, for a given training example (x, y) in a batch of training examples of the training data 102, the candidate predict function takes as input the example input x and processes the example input x using the current training parameters to generate a predicted output y’ for the example input x for the particular machine learning task. The candidate learn function takes as input the example output y in the training example (x, y) and the predicted output y’ and adjust the training parameters of the candidate machine learning algorithm based on an error between y and y’. The above process is repeated for the next training examples in the batch (i.e., the candidate predict function takes as input the next example input in the next training example of the batch and processes the next example input using the adjusted training parameters to generate the next predicted output and so on).

The sequence of candidate machine learning algorithms includes at least a subset of algorithms in which two or more of the setup function, predict function and leam function are different from one another.

In some implementations, the subsystem 120 may initialize the sequence of candidate ML algorithms as an empty sequence, i.e., none of three component functions has any instructions or code lines.

In some other implementations, the subsystem 120 may initialize the sequence of candidate ML algorithms by initializing each of the candidate setup function, the candidate predict function, and the candidate predict function of the first candidate machine learning algorithm in the sequence of candidate machine learning algorithms with one or more instructions (e.g., random instructions).

To generate a sequence of candidate machine learning algorithms, the subsystem 120 searches for one or more candidate machine learning algorithms through a machine learning algorithm search space.

The candidate machine learning algorithms can be represented as computer programs which are sequences of instructions that act on a virtual memory.

The machine learning algorithm search space is defined by a sequence of instructions including (i) instructions that specify the operations performed by a setup function of a given candidate machine learning algorithm, (ii) instructions that specify the operations performed by a learn function of the given candidate machine learning algorithm, and (iii) instructions that specify the operations performed by a predict function of the given candidate machine learning algorithm.

Each instruction includes an operation (also referred to as an op) from a predetermined set of operations. For example, the set of operations may include one or more of arithmetic operations (e.g., “multiply a scalar with a vector”), trigonometry operations, pre-calculus operations, linear algebra operations, or probability and statistic operations. In some cases, an operation in the set of operations may require real-valued constants (e.g., m and s for a random Gaussian sampling operation), which are searched for as well. To avoid biasing the choice of operations, a criterion may be applied to the predetermined set of operations to enforce that none of the operations in the set exceed a threshold level of complexity.

In addition, each instruction includes a set of arguments. For example, these arguments can be addresses in the virtual memory (e.g., “read the inputs from scalar address 0 and vector address 3; write the output to vector address 2”).

In some implementations, the subsystem 120 searches for one or more candidate machine learning algorithms through the machine learning algorithm search space by performing a regularized evolution search process. In particular, at each iteration of the process, the subsystem 120 selects a parent candidate machine learning algorithm from the current sequence of candidate machine learning algorithms. The subsystem 120 modifies the parent candidate machine learning algorithm to generate one or more child candidate machine learning algorithms (also referred to as “child algorithms” for simplicity) by using mutations. The mutations that produce a child algorithm from a parent algorithm is tailored to the search space. For example, the subsystem 120 may use a random choice among the following types of mutations: (i) insert a random instruction at a random location in a component function, (ii) remove an instruction at a random location in a component function, (iii) randomize all the instructions in a component function, or (iv) modify one of the arguments of an instruction in a component function by replacing it with a random choice (e.g. “swap the output address” or “change the value of a constant”). Examples of mutations are further described in detail below with reference to FIG. 4.

In some other implementations, instead of using regularized evolution search, the subsystem 120 searches for the one or more candidate machine learning (ML) algorithms through the machine learning algorithm search space by performing a random search process, in which the one or more candidate machine learning algorithms are selected randomly from the search space according to a particular distribution.

After searching for the one or more candidate ML algorithms, the subsystem 120 adds the one or more candidate ML algorithms to the current sequence of candidate ML algorithms.

In some implementations, the subsystem 120 may remove an existing candidate machine learning algorithm from the current sequence of candidate machine learning algorithms after one or more new candidate ML algorithms are added to the sequence.

For example, the subsystem 120 may remove the oldest candidate machine learning algorithm in the current sequence of candidate machine learning algorithms.

The subsystem 120 can repeat the search process multiple times until a criterion is satisfied, e.g., until the number of candidate ML algorithms in the sequence has reached a threshold number or until a predetermined level of accuracy of the candidate ML algorithms in the sequence is obtained.

The process for searching for one or more candidate machine learning algorithms to generate the sequence of candidate ML algorithms is described in more detail below with references to FIG. 2.

For each of the candidate ML algorithms in the sequence, the subsystem 120 sets up one or more training parameters for the candidate machine learning algorithm by executing the candidate setup function associated with the candidate ML algorithm. The subsystem 120 trains the candidate machine learning algorithm on the training data 102 to adjust the one or more training parameters by processing the set of training examples using the candidate predict function and the candidate learn function associated with the candidate ML algorithm. The subsystem 120 may train the candidate machine learning algorithm until one or more criteria are satisfied (e.g., until convergence, or until a predetermined level of performance is obtained).

The subsystem 120 evaluates a performance of the trained candidate machine learning algorithm by executing the candidate predict function on the set of validation examples to determine a performance metric for the trained candidate machine learning algorithm. For example, as shown in FIG. 1, the subsystem 120 determines performance metrics 112, 114, and 116 for candidate ML algorithm 106, 108 and 110, respectively.

The performance metric can be, for example, a validation loss, a training loss, a weighted combination of the validation loss and the training loss, or any metric that is appropriate for the particular machine learning task.

In some implementations, the subsystem 120 may evaluate a performance of a trained candidate machine learning algorithm by executing a candidate predict function of the trained candidate ML algorithm on validation examples of a single validation dataset (e.g., the validation dataset 104).

In some other implementations, the subsystem 120 may evaluate a performance of a trained candidate machine learning algorithm by executing a candidate predict function of the trained candidate ML algorithm on validation examples of multiple validation datasets. This would allow for better generalization of the output machine learning algorithm to new datasets (or new machine learning tasks).

After evaluating the algorithms, in some implementations, in some implementations, the subsystem 120 selects a trained candidate machine learning algorithm with the best performance metric among the trained candidate machine learning algorithms as the output machine learning algorithm 150 for the particular machine learning task. In some other implementations, the subsystem 120 selects a set of multiple trained candidate machine learning algorithms having the highest performance metrics, trains each candidate machine learning algorithm in the set longer, and then selects the algorithm with the best performance metric among the algorithms in the set as the output machine learning algorithm 150 for the particular machine learning task. An example of an output machine learning algorithm that could be generated by the above process is shown in FIG. 5.

In some implementations, to attain higher search speeds, the subsystem 120 may execute the search process in parallel by using a parallel budget of workers and using a central server. Each worker runs regularized evolution on its own population (i.e., its own sequence of candidate ML algorithms). The workers can exchange algorithms through migration. That is, the workers can periodically upload randomly selected candidate algorithms to the central server. The central server replies with candidate algorithms randomly sampled across all workers and replaces a portion (e.g., half, one- third, a quarter, or any other portion) of the local population (i.e. random migration).

For example, for each predetermined number of evaluations (e.g., for every 100- 10000 evaluations) of performance of candidate machine learning algorithms, each worker uploads a number of candidate algorithms to the central server to replace a corresponding portion of the local population. For example, the worker can upload - candidate algorithms which is half of a population of n algorithms. The central server replies with - algorithms sampled randomly across all workers and replaces half of the worker’s population.

In some implementations, to further speed up the search process (e.g., 4 times, 8 times or 10 times speedup), the subsystem 120 may use a functional equivalence checking (FEC) technique to detect equivalent candidate ML algorithms. This technique is useful because the search space is not heavy designed and thus it allows for mutations that do not have an effect on the accuracy (e.g. adding an instruction that writes to an address that is never read). When these mutations occur, the child algorithm behaves identically to its parent. The FEC technique prevents these identically functioning algorithms from being repeatedly evaluated (i.e. trained and validated in full many times), therefore saving the time and computational resource that would otherwise be required to evaluate these equivalent algorithms.

Generally, the subsystem 120 uses the FEC technique as follows. The subsystem 120 performs a novelty check that confirms whether the child candidate machine learning algorithm has a different behavior compared to the current sequence of candidate machine learning algorithms. In response to a determination that the novelty check is yes (which means that the child candidate machine learning algorithm has a different behavior compared to the current sequence of candidate machine learning algorithms), the system adds the child candidate machine learning algorithm to the current sequence of candidate machine learning algorithms. In response to a determination that the novelty check is no, the subsystem 120 skips the evaluation of the child candidate machine learning algorithm.

More particularly, the subsystem 120 maintains a cache that maps evaluated algorithm fingerprints to their accuracies. Before evaluating a candidate algorithm, the subsystem 120 fingerprints it and consults the cache to see if it has already been evaluated. If it has, the subsystem 120 reuses the stored accuracy instead of computing it again. This way, the subsystem 120 can keep the different implementations of the same candidate algorithm for the sake of diversity: even though they produce the same accuracy now, they may behave differently upon further mutation.

For example, to fingerprint a candidate algorithm, the subsystem 120 trains it for 10 steps and validates it on 10 validation examples. The 20 resulting predictions are then truncated and hashed to produce an integer fingerprint. The cache may hold a large number of fingerprint-accuracy pairs (e.g., 100,000 fingerprint-accuracy pairs).

Other techniques may be used to additionally improve the quality of the search. For example, some workers can be allowed to search on training data for additional machine learning tasks, in addition to the training data for particular machine learning task, to promote diversity.

In some implementations, after the output machine learning algorithm 150 is determined, the system 100 deploys the machine learning model defined by the output ML algorithm 150 and then uses the model to process requests received from users, e.g., through the API provided by the system. In other words, the system uses the machine learning model defined by the output ML algorithm 150 to generate new network outputs for new network inputs.

Instead of or in addition to using the machine learning model defined by the output ML algorithm 150, the system 100 can provide data specifying the output machine learning algorithm 150 to a user who submitted a request to find a machine learning model to perform a particular ML task, e.g., through the API.

FIG. 2 is a flow diagram of an example process 200 for searching for one or more candidate machine learning algorithms. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a machine learning algorithm search system, e.g., the machine learning algorithm search system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

While the process 200 describes searching for the candidate machine learning algorithm using evolution search, more generally, the system can search for candidate machine learning algorithms using any appropriate technique, e.g., random search, or searching using reinforcement learning or Bayesian optimization.

The system selects a parent candidate machine learning algorithm (also referred to as “parent algorithm”) from the current sequence of candidate machine learning algorithms (step 202). The parent candidate machine learning algorithm includes a parent setup function, a parent predict function, and a parent learn function.

In some implementations, to select a parent algorithm, the system randomly chooses multiple candidate machine learning algorithms from the current sequence of candidate machine learning algorithms.

In these implementations, the system selects a chosen candidate machine learning algorithm with the highest performance metric among the plurality of candidate machine learning algorithms as the parent candidate machine learning algorithm.

In some other implementations, to select a parent algorithm, the system selects a plurality of highest performing candidate machine learning algorithms from the current sequence of candidate machine learning algorithms. For example, the system selects the two highest performing algorithms or selecting ten currently highest performing algorithms and then randomly selecting two of them. The system then randomly chooses one of these selected highest performing algorithms as the parent algorithm.

The system modifies the parent candidate machine learning algorithm to generate one or more child candidate machine learning algorithms (step 204). Each child candidate machine learning algorithm (also referred to as “child algorithm”) includes a child setup function, a child predict function, and a child learn function.

To generate the one or more child algorithms from the parent algorithm, the system performs the following steps for one or more times.

For at least one of the parent setup function, the parent predict function, or the parent leam function of the parent algorithm, the system performs one or more mutations from a set of mutations. The set of mutations includes, for example, at least one of (i) inserting a random instruction (e.g., an instruction that has a random operation and/or a set of random arguments) into a random position in the current component function, (ii) removing an instruction at a random location in the current component function, (iii) randomizing all the instructions in the current component function, or (iv) modifying a random argument of another random instruction in the current component function. The system creates a child algorithm that has a child setup function, a child predict function, and a child learn function, which were created by modifying at least one of the parent setup function, the parent predict function, or the parent leam function, respectively, as described above.

The system adds the one or more child candidate machine learning algorithms to the current sequence of candidate machine learning algorithms (step 206). Before adding a child algorithm to the current sequence of candidate machine learning algorithms, the system performs a novelty check that confirms whether each child candidate machine learning algorithm has a different behavior compared to the current sequence of candidate machine learning algorithms. In response to a determination that the novelty check is yes (which means that the child candidate machine learning algorithm has a different behavior compared to the current sequence of candidate machine learning algorithms), the system adds the child candidate machine learning algorithm to the current sequence of candidate machine learning algorithms. In response to a determination that the novelty check is no, the system skips the evaluation of the child candidate machine learning algorithm and does not add the child algorithm to the current sequence. The novelty check is performed by using a functional equivalence checking technique.

In some implementations, the system evaluates each child candidate ML algorithm before adding the child algorithm to the current sequence of candidate machine learning algorithms. In some implementations, the system does not evaluate child algorithms before adding them to the current sequence of candidate machine learning algorithms. Instead, the system evaluates them when the system selects them from the sequence in order to determine a parent algorithm.

To evaluate a candidate machine learning algorithm, the system sets up one or more training parameters for the candidate machine learning algorithm by executing the setup function of the candidate ML algorithm, trains the candidate machine learning algorithm to adjust the one or more training parameters by processing the set of training examples using the candidate predict function and the candidate leam function of the candidate ML algorithm, and determines a performance metric for the trained candidate machine learning algorithm by executing the candidate predict function of the candidate ML algorithm on the set of validation examples.

In some implementations, the system may remove an existing candidate machine learning algorithm from the current sequence of candidate machine learning algorithms after one or more child candidate ML algorithms are added to the sequence. For example, the system may remove the oldest candidate machine learning algorithm in the current sequence of candidate machine learning algorithms or the worst performing candidate algorithm in the current sequence.

The system can repeat the above search process multiple times until a criterion is satisfied, e.g., until the number of candidate ML algorithms in the sequence has reached a threshold number or until a predetermined level of accuracy of the candidate ML algorithms in the sequence is obtained.

FIG. 3 is a flow diagram of an example process 300 for searching for an output machine learning algorithm to perform a machine learning task. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a machine learning algorithm search system, e.g., the machine learning algorithm search system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

The system receives a set of training examples (step 302). Each training example in the set of training examples includes an example input and an example output.

The system receives a set of validation examples (step 304). Each validation example in the set of validation examples comprising a validation input and a validation output.

The system generates a sequence of candidate machine learning algorithms to perform the particular machine learning task (step 306). Each candidate machine learning algorithm in the sequence includes a respective candidate setup function that initializes one or more training parameters of the candidate machine learning algorithm, a respective candidate learn function that adjusts training parameters of the candidate machine learning algorithm, and a respective candidate predict function that predicts an output for a given input using the adjusted training parameters. The process for generating a sequence of candidate machine learning algorithms is described in more detail above with reference to FIG. 1 and FIG. 2.

For each of the candidate ML algorithms in the sequence, the system performs steps 308-312 as follows.

The system sets up one or more training parameters for the candidate machine learning algorithm by executing the candidate setup function associated with the candidate ML algorithm (step 308).

The system trains the candidate machine learning algorithm to adjust the one or more training parameters by processing the set of training examples using the candidate predict function and the candidate learn function associated with the candidate ML algorithm (310).

The system evaluates a performance of the trained candidate machine learning algorithm by executing the candidate predict function on the set of validation examples to determine a performance metric for the trained candidate machine learning algorithm (step 312).

The system then selects a trained candidate machine learning algorithm with the best performance metric among the trained candidate machine learning algorithms as the output machine learning algorithm for the particular machine learning task (step 314). An example of an output machine learning algorithm that could be generated by the above process is shown in FIG. 5.

FIG. 4 shows mutation examples for modifying a parent candidate machine learning algorithm (also referred to as “parent algorithm”) to generate one or more child candidate machine learning algorithms (also referred to as “child algorithms”). In FIG. 4, parent algorithms are on the left and child algorithms are on the right. In this example, the parent and child algorithms are represented as computer programs that act on a small virtual memory with separate address spaces for scalar, vector and matrix variables (e.g. si, vl, ml), all of which are floating-point and share the dimensionality of the input features (F) of the input for the particular machine learning task. Programs are sequences of instructions. Each instruction has an operation - or op - that determines its function (e.g. “multiply a scalar with a vector”). Table SI includes a list of possible operations including the operations shown in the parent and child algorithms of FIG. 4.

In Type (i) mutation example, a child algorithm is created by inserting a random instruction into the corresponding parent algorithm. In Type (ii) mutation example, a child algorithm is created by replacing one or more instructions of the corresponding parent algorithm with a set of randomized instructions. In Type (iii) mutation example, a child algorithm is created by modifying an argument of an instruction of the corresponding parent algorithm.

FIG. 5 shows an example of an output machine learning algorithm 500. In this example, the particular machine learning task is an image classification and the output generated by a machine learning model defined by the output machine learning algorithm for a given image is scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.

In particular, instruction 502 in the Setup() function in FIG. 5 initializes a hyperparameter (e.g., learning rate) for the training of the machine learning model to perform the image classification task.

The Predict() function, which includes instructions 504, receives input features vO of an example input x in a training example (x,y) and processes the input features vO using the current training parameters of the output algorithm 500 to generate a predicted output si for the input features vO for the image classification task.

The Learn() function, which includes instructions 506, takes as input the example output y (i.e., label sO) in the training example (x, y) and the predicted output si. The Leam() function adjusts the training parameters of the output algorithm 500 based on an error s3 between sO and si.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.