BACKGROUND

[0001 ] This specification relates to reinforcement learning.

[0002] In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.

[0004] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

[0005] This specification describes a reinforcement learning system implemented as computer programs on one or more computers in one or more locations that selects actions to be performed by an agent interacting with an environment. Generally, the system uses an action selection policy neural network in selecting actions to be performed in response to observations characterizing states of the environment. So that the action selection policy neural network can be effectively used to select actions, the system trains the action selection policy neural network on trajectories generated as a result of interactions of the agent with the environment. In particular, the system can sample trajectories from a replay memory and then adjust the current values of the parameters of the neural network by training the neural network on the sampled transitions using an off-policy actor critic reinforcement learning technique.

[0006] Thus in one aspect a method, in particular of reinforcement learning, comprises maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment. Each trajectory may comprise respective action selection data at each of a plurality of time steps. The action selection data at each time step may identify one or more of: an observation characterizing a state of the environment, an action performed by the agent in response to the observation, a reward received in response to the agent performing the action, an action selection score assigned to at least the performed action in determining which action to perform in response to the observation. The method may then involve training an action selection neural network having action selection neural network parameters, here referred to as policy parameters, on the trajectories in the replay memory. The action selection neural network is configured to: receive an observation characterizing a state of the environment; and process the observation to generate a network output that defines a score distribution over possible actions that can be performed by the agent in response to the observation. Training the action selection neural network may comprise sampling a trajectory from the replay memory; and adjusting current values of the policy parameters by training the action selection neural network on the trajectory, preferably using an off-pol icy actor critic reinforcement learning technique.

[0007] The set of actions may be finite, defining a discrete action space or a continuous action space which has been discretized, or may be actions in a continuous action space. Where the action space is discrete or discretized the score distribution may be a categorical distribution defined by a probabil ity vector having elements for the possible actions. Where the action space is continuous the score distribution may be defined by one or more parameters of one or more continuous distributions, for example a mean and covariance of a distribution such as a Gaussian distribution.

[0008] In broad terms an actor critic technique employs an actor neural network to provide action scores and a critic neural network to provide one or more values for a state or possible action which may be used by the actor neural network. Off-policy learning uses stored experience for learning a policy. Some implementations of off-policy actor critic reinforcement learning techniques are described later.

[0009] Some advantages of implementations of the method include faster and more accurate learning.

[0010] Where the set of possible actions that can be performed by the agent in response to the observation is finite the network output may comprise the score distribution. Then, adjusting current values of the pol icy parameters using the off-policy actor critic reinforcement learning technique may comprise, for time steps in the trajectory from the final time step in the trajectory to the initial time step in the trajectory: processing the observation using the action selection neural network to determine a main score distribution; processing the observation using a critic neural network to determine a respective Q value for each of the possible actions; and determining, from the Q values and the main score distribution, a main gradient for the time step. A Q-value may provide a value for a observation-action combination, more particularly an estimate of a future reward were the action to be performed in response to the observation. The main gradient can be used to adjust the values of the policy parameters by gradient ascent or descent, for example by backpropagation through the action selection neural network. The main gradient may be used on its own or, in some examples, in combination with one or more additional gradient terms as described later.

[001 1 ] In some implementations determining the main gradient may comprise determining, from the Q values and the main score distribution, a value estimate for each time step in the trajectory. Determining the main gradient may also comprise determining a retrace estimate for the time step. Broadly the retrace estimate comprises an estimate of a target Q value for a time step, in particular an estimate of an optimal Q value that should have been generated by the critic network by processing the observation at the time step. Thus the retrace estimate may comprise a weighted sum of future rewards. The rewards may be weighted to correct for off-policy sampling using the agent's behavior as represented by the stored trajectories. The weighting may implement a form of importance sampling using a ratio of likelihoods of an action determined from the action selection neural network and from the stored trajectory. Thus, for example, the importance sampling may employ an importance weight obtained by dividing the score for action performed at the time step in the main score distribution by the action selection score in the action selection distribution. The value of an importance weight may be capped, e.g., to less than or equal to unity, to determine a truncated importance weight. This limits potentially large importance weight values, reducing importance weight variance.

[0012] In some implementations the retrace estimate for a time step may be determined from a current retrace estimate and the reward, for example by scaling the current retrace estimate by a discount factor equal to or less than unity and increasing the result dependent upon the reward. The main gradient may then be determined using a combination of the main score distribution, the Q values, the value estimate, and the retrace estimate, so that each of these is taken into account.

[0013] The method may further comprise determining a critic gradient from the Q value for the performed action and the retrace estimate. The method may then determine an update for current values of the parameters of the critic neural network from the critic gradient. The retrace estimate may then be updated.

[0014] One or more possible actions that can be performed by the agent in response to the observation may be defined in a continuous action space. The network output may then comprise one or more distribution parameters that define the score distribution over points in the continuous action space. One way of adjusting values of the policy parameters using the off-policy actor critic reinforcement learning technique may then comprise, for time steps in the trajectory from the final time step in the trajectory to the initial time step in the trajectory, processing the observation using the action selection neural network to define a main score distribution. The observation may also be processed using a stochastic dueling neural network to determine both a Q-value for an action in continuous action space and a value estimate for the observation. Thus at a time step the value estimate may be determined for the time step and a Q-value may be determined for the performed action. The Q-value may be stochastic, that is determined by sampling from a distribution, for example a probability distribution over possible actions. This distribution may be defined by the output of the action selection policy neural network. Thus the stochastic dueling neural network may comprise a neural network that is configured to process the observation to generate a deterministic value estimate and a stochastic Q value for the performed action. The stochastic dueling neural network may perform the role of the critic.

[001 5] The method may then further comprise sampling an alternate action from the main score distribution. A Q-value for the alternate action may then be determined. The alternate action optionally may also be used to determine an alternate importance weight as previously described; the alternate importance weight may be truncated.

[0016] A main gradient for the time step may be determined from the Q-value, the deterministic value estimate, the stochastic Q value, the main score distribution, and the sampled action. More particularly the performed action may be used to determine a first gradient term of the main gradient from the stochastic Q value and the alternate action may be used to determine a second gradient term of the main gradient from the Q-value for the alternate action. The first gradient term may be weighted by an importance weight for the performed action; the second gradient term may be weighted by an importance weight for the alternate action. As previously, the main gradient may be supplemented by one or more additional gradient terms as described later. The main gradient for the time step, for updating the action selection neural network by adjusting the policy parameters, may then be determined.

[0017] Adjusting current values of the policy parameters may further comprise processing the observation to generate an auxiliary score distribution. This processing may use an average neural network having a plurality of average network parameters and may be in accordance with current values of the average network parameters. As described later, the average neural network may have parameter values which represent a moving average of the policy parameters. The method may thus further comprise determining an auxiliary gradient of a measure, such as a Kullback-Leibler divergence, of a difference between the auxiliary score distribution and the main score distribution. The method may then determine a final gradient from the main gradient and the auxiliary gradient, and then update the current values of the policy parameters from the final gradient. This can help to stabilize learning, in particular by constraining policy parameter updates not to deviate too far from values representing a running average of previous policies.

[0018] A scaling factor may be applied to the main gradient to determine the final gradient, that is to reduce the gradient or rate of change of the policy parameters. The scaling factor may be appl ied conditional upon the auxiliary gradient being greater than a constraint.

[0019] In another aspect a method of training a main neural network having a plurality of main network parameters to determine trained values of the main network parameters may comprise optimizing an objective function. Thus the method comprising may comprise receiving a batch of training data comprising a plurality of training examples. Then, for each of the plural ity of training examples in the batch, the method may include: processing, using the main neural network in accordance with current values of the main network parameters, the training example to generate a main network output for the training example; determining a main gradient of the objective function from the main network output; processing, using an average neural network having a plurality of average network parameters and in accordance with current values of the average network parameters, the training example to generate an average network output for the training example; determining an auxiliary gradient of a measure of a difference between the average network output and the main network output; determining a final gradient from the main gradient and the auxiliary gradient; and determining an update to the current values of the main network parameters from the final gradient.

[0020] Corresponding features to those previously described may also be employed in the context of this method.

[0021 ] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The subject matter described in this specification introduces an off-policy actor-critic reinforcement learning technique that is stable, sample efficient, and can be applied learn large scale reinforcement learning tasks. In particular, the described techniques can match or exceed the state of the art in both discrete action space tasks and continuous control tasks while remaining stable and sample efficient during learning. I n particular, by incorporating truncated importance sampling with bias correction, stochastic dueling network architectures, and efficient trust region policy optimization as described below, the described techniques can effectively train a neural network to select actions to be performed by an agent from a continuous action space. Effectively training a neural network to select actions from a continuous action space can be highly advantageous for complex tasks, e.g., tasks that include controlling a robot or an autonomous vehicle . By maintaining an average network during the training and using the average network to scale gradients before they are applied to the policy parameters, the stability of the training process is improved, resulting in reduced training times and improved performance once the training is completed. More generally, the described average network can be used to improve the stability of the training of any of a variety of different kinds of neural networks, resulting in reduced training times and improved performance once the training is completed.

[0022] Applications of the described methods and systems include reinforcement learning tasks where the agent is a robot agent interacting with the environment to accomplish a task or an autonomous or semi-autonomous vehicle navigating through the environment.

Alternatively the agent may control actions in an environment comprising items of equipment, for example in a data center or grid mains power or water distribution system, or in a manufacturing plant. Then the observations may comprise observations of power or water usage by equipment, or observations of power generation or distribution control, or observations of usage of a resource or of waste production. The agent may control actions in the environment to increase efficiency and/or reduce the environmental impact of operations in the environment.

[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 shows an example reinforcement learning system.

[0025] FIG. 2 is a flow diagram of an example process for training the action selection policy neural network when the set of possible actions to be performed by the agent is finite. [0026] FIG. 3 is a flow diagram of an example process for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is finite.

[0027] FIG. 4 is a flow diagram of an example process for training the action selection policy neural network when the set of possible actions to be performed by the agent is not finite.

[0028] FIG. 5 is a flow diagram of an example process for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is not finite.

[0029] FIG. 6 is a flow diagram of an example process for training a main neural network using an average neural network.

[0030] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[003 1 ] FIG. 1 shows an example reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0032] The reinforcement learning system 100 selects actions 104 to be performed by an agent 102 interacting with an environment 106 at each of multiple time steps. In order for the agent 102 to interact with the environment 106, the system 100 receives an observation 108 characterizing the current state of the environment 106 and selects an action 104 to be performed by the agent 102 in response to the received observation 108.

[0033] In some implementations, the environment 106 is a simulated environment and the agent 102 is implemented as one or more computer programs interacting with the simulated environment by navigating through the simulated environment. For example, the simulated environment may be a video game and the agent 102 may be a simulated user play ing the video game by navigating through the simulated environment. As another example, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent 102 is a simulated vehicle navigating through the motion simulation environment. In these cases, the actions 104 may be control inputs to control the simulated user or simulated vehicle and the observations can include images of the simulated environment, features characterizing the state of the simulated user or simulated vehicle, or both. [0034] In some other implementations, the environment 106 is a real-world environment and the agent 102 is a mechanical agent interacting with the real-world environment. For example, the agent 102 may be an autonomous or semi-autonomous vehicle navigating through the environment. In these cases, the actions 104 may be control inputs to control the robot or the autonomous vehicle and the observations can include images captured by sensors of the mechanical agent, other sensor data captured by other sensors of the mechanical agent.

[0035] At each time step during the interaction of the agent 102 with the environment 106, the system 100 uses an action selection policy neural network 1 12 to select the action to be performed by the agent 102 in response to the observation 108 at the time step.

[0036] In particular, the action selection policy neural network 1 12 is configured to receive an input including an observation 108 and to process the input in accordance with a set of parameters of the action selection policy neural network, referred to in this specification as policy parameters, to generate a network output that the system 100 uses to determine an action 104 to be performed by the agent 102 at the time step. The network output defines a score distribution over possible actions that can be performed by the agent to interact with the environment in response to the observation.

[0037] In some cases, the set of possible actions that can be performed by the agent is finite, i.e., the system selects from a pre-determined finite, discrete set of actions at each time step. In these cases, i.e., when the set of possible actions that can be performed is finite, the network output is a score distribution over the set of possible actions.

[0038] In other cases, the set of possible actions is not finite, e.g., actions are represented by points in a continuous action space. For example, the actions may be torques applied to various joints of a mechanical agent that can be any real value in some predetermined range. In cases where the space of possible actions is continuous, the network output can be parameters, e.g., mean and covariance parameters, of a distribution over the possible actions.

[0039] In order for the action selection policy neural network 1 12 to be effectively used to select actions to be performed by the agent, a training engine 1 18 trains the action selection policy neural network 1 12 to generate policy outputs that maximize the expected cumulative reward received by the system 100, e.g. a long-term time-discounted sum of rewards received by the system 100, as a result of interactions by the agent with the environment.

[0040] In particular, during the training, the system 100 stores trajectories generated as a result of interactions of the agent with the environment in a replay memory 1 50. Each trajectory in the replay memory 150 includes respective action selection data at each of multiple time steps. For example, each trajectory in the replay memory 150 may include the same fixed number of time steps. The trajectory data at a given time step identifies: (i) an observation characterizing a state of the environment, (ii) an action performed by the agent in response to the observation, (iii) a reward received in response to the agent performing the action, and (iv) the score distribution for the actions in the set of actions that was used in determining which action to perform in response to the observation. This distribution will be referred to in this specification as an action score distribution.

[0041 ] In particular, the system 100 uses the action selection policy neural network 1 12 to select the actions that are performed by the agent to generate the trajectories that are stored in the replay memory 150. However, because the values of the policy parameters change during the training, it may not be possible for the system 100 to re-generate the same score distribution that was previously generated by the action selection policy neural network 1 12 for a given observation at the time at which the system 100 samples the trajectory from the replay memory 150. In order for the system 100 to track the scores that were generated for previously performed actions, the system 100 stores the action score distribution (or parameters defining the action score distribution) in the replay memory 150.

[0042] To train the action selection policy neural network 1 12, the training engine 1 18 samples trajectories from the replay memory 150 and trains the action selection policy neural network 1 12 on the sampled trajectories using an off-policy actor-critic reinforcement learning technique. As part of the actor-critic technique, the training engine 1 18 trains the action selection policy neural network 1 12 jointly with a critic neural network 130.

[0043] Generally, the critic neural network 1 30 is a neural network having a set of parameters, referred to in this specification as critic parameters. When the set of actions is finite, the critic neural network is configured to receive an observation and to process the observation to generate an output that defines a respective Q value for each action in the set of actions. The Q value for a given action is an estimate, according to the current values of the critic parameters, of the cumulative reward that would be received by the system 100 if the agent performed the given action in response to the observation.

[0044] When the set of actions is finite, the critic neural network 130 and the action selection policy neural network 1 12 can share some parameters. For example, the critic neural network 130 and the action selection policy neural network 1 12 can be a single neural network with two heads, one that outputs the Q values for the actions and another that outputs the score distribution. This neural network may include, for example, a convolutional neural network and/or a recurrent neural network. [0045] Training the action selection policy neural network 1 12 (and the critic neural network 130) when the set of actions is finite is described in more detail below with reference to FIGS. 2 and 3.

[0046] When the set of actions is not finite, for example in a continuous action space, the critic neural network 130 can be a stochastic dueling neural network. A stochastic dueling neural network is a neural network that generates deterministic value estimates for states characterized by input observations while also generating stochastic Q values. In particular, the stochastic dueling neural network is configured to receive an input observation, an action of interest, and n actions sampled from a probability distribution over the set of possible actions. For example, the n actions can be sampled from a probability distribution defined by the output of the action selection policy neural network 1 12.

[0047] The stochastic dueling neural network is configured to process the observation to generate the deterministic value estimate for the state characterized by the observation. The value estimate is an estimate of the cumulative reward received starting from the environment being in the state characterized by the observation if actions to be performed by the agent are selected using the action selection policy neural network 1 12.

[0048] The stochastic dueling neural network is also configured to process the observation, the action of interest, and the n actions sampled from the probability distribution to generate a respective advantage estimate for each of the actions. An advantage estimate for a given action is a relative measure of importance of the action, i.e., how much cumulative reward will be received if the action is preformed relative to how much cumulative reward will be received if other actions are performed.

[0049] The stochastic dueling neural network then combines the value of the state, and the advantage estimates to generate the stochastic Q value for the action of interest. The Q value is stochastic because during different processing iterations, the n actions that are sampled may be different, resulting in different Q values being generated for the same action of interest. In particular, the Q value for the action of interest at in response to an observation xi satisfies:

Qe _' ^x t> ^a t) = V (x _t ) + A (x _t , a _t ) - ±∑? _{= 1} A (,x _t , u _t ),

where V{x _t ) is the value estimate for the state characterized by the observation , A (x _t , a _t ) is the advantage estimate for the action of interest, and A x _t , u ) is the advantage estimate for the i ^' -th sampled action. [0050] Training the action selection policy neural network 1 12 (and the critic neural network 130) when the set of actions is not finite, i.e., the space of possible actions is continuous, is described in more detail below with reference to FIGS. 4 and 5.

[0051 ] In some implementations, to assist in the training of the action selection policy neural network 1 12, the system 100 maintains an average neural network 170. The average neural network 170 is a network that generates the same kind of output and has the same architecture as the action selection policy neural network 1 12, but has different parameter values from those of the action selection policy neural network 1 12 during the training of the action selection pol icy neural network 1 12. In particular, during the training, the system 100 updates the values of the parameters of the average network so that they represent a moving average of the pol icy parameters. The parameters of the average neural network will be referred to in this specification as "average network parameters." Using the average network during training will be described below with reference to FIGS. 2-6.

[0052] FIG. 2 is a flow diagram of an example process 200 for training the action selection policy neural network when the set of possible actions to be performed by the agent is finite. For convenience, the process 200 will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 200.

[0053] The system samples a trajectory from the replay memory (step 202). As described above, the trajectory includes respective action selection data at each of a plurality of time steps.

[0054] For each time step in the trajectory starting from the last time step, the system processes the observation at the ti me step using the action selection policy neural network in accordance with current values of the policy parameters to generate a main score distribution for the time step and processes the observation at the time step using the critic neural network to generate a set of Q values for the actions for the time step (step 204).

[0055] If the system is using an average neural network for the training, the system also processes the observation at the time step using the average neural network in accordance with current values of the average network parameters to generate an auxiliary score distribution for the time step.

[0056] The system determines a respective action selection parameter update and a respective critic parameter update for each of the time steps using the main score distributions for the time steps, the sets of Q values for the time steps and, if generated, the auxiliary score

I I distributions for the time steps (step 206). Determining these updates for a given time step is described in more detail below with reference to FIG. 3.

[0057] The system updates the current values of the policy parameters and the current values of the critic parameters using the action selection parameter updates and the critic parameter updates for the time steps (208), i.e., by adding each update to the corresponding set of parameters.

[0058] If the system is using an average neural network, the system updates the current values of the average network parameters using the updated values of the policy parameters to determine updated values of the average network parameters. In particular, the updated values of the average network parameters may be a weighted sum of the current values of the average network parameters and the updated values of the updated values of the policy parameters. A weight for combining these values can be selected to control the influence of the average neural network on the update of the policy parameters and hence provide a "soft" update.

[0059] The system can repeatedly perform the process 200 for different trajectories sampled from the replay memory to train the action selection pol icy neural network, i.e., to determine trained values of the policy parameters from initial values of the policy parameters.

[0060] FIG. 3 is a flow diagram of an example process 300 for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is finite. For convenience, the process 300 will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 300.

[0061 ] The system performs the process 300 for each time step in the trajectory, starting from the last time step in the trajectory and continuing until the first time step in the trajectory.

[0062] The system determines an importance weight for the time step (step 302) from the score for the action performed at the time step in the main score distribution for the time step and the action selection score for the performed action in the action selection distribution for the time step, i.e., the distribution identified in the action selection data for the time step.

[0063] In particular, the system determines the importance weight by dividing the score for action performed at the time step in the main score distribution by the action selection score in the action selection distribution. [0064] The system also determines a truncated importance weight for the time step as the minimum of a positive constant, e.g., one, and the importance weight.

[0065] The system determines a value estimate for the time step (step 304) from the Q values and the main score distribution for the time step. In particular, the system determines a weighted sum of the Q values with each Q value being weighted by the score for the corresponding action in the main score distribution.

[0066] The system determines a Retrace estimate for the time step by multiplying a current Retrace estimate by a discount factor and adding the reward for the time step to the product (step 306). The current Retrace estimate is the Retrace estimate after the preceding iteration of the process 300. For the first iteration of the process, i.e., for the last time step in the trajectory, the system sets the current Retrace estimate to zero. Generally, the Retrace estimate is an estimate of a target Q value for the time step, i.e., an estimate of the Q value that should have been generated by the critic network by processing the observation at the time step.

[0067] The system determines a first gradient for the time step that is based on the performed action at the time step (step 308). In particular, the first gradient term satisfies:

min{c, p _i }V^, _(x . ₎ log/( _i |^ ₀ (x _i ))((? ^ret - where p; is the truncated importance weight for the time step, the gradient is taken with respect to ψρ' ( [ ), which is the output of the action policy network that defines the main score distribution, is the main score distribution score for the time step, Oj is the action performed at the time step, Q ^ret is the Retrace estimate for the time step, and V _t is the value estimate for the time step.

[0068] The system determines a second gradient for the time step that is based on all of the actions in the set of actions (step 3 10). Generally, the second gradient term can be viewed as a correction term for the first gradient term. In particular, the second gradient satisfies: ∑ _Q min{0,l - ^■ ^}f(a^ _e > (x _i ))V _{cPgr (Xi)} ogf(a^ ₉ ' (x _i ))(Q _Qi (x _i , a _i ) - where the sum is a sum over all of the actions a in the finite set of actions, _ί ( ) is the importance weight for the action a, i.e., the ratio between the score for the action a in the main score distribution and the score for the action a in the action score distribution, a; ) is the Q value for the performed action. [0069] The system determines a main gradient for the time step from the first gradient and the second gradient (step 3 12), e.g., by adding the first gradient term and the second gradient term.

[0070] In implementations where the system uses an average network, the system determines an auxiliary gradient of a measure of difference between the auxiliary score distribution for the time step and the main score distribution for the time step (step 3 14). In particular, the auxiliary gradient can be equal to the gradient with respect to the main distribution, i.e., the output of the action policy neural network, of the L divergence between (i) the auxiliary score distribution and (ii) the main score distribution.

[007 1 ] The system determines a final gradient from the main gradient and the auxiliary gradient (step 3 16). In particular, the system determines, from the main gradient and the auxiliary gradient, a scaling factor for the main gradient and then applies the scaling factor to the main gradi l gradient. For example, the scaling factor 5 can satisfy :

where k is the auxiliary gradient, g is the main gradient, and δ is a constant value. The system can then apply the scaling factor by subtracting the scaling factor from the main gradient.

[0072] The system determines an update to the current values of the policy parameters from the final gradient (step 3 18).

[0073] In particular, the system can backpropagate the final gradient through the action selection policy neural network to determine a gradient with respect to the parameters of the action selection policy neural network that is used to update the current values of the policy parameters.

[0074] In implementations where the system does not use an average neural network, the system can use the main gradient as the final gradient or can scale the main gradient using some other technique, e.g., a trust region technique that limits the per-step changes to the parameters. An example of such a technique is the Trust Region Policy Optimization (TRPO) technique described in Schulman, et al, J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization, in ICML, 2015.

[0075] The system determines an update to the current values of the critic parameters that uses the Retrace estimate as the target Q value (step 320). In particular, the update to the critic parameters can satisfy : ν _βί (<? ^,¾ί - <?¾(*-. <¾ ))

where θ _ν ' are the critic parameters.

[0076] The system updates the Retrace estimate before the estimate is used in the next iteration of the process 300 (step 322). For example, the updated Retrace estimate can satisfy:

P. (<? ^m - <?<¾(*.. a<

where p; is the truncated importance weight.

[0077] The system then uses the updated Retrace estimate as the current estimate for the next iteration of the process 300.

[0078] FIG. 4 is a flow diagram of an example process 400 for training the action selection policy neural network when the set of possible actions to be performed by the agent is not finite, i.e., when the actions are selected from a continuous action space. For convenience, the process 400 will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 400.

[0079] The system samples a trajectory from the replay memory (step 402). As described above, the trajectory includes respective action selection data at each of a plurality of time steps.

[0080] For each time step in the trajectory, the system processes the observation at the time step using the action selection policy neural network in accordance with current values of the policy parameters to generate parameters of a main score distribution for the time step (step 404). If the system is using an average neural network for the training, the system also processes the observation at the time step using the average neural network in accordance with current values of the average network parameters to generate parameters of an auxiliary score distribution for the time step.

[008 1 ] For each time step in the trajectory, the system processes the observation at the time step and the performed action at the time step using the critic neural network (which in this case is a stochastic dueling network) to generate a stochastic Q value for the performed action and a value estimate for the time step (step 406).

[0082] The system determines a respective action selection parameter update and a respective critic parameter update for each of the time steps using the main score distributions for the time steps, the sets of Q values for the time steps and, if generated, the auxiliary score distributions for the time steps (step 408). Determining these updates for a given time step is described in more detail below with reference to FIG. 5.

[0083] The system updates the current values of the policy parameters and the current values of the critic parameters using the action selection parameter updates and the critic parameter updates for the time steps (step 410), i.e., by adding each update to the corresponding set of parameters.

[0084] If the system is using an average neural network, the system updates the current values of the average network parameters using the updated values of the policy parameters to determine updated values of the average network parameters. In particular, the updated values of the average network parameters are a weighted sum of the current values of the average network parameters and the updated values of the policy parameters.

[0085] The system can repeatedly perform the process 400 for different trajectories sampled from the replay memory to train the action selection policy neural network, i.e., to determine trained values of the policy parameters from initial values of the policy parameters.

[0086] FIG. 5 is a flow diagram of an example process 500 for determining an action selection parameter update and a critic parameter update for a time step when the set of possible actions to be performed by the agent is not finite. For convenience, the process 500 will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. 1 , appropriately programmed in accordance with this specification, can perform the process 500.

[0087] The system performs the process 500 for each time step in the trajectory, starting from the last time step in the trajectory and continuing until the first time step in the trajectory.

[0088] The system samples an alternate action from the main score distribution (step 502). That is, the system samples an alternate action from the continuous action space such that the likelihood that any given action is sampled is defined by the score for the action in the main score distribution.

[0089] The system determines an importance weight for the time step and an alternate importance weight for the time step (step 504). In particular, the system determines the importance weight by dividing (i) the score for action performed at the time step, i.e., the action identified in the action selection data for the time step, in the main score distribution for the time step by (ii) the action selection score in the action selection distribution for the time step and determines the alternate importance weight by dividing (i) the score for alternate action in the main score distribution for the time step by (ii) the score for the alternate action in the action selection distribution for the time step.

[0090] The system determines a Retrace estimate for the time step by multiplying a current Retrace estimate by a discount factor and adding the reward for the time step to the product (step 506). The current Retrace estimate is the Retrace estimate after the preceding iteration of the process 500.

[0091 ] The system determines an off-policy corrections (OPC) estimate by multiplying a current OPC estimate by the discount factor and adding the reward for the time step to the product (step 508). Like the current Retrace estimate, the current OPC estimate is the OPC estimate after the preceding iteration of the process 500.

[0092] For the first iteration of the process, i.e., for the last time step in the trajectory, the system sets both the current Retrace estimate and the current OPC estimate to zero.

[0093] The system determines a first gradient term for the time step that is based on the performed action at the time step (step 510). In particular, the first gradient term satisfies:

min{ , Pi}V p _e , ( _Xi ) ogf( L (Xi))(Q ^opc - V _t ),

where c is a constant value, p, is the importance weight for the time step, is the main score distribution for the time step, is the performed action at the time step, φ _θ ι (χ ) are the parameters that are output by the action selection policy network (i.e., the parameters that define the main score distribution for the time step), Q ^opc is the OPC estimate for the time step and is the value estimate for the time step.

[0094] The system determines a second gradient term for the time step that is based on the alternate action sampled for the time step (5 12). In particular, the second gradient term satisfies:

min|o, l -^) (<3¾(^, ;) - V _i )V _(Pg , _{Xi) }ogf(a _i ' ^ _e < (x _i )) _l where p- is the alternate importance weight for the time step, and a[) is the Q value for the alternate action · in response the observation at the time step as generated by the stochastic dueling network.

[0095] The system determines a main gradient for the time step from the first gradient and the second gradient (step 5 14), e.g., by adding the first gradient term and the second gradient term.

[0096] In implementations where the system uses an average network, the system determines an auxiliary gradient of a measure of difference between the auxil iary score distribution for the time step and the main score distribution for the time step (step 5 16). In particular, the auxiliary gradient can be equal to the gradient with respect to the parameters of the main distribution of the KL divergence between (i) the auxiliary score distribution and (ii) the main score distribution.

[0097] The system determines a final gradient from the main gradient and the auxiliary gradient (step 518). In particular, the system determines, from the main gradient and the auxiliary gradient, a scaling factor for the main gradient and then applies the scaling factor to the main gradi l gradient. For example, the scaling factor s can satisfy:

where k is the auxi liary gradient, g is the main gradient, and δ is a constant value. The system can then apply the scaling factor by subtracting the scaling factor from the main gradient.

[0098] The system determines an update to the current values of the policy parameters from the final gradient (step 520).

[0099] In particular, the system can backpropagate the final gradient through the action selection policy neural network to determine a gradient with respect to the parameters of the action selection policy neural network that is used to update the current values of the policy parameters.

[0100] In implementations where the system does not use an average neural network, the system can use the main gradient as the final gradient or can scale the main gradient using some other technique, e.g., a trust region technique that limits the per-step changes to the parameters. An example of such a technique is the Trust Region Policy Optimization (TRPO) technique described in Schulman, et al, J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.

[0101 ] The system determines an update to the current values of the critic parameters (step 522). In particular, the update to the critic parameters can be a sum of an update of a gradient that moves the critic parameters in the direction of a target Q value and another gradient that moves the critic parameters in the direction of a target value estimate. For example, the update can satisfy :

min{l, _Pi } (Q ^ret - Qefa, ^ )) V«¾lty (x _f ),

where θ _ν ' are the critic parameters. [0102] The system updates the Retrace estimate and the OPC estimate before the estimates are used in the next iteration of the process 500 (step 524). For example, the updated Retrace estimate can satisfy:

where c _L is equal to min ( ;)rfj and d is a constant, e.g., the dimensionality of the action space. The optional exponent of [Id can lead to faster learning. The updated OPC estimate can satisfy:

(<2°" ^c - <?«¾(*„ a, ))+l¾(x _£ ).

[0103] The system then uses the updated Retrace and OPC estimates as the current estimates for the next iteration of the process 500.

[0104] The above description has described the average network as being used only to adjust the gradients of the action selection policy neural network before the gradients are used to update the parameters of the action selection policy network. However, an average neural network can be used to improve the training of any neural network (referred to as a "main" neural network) by scaling the gradients of the main neural network before they are used to update in updated the values of the parameters of the main neural network (referred to as the "main parameters").

[0105] For example, the main neural network can be the action selection policy neural network described above or can be a different neural network that receives a different kind of network input and generates a different kind of network output.

[0106] General ly, the system can use the average neural network during the training of the main neural network by adjusting gradients based on a measure of difference between the output of the main neural network and the output of the average neural network.

[0107] The main neural network can be configured to receive any kind of digital data input and to generate any kind of score, classification, or regression output based on the input.

[0108] For example, if the inputs to the main neural network are images or features that have been extracted from images, the output generated by the main neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.

[0109] As another example, if the inputs to the main neural network are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the output generated by the main neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

[01 10] As another example, if the inputs to the main neural network are features of an impression context for a particular advertisement, the output generated by the main neural network may be a score that represents an estimated likelihood that the particular advertisement will be clicked on.

[01 1 1 ] As another example, if the inputs to the main neural network are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the main neural network may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item.

[01 12] As another example, if the input to the main neural network is a sequence of text in one language, the output generated by the main neural network may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

[01 13] As another example, if the input to the main neural network is a sequence representing a spoken utterance, the output generated by the main neural network may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance.

[01 14] As above, whenever an average neural network is used in training of a main neural network, the average neural network has the same architecture as the main neural network but different parameter values from those of the main neural network at any given point during the training. In particular, during the training, the current values of the average network parameters represent a running average of values of the main network parameters.

[01 15] FIG. 6 is a flow diagram of an example process 600 for training a main neural network using an average neural network. For convenience, the process 600 will be described as being performed by a system including one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. I , appropriately programmed in accordance with this specification, can perform the process 600. [01 16] The system receives a batch of training data for use in training the main neural network to optimize an objective function (step 602). Generally, the training data includes multiple training examples and specifies, for each training example, a target output that should be generated by the main neural network for the training example. The objective function is general ly a function of, for a given training example, the output generated by the main neural network for the training example and the target output for the training example. For example, the objective function can be a cross-entropy loss function or a maximum likelihood objective function.

[01 17] For each of the plurality of training examples in the batch, the system processes the training example using the main neural network and in accordance with current values of the main network parameters the training example to generate a main network output for the training example (step 604).

[01 18] For each of the plurality of training examples in the batch, the system determines a main gradient of the objective function from the main network output (step 606) using conventional machine learning training techniques.

[01 19] For each of the plurality of training examples in the batch, the system processes the training example using the average neural network and in accordance with current values of the average network parameters to generate an average network output for the training example (step 608).

[0120] For each of the plurality of training examples in the batch, the system determines an auxiliary gradient of a measure of a difference between the average network output and the main network output (step 6 10). In particular, the auxiliary gradient is a gradient with respect to the main network output and the measure of the difference is a ullback-Leibler divergence between (i) the average network output and (ii) main network output.

[012 1 ] For each of the plurality of training examples in the batch, the system determines a final gradient from the main gradient and the auxiliary gradient (step 612). In particular, the system determines, from the auxiliary gradient and the main gradient, a scaling factor for the main gradient and applies the scaling factor to the main gradient to determine the final gradie ling factor 5 can satisfy :

where k is the auxiliary gradient, g is the main gradient, and δ is a constant value. The system can apply the scaling factor to the main gradient by subtracting the scaling factor from the main gradient. [0122] For each of the plurality of training examples in the batch, the system determines an update to the current values of the main network parameters from the final gradient (step 614). For example, the system can backpropagate the final gradient through the main neural network to determine an update for each of the main network parameters. Depending on the training technique being used, the system can either directly use the backpropagated gradients as the updates or can modify the backpropagated gradients to generate the update, e.g., by applying a learning rate to the backpropagated gradients.

[0123] The system updates the current values of the main network parameters using the updates for the training examples in the batch to determine updated values of the main network parameters, e.g., by adding the updates to the current values of the main network parameters (step 616).

[0124] The system updates the current values of the average network parameters using the updated values of the main network parameters to determine updated values of the average network parameters (step 18). In particular, the updated values of the average network parameters are a weighted sum of the current values of the average network parameters and the updated values of the main network parameters.

[0125] The system can repeat the process 600 multiple times on multiple different batches of training data to train the main neural network, i.e., to determine trained values of the main network parameters from initial values of the parameters.

[0126] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0127] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0128] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0129] A computer program, which may also be referred to or described as a program, software, a software appl ication, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0130] In this specification the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. General ly, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. [013 1 ] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0132] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0133] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0134] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0135] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0136] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0137] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this spec ification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0138] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0139] While this specification contains many specific implementation detai ls, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0140] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[014 1 ] Particular embodi ments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompany ing figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.