LEARNING GOALS

BACKGROUND

[0001] This specification relates to reinforcement leaming.

[0002] In a reinforcement leaming system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

[0003] Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.

[0004] Neural networks are machine leaming models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

[0005] This specification generally describes how a system implemented as one or more computers in one or more locations can process demonstration sequences to determine features that are discriminative for each of multiple subtasks of a reinforcement leaming task to be performed by an agent interacting with an environment. The system can then use the discriminative features to generate rewards to train the agent to perform the reinforcement learning task.

[0006] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The described systems can partition a reinforcement leaming task into subtasks and thereby detect intermediate goals in an unsupervised manner, i.e., directly from a video or other sequence of images of a demonstrator performing the task and without any explicit specification of sub- goals. More specifically, the system can determine the partitioning from a very small number of task demonstrations, e.g., from less than twenty demonstration sequences. The system can then use the generated partitioning to generate a reward function for use in training a reinforcement learning agent to perform the task. Thus, using the described techniques, reward functions which are dense and smooth can be generated in an

unsupervised manner and can be used to effectively train a reinforcement learning agent to perform a complex task solely from a small number of unlabeled demonstration images. Alternatively, the reward functions can be used to augment an existing reward signal, e.g., a manually designed signal, thereby speeding up the training of the reinforcement learning agent, improving the performance of the trained reinforcement learning agent, or both. Because a given reinforcement learning task likely has implicit sub-goals and steps that comprise more complex behaviors, extracting such sub-goals as described in this specification can allow the agent to make maximal use of information contained in a demonstration during training without requiring costly and often infeasible labelling of the demonstration data.

[0007] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subj ect matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 shows an example reinforcement learning system.

[0009] FIG. 2 is a flow diagram of an example process for partitioning a reinforcement learning task into subtasks.

[0010] FIG. 3 is a flow diagram of an example process for generating a reward.

[0011] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0012] This specification generally describes how a system implemented as one or more computers in one or more locations can partition a reinforcement learning task to be performed by an agent into subtasks using demonstration sequences, determine features that are discriminative for each of the subtasks, and then use those discriminative features to generate rewards during the training of the agent.

[0013] FIG. 1 shows an example reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. [0014] The reinforcement leaming system 100 is a system that trains a reinforcement learning agent 102 to interact with an environment 106 to perform a reinforcement learning task using demonstration data 130 of the task being performed by a demonstrator.

[0015] Generally, the environment 106 is a real-world environment and the reinforcement learning agent 102 is a mechanical agent interacting with the environment 102. For example, the agent 102 may be a robot 102 that interacts with the environment to perform the reinforcement learning task. Examples of tasks that the agent 102 can perform include moving a specified object from one location to another in the environment, locating a specified object in the environment, or navigating from one location to another in the environment.

[0016] The reinforcement learning system 100 trains the reinforcement leaming agent 102 by training a policy for controlling the agent 102, i.e., a policy that defines actions to be performed by the agent 102 while the agent 102 attempts to perform the reinforcement learning task.

[0017] For example, the policy may be a neural network that receives an observation characterizing the state of the environment, e.g., an image of the environment as captured by the agent 102, and generates a policy output that defines the action to be performed by the agent 102 in response to the observation, e.g., that generates parameters of a distribution over possible joint torques for the joints of the agent 102.

[0018] As another example, the policy may be a simpler controller, e.g., a linear-Gaussian parameterization of the policy, that takes as input a lower-dimensional state representation, e.g., the joint angles and angular velocities of the agent 102, and outputs the parameters of a distribution over possible joint torques for the agent 102.

[0019] In particular, the demonstration data 130 used by the system 100 to train the agent 102 includes multiple demonstration sequences. A demonstration sequence is a sequence of images of the environment that are captured while an instance of the reinforcement leaming task is being performed by a demonstrator, e.g., another mechanical agent or a human demonstrator.

[0020] To use the demonstration sequences to train the agent 102, the system 100 processes each image in the demonstration sequence through an image processing neural network 140 to determine feature values 142 for a respective set of features for the image.

[0021] Generally, the image processing neural network 140 is a neural network that has been pre-trained on an image processing task, e.g., an object detection or image classification task, without further modification of the values of the parameters of the image processing neural network 140. For example, the image processing neural network 140 may be an Inception image classification neural network that has been pre-trained on an image classification data set. Such a neural network is described in more detail in Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, 2016.

[0022] For each image, the system 100 determines the feature values for the image from activations generated by one or more of the hidden layers of the network 140 during the processing of the image. For example, the feature values may be the output of a

predetermined one of the hidden layers of the neural network 140. As another example, the feature values may include outputs of multiple hidden layers of the neural network 140, e.g., each layer after a predetermined layer in the network.

[0023] A task partitioning engine 150 determines, from the feature values 142 for the images in the demonstration sequences, a partitioning of the reinforcement learning task into subtasks such that each image in each demonstration sequence is assigned to a respective subtask. Thus, completing the reinforcement learning task can be decomposed into accomplishing multiple intermediate goals, i.e., completing each of the subtasks in the partitioning. The task partitioning engine 150 can make this determination based on the images in the demonstration sequences and without any external labeling of the subtasks, i.e., in an unsupervised manner. Partitioning the reinforcement learning task into subtasks using the demonstration images is described in more detail below with reference to FIG. 2.

[0024] The task partitioning engine 150 then determines, from the feature values for the images in the demonstration sequences, a respective set of discriminative features for each of the subtasks. The discriminative features for a given subtask are features that are highly discriminative features of a specific goal to be achieved in the subtask, i.e., relative to the other subtasks in the partition, while remaining invariant to unrelated variation (e.g., lighting, color, and viewpoint). Determining discriminative features for each of the subtasks is also described in more detail below with reference to FIG. 2.

[0025] Once the task partitioning engine 150 has determined the discriminative features for each of the subtasks, a reward engine 160 can use this information to generate rewards for the agent 102 based on feature values for the discriminative features for images of the environment 160 generated during the training of the agent 102, i.e., to reflect the agent's progress in completing one or more of the subtasks. This reward can then be used to improve the training of the agent, i.e., to speed up the training, to improve the performance of the agent on the task once trained, or both. [0026] In particular, a training engine 170 trains the agent 102, i.e., trains the policy, using a reinforcement learning technique. In reinforcement learning, a policy is learned that, when used to select actions to be performed by an agent, maximizes a measure of cumulative reward received by the agent as a result of interacting with the environment. For example, the measure can be the time-discounted sum of rewards received during the performance of the task. Generally, rewards are numeric values that represent how well the agent is performing a given task and are used in the reinforcement learning technique to adjust the policy to improve the performance of the agent on the task.

[0027] The training engine 170 can use any appropriate reward-based reinforcement learning technique to train the agent 102. For example, when the policy is a deep neural network, the engine can use a deep deterministic policy gradient (DDPG) technique. Such a technique is described in more detail in Lillicrap, Timothy, et al, Continuous control with Deep

Reinforcement Learning, ICLR 2016. As another example, when the policy is a simpler controller, the engine can use a path-integral based technique. An example of such a technique is described in Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine. Path integral guided policy search.

[0028] In some cases, the reward that is generated by the reward engine 160 is the only reward that the training engine 170 uses to train the agent 102 using the reinforcement learning technique. In other cases, the training engine 170 may also receive other rewards from an external source and combine the rewards generated by the reward engine 160 with the external rewards, e.g., as a weighted average or a weighted sum, and use the total reward as the reward for the reinforcement learning technique.

[0029] FIG. 2 is a flow diagram of an example process 200 for partitioning a reinforcement learning task into subtasks. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. l, appropriately programmed, can perform the process 200.

[0030] The system obtains multiple demonstration sequences (step 202). As described above, each demonstration sequence is a sequence of images of a demonstrator performing the reinforcement learning task that the agent will be trained to perform. In some cases, certain characteristics of the tasks performed by the demonstrator or demonstrators differ between the various demonstration sequences and from the task that the agent will be required to perform. For example, the initial state of the task may be different between some or all of the demonstration sequences. [0031] The system processes each image in each demonstration sequence through an image processing neural network to determine, for each image, feature values for a respective set of features for the image (step 204). In particular, the system determines the feature values for a given image from the activations generated by one or more of the hidden layers of the image processing neural network. For example, the feature values may be the activations of a predetermined hidden layer of the neural network or the activations of multiple

predetermined hidden layers of the neural network.

[0032] The system determines a partitioning of the reinforcement task into multiple subtasks (step 206), with each image in each demonstration sequence being assigned to a respective subtask of the multiple subtasks.

[0033] Generally, the system partitions the reinforcement learning task such that each image in each partition is abstractly similar to each other image in the partition. In some cases, the system accomplishes this by selecting a partitioning that (approximately) minimizes average feature value variance between images within each partition.

[0034] In some implementations, the system selects the partitioning that minimizes average feature value variance by recursively adjusting the partitioning to determine the partitioning that minimizes the average feature value variance starting from a predetermined minimum subtask size and that satisfies some criteria, e.g., that each subtask includes at least a predetermined minimum number of images and that the total number of partitions is fixed to a predetermined number.

[0035] In some other implementations, the system selects the partitioning that approximately minimizes average feature value variance by first splitting the entire sequence in two and then recursively splitting each new sequence in two as described above until the total number of partitions has been reached.

[0036] In some implementations, the system partitions each demonstration sequence individually and then aligns the partitioning prior to the selection of the discriminative features. In some other implementations, the system jointly partitions the demonstration sequences, with the recursion taking into consideration the feature values in all of the demonstration sequences.

[0037] The system determines, from the feature values for the images in the demonstration sequences, a respective set of discriminative features for each of the plurality of subtasks (step 208). As described above, the discriminative features for a given subtask are features that are highly discriminative features of a specific goal to be achieved in the subtask while remaining invariant to unrelated variation (e.g., lighting, color, and viewpoint). [0038] In some implementations, to determine the discriminative features for a given subtask, the system processes the features using a classifier, i.e., a deep or a shallow classifier, that is configured to generate an output identifying the discriminative features of a given subtask.

[0039] In some other implementations, the system first normalizes the feature values of the feature across the images in the demonstration sequences to determine a normalized feature value of the feature for each image, i.e., by subtracting the mean of the feature for all of the demonstration images and dividing by the standard deviation of the feature for all of the demonstration images.

[0040] The system then computes, for each subtask, distribution statistics, e.g., a mean and a standard deviation, for normalized feature values of the feature for images that are not in the subtask and the same distribution statistics for normalized feature values of the feature for images that are in the subtask.

[0041] For a given subtask, the system can then determine a score for each feature for the subtask from the distribution statistics for the normalized feature values of the feature for images that are not in the subtask and the distribution statistics for the normalized feature values of the feature for images that are in the subtask. For example, the score zi for a feature i for a given subtask can satisfy:

z _t = α \μ - μϊ \ - (σ+ - σ ),

where α is a predetermined positive constant, μ+ is the average feature value for the feature for the images in the subtask, μϊ is the average feature value for the feature for the images not in the subtask, σ _έ ⁺ is the standard deviation of the feature values for the feature for the images in the subtask, and a is the standard deviation for the feature for the images not in the subtask.

[0042] The system can then select, for each subtask, a fixed number of highest-scoring features as the discriminative features for the subtask.

[0043] FIG. 3 is a flow diagram of an example process 300 for generating a reward for a reinforcement learning technique. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG. l , appropriately programmed, can perform the process 300.

[0044] The system can repeatedly perform the process 300 during the training of the reinforcement agent to provide rewards for use in the training of the agent. [0045] The system receives a current image characterizing a current state of the environment (step 302). Generally, the image is an image of the agent performing the task during the training of the agent.

[0046] The system processes the current image through the image processing neural network to determine feature values for the current image (step 304). The system uses the same intermediate outputs of the image processing neural network to generate the feature values as were used for the demonstration images, i.e., uses the outputs of the same one or more hidden layers of the neural network as the feature values.

[0047] The system generates, for each of one or more of the subtasks, a respective perception-based reward from the feature values of the features for the subtask (step 306). Generally, the reward for a given subtask is based on how similar the feature values for the discriminative features of the current image are to the feature values of the discriminative features for the demonstration images that are in the given subtask. In some

implementations, the reward for the given subtask is based on a comparison between the feature values for the current image and the average of the feature values for the

demonstration images in the subtask. For example, in some implementations the reward R _Q (s _t ) for a subtask g for the current image st satisfies:

where n is the total number of subtasks, j ranges over the M discriminative features of the subtask g, Sij _t is the feature value for feature j of the current image, μ is the average of the feature values for the feature j of the demonstration images in the subtask, and aj _t is the standard deviation of the feature values for the feature j of the demonstration images in the subtask.

[0048] In some other implementations, the system instead processes the feature values (either for the discriminative features or for all of the features) for the current image using a classifier that has been trained to receive the feature values and to generate the reward for the subtask. That is, the system maintains a respective classifier for each subtask and processes the feature values through each of the classifiers to determine the rewards for each of the subtasks. The classifier can be a linear layer that has been trained on the demonstration images to receive feature values and to map the received feature values to a single reward score.

[0049] The system provides the one or more perception-based rewards for use in training the agent to perform the reinforcement learning task (step 308). In particular, because reinforcement learning techniques typically train the agent using a single reward received at each training time step, the system combines the perception-based rewards and uses the combined reward in the training of the agent, i.e., as the only reward or in combination with another reward derived from a different source. In some implementations, the combined reward partially rewards intermediate steps but emphasizes later rewards, i.e., because successfully completing later subtasks is more indicative of successfully completing the task. For example, in some implementations the combined reward R(a) for a given set of feature values a satisfies:

R(a) =∑? ₌₂ R _i (a) * 2 ⁽ⁱ - ¹

where n is the total number of sub-tasks and #;(a) is the reward for subtask i. In this example, the first sub-task is ignored because it is assumed to be the initial starting state in the demonstration sequences.

[0050] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0051] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0052] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0053] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0054] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0055] Similarly, in this specification the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. [0056] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0057] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0058] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0059] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0060] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0061] Machine learning models can be implemented and deployed using a machine learning framework, .e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0062] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0063] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0064] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0065] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0066] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.