CROSS-REFERENCE TO RELATED APPLICATION [0001] This application claims the benefit of U.S. Provisional Application No. 62/936,259, filed on November 15, 2019. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

BACKGROUND

[0002] This specification relates to processing point cloud data using neural networks to generate an output characterizing one or more agents in the environment.

[0003] The environment may be a real-world environment, and the agents may be, e.g., pedestrians in the vicinity of an autonomous vehicle in the environment. Making predictions about pedestrians in the environment is a task required for motion planning, e.g., by the autonomous vehicle.

[0004] Autonomous vehicles include self-driving cars, boats, and aircraft. Autonomous vehicles use a variety of on-board sensors and computer systems to detect nearby objects and use such detections to make control and navigation decisions.

[0005] Some autonomous vehicles have on-board computer systems that implement neural networks, other types of machine learning models, or both for various prediction tasks, e.g., object classification within images. For example, a neural network can be used to determine that an image captured by an on-board camera is likely to be an image of a nearby car. Neural networks, or for brevity, networks, are machine learning models that employ multiple layers of operations to predict one or more outputs from one or more inputs. Neural networks typically include one or more hidden layers situated between an input layer and an output layer. The output of each layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer.

[0006] Each layer of a neural network specifies one or more transformation operations to be performed on input to the layer. Some neural network layers have operations that are referred to as neurons. Each neuron receives one or more inputs and generates an output that is received by another neural network layer. Often, each neuron receives inputs from other neurons, and each neuron provides an output to one or more other neurons.

[0007] An architecture of a neural network specifies what layers are included in the network and their properties, as well as how the neurons of each layer of the network are connected. In other words, the architecture specifies which layers provide their output as input to which other layers and how the output is provided.

[0008] The transformation operations of each layer are performed by computers having installed software modules that implement the transformation operations. Thus, a layer being described as performing operations means that the computers implementing the transformation operations of the layer perform the operations.

[0009] Each layer generates one or more outputs using the current values of a set of parameters for the layer. Training the neural network thus involves continually performing a forward pass on the input, computing gradient values, and updating the current values for the set of parameters for each layer using the computed gradient values, e.g., using gradient descent. Once a neural network is trained, the final set of parameter values can be used to make predictions in a production system.

SUMMARY

[10] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that process a temporal sequence of point cloud data inputs to make predictions about agents, e.g., pedestrians, vehicles, bicyclists, motorcyclists, or other moving objects, characterized by the point cloud data inputs.

[11] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[12] Detecting agents and, more specifically, pedestrians and predicting future trajectories for them are critical tasks for numerous applications, such as autonomous driving. In particular, to drive safely and smoothly, autonomous vehicles not only need to detect where the objects are currently (i.e. object detection), but also need to predict where they will go in the future (i.e. trajectory prediction). Among the different types of objects that are frequently encountered by self-driving cars, pedestrian is an important and difficult type for existing techniques to detect and predict accurately. The difficulty comes at least in part from the complicated properties of pedestrian appearance and behavior, e.g. deformable shape of pedestrian bodies and interpersonal relations between pedestrians.

[13] Existing systems either treat the detection and trajectory prediction as separate tasks or simply add a trajectory regression head on top of an object detector.

[14] The described techniques, on the other hand, employ an end-to-end two-stage neural network, referred to as a spatio-temporal-interactive network. In addition to 3D geometry modeling of pedestrians, the spatio-temporal-interactive network models the temporal information for each of the pedestrians. To do so, the spatio-temporal-interactive network predicts both current and past locations in the first stage, so that each pedestrian can be linked across frames and comprehensive spatio-temporal information can be captured in the second stage. Also, the spatio-temporal-interactive network models the interaction among objects with an interaction graph, to gather information among the neighboring objects for any given pedestrian. This results in a system that achieves state-of-the-art results in both object detection and future trajectory prediction for agents, e.g., for the pedestrian agent type.

[15] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[16] FIG. 1 is a diagram of an example system.

[17] FIG. 2 is a flow diagram of an example process for processing a temporal sequence of point cloud data.

[18] FIG. 3 illustrates the operation of the point cloud processing system for a given temporal sequence of point cloud inputs.

[19] FIG. 4 shows an example of the generation of the temporal region proposal features of a given temporal region proposal by the STI feature extractor.

[20] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[21] This specification describes how a vehicle, e.g., an autonomous or semi- autonomous vehicle, can generate predictions characterizing surrounding agents in the vicinity of the vehicle in the environment by processing a temporal sequence of point cloud inputs.

[22] Each point cloud input includes point cloud data generated from data captured by one or more sensors of a vehicle at a corresponding time step. The point cloud data includes data defining a plurality of three-dimensional points, i.e., i.e., coordinates of the points in some specified coordinate system, and, optionally, features for each of the plurality of three- dimensional points, e.g., intensity, second return, and so on. [23] The sequence is referred to as a “temporal” sequence because the point cloud inputs are ordered within the sequence according to the time at which the sensors captured the data used to generate the point cloud data.

[24] In this specification, an “agent” can refer, without loss of generality, to a vehicle, bicycle, pedestrian, ship, drone, or any other moving object in an environment.

[25] While this description generally describes point cloud processing techniques being performed by an on-board system of an autonomous vehicle, more generally, the described techniques can be performed by any system of one or more computers in one or more locations that receives or generates temporal sequences of point clouds.

[26] FIG. 1 is a diagram of an example system 100. The system 100 includes an on board system 110 and a training system 120.

[27] The on-board system 110 is located on-board a vehicle 102. The vehicle 102 in FIG. 1 is illustrated as an automobile, but the on-board system 102 can be located on-board any appropriate vehicle type. The vehicle 102 can be a fully autonomous vehicle that determines and executes fully-autonomous driving decisions in order to navigate through an environment. The vehicle 102 can also be a semi-autonomous vehicle that uses predictions to aid a human driver. For example, the vehicle 102 can autonomously apply the brakes if a prediction indicates that a human driver is about to collide with another vehicle.

[28] The on-board system 110 includes one or more sensor subsystems 130. The sensor subsystems 130 include a combination of components that receive reflections of electromagnetic radiation, e.g., lidar systems that detect reflections of laser light, radar systems that detect reflections of radio waves, and camera systems that detect reflections of visible light.

[29] The sensor data generated by a given sensor generally indicates a distance, a direction, and an intensity of reflected radiation. For example, a sensor can transmit one or more pulses of electromagnetic radiation in a particular direction and can measure the intensity of any reflections as well as the time that the reflection was received. A distance can be computed by determining how long it took between a pulse and its corresponding reflection. The sensor can continually sweep a particular space in angle, azimuth, or both. Sweeping in azimuth, for example, can allow a sensor to detect multiple objects along the same line of sight.

[30] The sensor subsystems 130 or other components of the vehicle 102 generate temporal sequences of multiple point cloud inputs using the sensor data generated by one or more of the sensors. Each point cloud input in the temporal sequence includes points that correspond to reflections of laser light transmitted by one of the sensors, i.e., three- dimensional points that correspond to locations where a reflection occurred.

[31] The sensor subsystems 130 send the temporal point cloud sequences 132 to a point cloud processing system 150.

[32] The point cloud processing system 150 processes the temporal sequence 132 to generate a predicted output 152 that characterizes the scene, e.g., an object detection output that identifies locations of one or more agents in the scene, a behavior prediction output that predicts the future trajectory of the agents in the scene, or both.

[33] Generally, the point clouding system 150 processes the temporal sequence 132 using a spatio-temporal-interactive neural network to generate the predicted output 152. Processing the point clouds will be described in more detail below with reference to FIGS. 2- 4.

[34] The on-board system 110 also includes a planning system 160. The planning system 160 can make autonomous or semi-autonomous driving decisions for the vehicle 102, e.g., by generating a planned vehicle path that characterizes a path that the vehicle 102 will take in the future.

[35] The on-board system 100 can provide the predicted output 152 generated by the point cloud processing system 150 to one or more other on-board systems of the vehicle 102, e.g., the planning system 160 and/or a user interface system 165.

[36] When the planning system 160 receives the predicted output 152, the planning system 160 can use the predicted output 152 to generate planning decisions that plan a future trajectory of the vehicle, i.e., to generate a new planned vehicle path. For example, the predicted output 152 may contain a prediction that a particular surrounding agent is likely to cut in front of the vehicle 102 at a particular future time point, potentially causing a collision. In this example, the planning system 160 can generate a new planned vehicle path that avoids the potential collision and cause the vehicle 102 to follow the new planned path, e.g., by autonomously controlling the steering of the vehicle, and avoid the potential collision.

[37] When the user interface system 165 receives the predicted outputs 152, the user interface system 165 can use the predicted output 152 to present information to the driver of the vehicle 102 to assist the driver in operating the vehicle 102 safely. The user interface system 165 can present information to the driver of the agent 102 by any appropriate means, for example, by an audio message transmitted through a speaker system of the vehicle 102 or by alerts displayed on a visual display system in the agent (e.g., an LCD display on the dashboard of the vehicle 102). In a particular example, the predicted output 152 may contain a prediction that a particular surrounding agent is likely to step out in front of the vehicle 102, potentially causing a collision. In this example, the user interface system 165 can present an alert message to the driver of the vehicle 102 with instructions to adjust the trajectory of the vehicle 102 to avoid a collision or notifying the driver of the vehicle 102 that a collision with the particular surrounding agent is likely.

[38] To generate the predicted output 152, the point cloud processing system 150 can use trained parameter values 195, i.e., trained model parameter values of the spatio-temporal- interactive neural network that is used by the point cloud processing system 150, obtained from a model parameters store 190 in the training system 120.

[39] The training system 120 is typically hosted within a data center 124, which can be a distributed computing system having hundreds or thousands of computers in one or more locations.

[40] The training system 120 includes a training data store 170 that stores all the training data used to train the trajectory prediction system i.e., to determine the trained parameter values 195 of the point cloud processing system 150. The training data store 170 receives raw training examples from agents operating in the real world. For example, the training data store 170 can receive a raw training example 155 from the vehicle 102 and one or more other agents that are in communication with the training system 120. The raw training example 155 can be processed by the training system 120 to generate a new training example. The new training example can include a temporal sequence of point cloud data that can be used as input for the point cloud processing system 150. The new training example can also include outcome data, e.g., data characterizing the state of the environment surrounding the agent from which the training example 155 was received at one or more future time points, data identifying the objects that are measured in the temporal sequence, or both. This outcome data can be used to generate ground truth outputs, e.g., ground truth trajectories, ground truth detection outputs, or both, for one or more agents in the vicinity of the vehicle or other agent. Each ground truth trajectory identifies the actual trajectory (as derived from the outcome data) traversed by the corresponding agent at the future time points. For example, the ground truth trajectory can identify spatial locations in an agent-centric coordinate system to which the agent moved at each of multiple future time points. Each ground truth detection output identifies the regions in the point clouds in the temporal sequence that correspond to actual measurements of the agent.

[41] The training data store 170 provides training examples 175 to atraining engine 180, also hosted in the training system 120. The training engine 180 uses the training examples 175 to update model parameters that will be used by the point cloud processing system 150, and provides the updated model parameters 185 to model parameters store 190. Once the parameter values of the point cloud processing system 150 have been fully trained, the training system 120 can send the trained parameter values 195 to the point cloud processing system 150, e.g., through a wired or wireless connection.

[42] FIG. 2 is a flow diagram of an example process 200 for processing a temporal sequence of point cloud data inputs. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a trajectory prediction system, e.g., the point cloud processing system 150 of FIG.

1, appropriately programmed in accordance with this specification, can perform the process 200

[43] When performed by a system on-board an autonomous vehicle, the system can repeatedly perform the process 200 as the autonomous vehicle navigates through the environment in order to improve the navigation of the autonomous vehicle.

[44] The system obtains, i.e., receives or generates, a temporal sequence that has respective point cloud inputs at each of a plurality of time steps during a time interval (step 202). Each point cloud input includes point cloud data generated from data captured by one or more sensors of a vehicle at the time step.

[45] The system generates, from the temporal sequence, a respective feature representation for each of a plurality of time windows within the time interval (step 204). In some cases each time window corresponds to a respective one of the time steps. In other cases, each time window corresponds to multiple time steps, i.e., is generated from the point cloud data at multiple ones of the time steps. Each feature representation includes respective features for each of a plurality of spatial locations in the feature representation and each spatial location in each feature representation maps to a corresponding location in the environment. For example, each “pixel” (spatial location) in the feature representation can be mapped to a corresponding region of the environment according to some coordinate system, e.g., a perspective view or a top-down view. Generating the feature representations will be described in more detail below with reference to FIG. 3. [46] The system processes the feature representations using a temporal region proposal neural network to generate a plurality of temporal region proposals (step 206).

[47] Each temporal region proposal corresponds to a possible agent in the environment and each temporal region proposal identifies a respective spatial region in each of the feature representations, i.e., a region that includes multiple contiguous spatial locations in the feature representation. That is, each temporal region proposal identifies respective spatial regions in multiple ones of the feature representations.

[48] For each temporal region proposal, the respective spatial region identified by the temporal region proposal in any given feature representation is a prediction of where in the environment the corresponding possible agent was located during the time window corresponding to the given feature representation.

[49] The agents are referred to as “possible” agents because not all of the proposals may correspond to actual agents in the environment, i.e. the temporal region proposal neural network may generate more proposals than there actual agents in the environment.

[50] Generating the temporal region proposals will be described in more detail below with reference to FIG. 3.

[51] The system generates, for each temporal region proposal and from the feature representations, temporal region proposal features (step 208). The temporal region proposal features for any given proposal characterize the spatial regions identified in the proposal. Generating these features is described in more detail below with reference to FIG. 4.

[52] For each temporal region proposal, the system can then process the temporal region proposal features to generate one or more predictions for the corresponding possible agent (step 210).

[53] For example, the system can process the temporal region proposals to generate a first output that predicts a future trajectory after the time interval of the possible agent corresponding to the temporal region proposal.

[54] As another example, the system can process the temporal region proposal features to generate a second output that identifies a current location in the environment of the corresponding possible agent at the end of the time interval.

[55] As another example, the second output can also include a final confidence score that represents a likelihood that the corresponding possible agent is an actual agent in the environment. [56] FIG. 3 illustrates the operation of the point cloud processing system for a given temporal sequence of point cloud inputs.

[57] As shown in FIG. 3, the system receives an input temporal sequence 302 that includes three point clouds at each of three time steps during a time interval.

[58] The system then processes the sequence 302 using a spatio-temporal-interactive neural network that includes, in the example of FIG. 3, an encoder neural network 310, a backbone neural network 320, a temporal region proposal neural network 330, a spatio- temporal-interactive (STI) feature extractor 340, an object detection head 350, and a trajectory prediction head 360.

[59] The system processes the temporal sequence 302 using an encoder neural network 310 to generate a respective initial feature representation 312 for each of a plurality of time windows within the time interval. While the example of FIG. 3 shows that there are the same number of initial feature representations 312 as there are point clouds in the input sequence, in practice, to reduce the memory usage of the processing pipeline, the system can generate a reduced number of feature representations so that each time window corresponds to multiple time steps.

[60] In particular, to generate the initial feature representation 312 for a given time interval, the system can process each of the point clouds at time steps in the given time interval using the encoder neural network to generate a respective pseudo image, i.e., an H x W x C tensor, for each point cloud and then concatenates the resulting pseudo images, i.e., along the depth dimension, to generate the feature representation for the time interval.

[61] The encoder neural network 310 can be any appropriate encoder neural network that maps an input point cloud to a pseudo image. For example, the encoder neural network can assign the points in the point cloud to voxels and then generate a respective feature vector for each voxel. One example of such an encoder neural network is described in Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490-4499, 2018. Another example of such an encoder neural network is described in Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12697-12705, 2019. [62] The system processes the initial feature representations 312 using a backbone neural network 320 to generate the feature representations 322 (also referred to as backbone features).

[63] In particular, the system processes each initial feature representation 312 using the backbone neural network 320 to generate a respective feature representation 322 for the corresponding time window. Each feature representation has the same spatial dimensionality as the corresponding initial feature representation but may include a different number of channels than the initial feature representation.

[64] The backbone neural network 320 can be any appropriate convolutional neural network that is configured to receive an input pseudo image and to process the input pseudo image to generate an output representation that has the same spatial dimensionality as the input pseudo image. As a particular example, the backbone neural network 320 can have a U-net architecture, which is described in more detail in Olaf Ronneberger, Philipp Fischer, and Thomas Brox. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.

[65] The system then processes the feature representations 322 using to generate a plurality of temporal region proposals 332.

[66] Each temporal region proposal 332 corresponds to a possible agent in the environment and each temporal region proposal 332 identifies a respective spatial region in each of the feature representations, i.e., a region that includes multiple contiguous spatial locations in the feature representation.

[67] For each temporal region proposal 332, the respective spatial region identified by the temporal region proposal in any given feature representation 322 is a prediction of where in the environment the corresponding possible agent was located during the time window corresponding to the given feature representation. In other words, for a particular feature representation, the spatial region is a prediction of where the possible agent was located during the corresponding time window.

[68] In particular, the temporal region proposal neural network 330 can generate a fixed number of candidate temporal region proposals, each corresponding to a different candidate possible agent in the environment. The agents are referred to as “possible” agents because not all of the proposals may correspond to actual agents in the environment. [69] More specifically, the temporal region proposal neural network 330 can generate a respective candidate temporal region proposal corresponding to each of a fixed number of fixed anchor regions. Each anchor region is a different contiguous spatial region, e.g., a box, within the spatial dimensions of the feature representations. Thus, the output of the neural network 330 includes, for each candidate temporal region proposal and for each feature representation, a regression vector that identifies a region in each feature representation relative to the anchor region corresponding to the temporal region proposal. That is, for candidate temporal region proposal, the output includes a respective regression vector for each of the plurality of feature representations.

[70] In particular, each anchor region can be defined by fixed coordinates of a box in the feature representation and a fixed heading of the box. Each regression vector defines a respective delta value for each of the coordinates and for the heading. The delta values can be applied to the fixed coordinates and headings of the box to generate the spatial region specified by the temporal proposal. By having different delta values and different headings for different feature representations, the neural network 330 can model the trajectory of the possible agent across time.

[71] As a specific example, each anchor can be specified by a set of values that include the x, y coordinates of the box center, the width w of the box, the length 1 of the box, and the heading h of the box. The regression vector for the feature representation corresponding to the most recent time window can include delta values for the x and y coordinates, the width of the box, the length of the box, and the heading of the box. For the other feature representations, the regression vectors can include only delta values for the x and y coordinates, and the heading of the box, since the size of the agent should not change across different time windows within the time intervals.

[72] The delta values can then be mapped to a respective spatial region within each feature representation by transforming the anchor box using the delta values in the corresponding regression vector. An example of the transformations that are applied to the most recent feature representation (at time t = 0) is shown below, where the superscript “a ” refers to a value for an anchor box, the superscript “gt” refers to a value for the corresponding ground truth box, and the prefix “d” refers to a delta value for the value that follows:

[73] By applying equations (1) through (5) above, the system can determine ground truth delta values that are used as targets for the training of the neural network 330.

[74] In other words, the system can compute a loss between predicted delta values and the delta values that are found by applying equations (1) through (5), e.g.., an LI loss, a smooth LI loss, or other loss that measures a distance between two vectors, in order to update the parameters of the neural network 330 and the neural networks 310 and 320, e.g., through stochastic gradient descent with

[75] After training, by reversing equations (1) through (5) to solve for the values with the “gt” superscripts given the delta values and the anchor box values, the system can obtain the values that define a region proposal for the most recent time window, i.e., the region proposal will be defined by the “gt” superscripted values in equation (1) through (5) given the predicted “d” prefixed values and the fixed “a” superscripted values.

[76] Similar equations can be used to compute losses and generate proposals for earlier time windows in the time window, except with the width and length being fixed to the width and length predicted for the most recent time window.

[77] Thus, the system can train the neural networks 330, 320, and 310 to minimize a combination, e.g., a sum or a weighted sum, of losses for the time intervals in the time window.

[78] To generate the regression vectors for the feature representations, the neural network 330 concatenates the feature representations in the channel dimension and applies a lxl convolution to the concatenated feature representations to generate a temporal-aware feature map. The neural network 330 then generates the regression vectors for each of the candidate temporal proposals by applying lxl convolutional layers over the temporal-aware feature map.

[79] In some implementations, the candidate temporal region proposals are the final set of temporal region proposals 332. [80] In some other implementations, however, the neural network 330 filters the candidate temporal region proposals to generate the temporal region proposals 332.

[81] In particular, the neural network 330 can generate the plurality of candidate temporal region proposals and a confidence score for each of the candidate temporal region proposals. The confidence score indicates a predicted likelihood that the candidate temporal region proposal corresponds to an actual agent in the environment. The neural network 330 can also generate these confidence scores by applying lxl convolutions over the temporal- aware feature map.

[82] The neural network 330 can then filter the candidate temporal region proposals by applying non-maximum suppression to the regions of the feature representation corresponding to the most recent time window identified by the candidate temporal region proposals based on the confidence scores to remove redundant candidate temporal region proposals.

[83] In these implementations, the system can include a classification loss in the loss function used to train the neural networks 330, 320, and 310, e.g., a cross-entropy loss, that, if the anchor has a corresponding ground truth object, measures an error between the confidence score and a target confidence score that is equal to 1 if the region in the most recent feature representation in the proposal has more than a threshold amount of overlap as measured by intersection over union with the ground truth region and 0 otherwise.

[84] The system then generates, for each of the temporal region proposals 332, respective temporal region proposal features 342 using a spatio-temporal-interactive (STI) feature extractor 340.

[85] FIG. 4 shows an example of the generation of the temporal region proposal features 342 of a given temporal region proposal 332 by the STI feature extractor 340.

[86] As shown in FIG. 4, the STI feature extractor generates local geometry features 410, local dynamics features 420, and history path features 430.

[87] To generate the local geometry features 410 for the temporal region proposal 322, the extractor 340 crops each feature representation to include only the region within that feature representation that is identified by the temporal region proposal 322. That is, as described above, the temporal region proposal 322 identifies a respective region within each feature representation. The extractor 340 generates local geometry features that include only the identified regions with each of the feature representations. In some cases, to simplify computation, the system approximates the identified regions by mapping each identified region to the closest standing box within the feature representation to the identified region.

[88] To generate the local dynamics features 420 for the temporal region proposal 322, the extractor 340 identifies a spatial region that includes all of the regions identified by the temporal region proposal 322 in all of the feature representations and cropping each feature representation to include only the identified region. That is, the extractor 340 generates a “meta box” that covers the whole movement of the possible agent within the time interval by identifying a region that is a super set of, i.e., includes all of, the regions identified by the temporal region proposal 322 in any of the feature representations. In particular, the identified region can be the smallest spatial region that includes all of the regions identified by the temporal region proposal in all of the feature representations. In some cases, to simplify computation, the system approximates the smallest spatial region by mapping each identified region to the closest standing box within the feature representation to the identified region and then computing the smallest box that includes all of the closest standing boxes for all of the identified regions. Because the local dynamics feature 420 covers all of the movement of the possible agent, the dynamics feature captures the direction, curvature and speed of the agent, all of which are useful for future trajectory prediction.

[89] To history path feature 430 for the temporal region proposal 322 identifies a location displacement of the region regions identified by the temporal region proposal 322 across the feature representations. In particular, the extractor 340 generates the history path feature 430 based on, for each pair of feature representations that includes the feature representation corresponding to the most recent time window, the difference in the location of (i) the center of the spatial region in the feature representation for the most recent time window and (ii) the center of the spatial region in the other feature representation in the pair. For example, the system can compute, for each pair, the difference between the x and y coordinates of the centers of the two spatial regions in the pair and then processes these differences using a multi-layer perceptron (MLP) to generate the history path feature 430.

[90] The extractor 340 then generates the temporal region proposal features 342 for the proposal 332 from the features 410, 420, and 430.

[91] As a particular example, the extractor 340 can generate the temporal region proposal features 342 by processing the local geometry features 410 and the local dynamics features 420 using a neural network, e.g., a convolutional neural network, e.g., a ResNet, that has a pooling layer, e.g., a global average pooling layer, as the output layer of the neural network, to generate a combined feature and concatenating the combined feature with the history path feature 342. Processing the local geometry features 410 and the local dynamics features 420 using the neural network can aggregate spatial and temporal knowledge across the feature representation for the proposal 332.

[92] For many types of agents, e.g., pedestrians, the future trajectory of any given agent could be influenced by the surrounding agents’ behaviors. In order to model such interactions among agents, the extractor 340 generates, for each temporal region proposal and from the temporal region proposal features 342 for the proposals 332, an interaction embedding 440 that represents interactions of the corresponding possible agent with the possible agents corresponding to the other temporal region proposals.

[93] To generate the interaction embedding 440, the extractor 340 represents each temporal proposal as a graph node, with the embedding of each node being the proposal features 342 for the corresponding temporal proposal.

[94] The extractor 340 computes an interaction score for each pair of two nodes in the graph from the proposal features 342 for the temporal proposals represented by the two nodes. In particular, the extractor 340 can apply a first learned function to both of the proposal features 342 to generate respective transformed proposal features for both of the nodes and then generate the interaction score by applying a second learned function to the transformed proposal features. For example, both of the learned functions can be fully- connected layers.

[95] The extractor 340 then determines the interaction embedding gi 440 for each particular node i as follows: where j ranges over all of the temporal region proposals, v _tj is the interaction score between node i and node j, ft is the embedding of node i, and y is a learned mapping function, e.g., a fully-connected layer.

[96] Returning to the example of FIG. 3, the system generates two predicted outputs for each temporal region proposal 332. In particular, the system processes the temporal region proposal features 342 for any given region using an object detection head 350 to generate an object detection output and processes the temporal region proposal features 342 for the given region using a trajectory prediction head 360 to generate a trajectory prediction output. [97] The object detection head 350 includes a first set of one or more fully connected layers that process the temporal region proposal features 342 to regress a current location in the environment of the corresponding possible agent at the end of the time interval. For example, the output of the object detection head 350 can be the regressed coordinates of a region, e.g., a bounding box, in the feature representations that represents the predicted, location of the possible agent at the end of the time interval.

[98] The object detection head 350 can also include a second set of one or more fully connected layers that generate a classification for the possible agent that includes a confidence score that represents a likelihood that the corresponding possible agent is an actual agent in the environment.

[99] Optionally, the system can use these confidence scores to perform non-max suppression on the regressed locations generated by the object detection 350 before outputting the object detection outputs, e.g., to a planning system of the autonomous vehicle.

[100] The trajectory prediction head 360 predicts the future trajectory of the possible agent. More specifically, because, as indicated above, the future trajectory of an agent could be influenced by the surrounding agents’ behavior, the trajectory prediction head 360 processes the interaction embedding for the temporal feature proposal and the temporal region proposal features for the proposal using one or more neural network layers, e.g., fully- connected layers, to regress the future trajectory output. The future trajectory output can include the x,y coordinates and, optionally, the heading of the possible agent at each of multiple future time points.

[101] During training, the system can use the object detection output, the classification output, and the trajectory prediction output to adjust the values of the parameters of the heads 350 and 360, the detector 340, and, optionally, the neural networks 330, 320, and 310 through backpropagation. In particular the system can train these components to minimize a loss that is a combination of, e.g., a weighted sum or a sum, of an object detection loss, e.g., an LI loss, a smooth L2 loss or other regression loss, a classification loss, e.g., a cross-entropy loss, and a trajectory prediction loss, e.g., an LI loss, a smooth LI loss or other regression loss using the respective ground truth outputs for each of the proposals that have a corresponding ground truth agent.

[102] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[103] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, off-the-shelf or custom-made parallel processing subsystems, e.g., a GPU or another kind of special-purpose processing subsystem. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[104] A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[105] For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[106] As used in this specification, an “engine,” or “software engine,” refers to a software implemented input/output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[107] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[108] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[109] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[110] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g., a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

[111] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. [112] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[113] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[114] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[115] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain some cases, multitasking and parallel processing may be advantageous.