CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 62/266,817 filed December 14, 2015, which is incorporated by reference.

BACKGROUND OF THE INVENTION

[0002] Recent years have seen the development and deployment of commercial sports tracking systems for tracking the movement of players, balls, or other objects on a sports playing field. These tracking systems vary in their operation, and include purely optically-based systems (e.g., using multiple cameras), radio-based systems (e.g., using RFID tags embedded in player equipment), satellite-based systems (e.g., GPS) and hybrid systems. Generally, regardless of the type of tracking system employed, the output of such a system includes the (x, y) location of players, recorded at a high-frame rate. In this manner, the players' behavior has been essentially "digitized" allowing individual game plays to be visualized via multi-agent trajectories.

Although this behavior can be displayed graphically, describing the subtle movement of players via tags or text labels requires an enormous amount of labels and effort (i.e., a picture is worth a thousand words). Moreover, the usefulness of such a system is limited if there is not an ability to store, catalog and retrieve individual game sequences in an efficient manner.

BRIEF SUMMARY OF THE INVENTION

[0003] A system is provided for interactive analysis of sports games using gathered trajectory information. The system processes sequences (e.g., "plays") of a game from gathered tracking data in an efficient manner that permits a user to query a database of plays using a graphical representation of the raw trajectories and to interactively find plays that are similar. A user can use selected "exemplar" plays, or user-drawn plays on an interface. [0004] The system also permits interactive statistical analysis by the user based on a graphical representation of the game players and trajectories. For example, the system allows a user to specify a current play-of-interest (such as by selecting the play from a list of exemplar plays, or by manipulating graphical objects on a screen to represent the play) as a query to the database of plays. Using statistical information associated with the plays in the database, the system can present a statistical probability for a particular event occurring in the queried state. With respect to FIGURE 9a, a basketball game state is shown with ten players positioned on the court, five on offense and five on defense. The system queries the database using the game situation (or a subset of the elements of the game situation), and displays a statistical probability of a successful outcome for players in that situation (e.g., 34% for Brown), based on similarly situated plays in the database that were retrieved in response to the query. Other types of statistical event probabilities (e.g., pass probability, foul probability, turnover probability, etc.) can also be estimated and presented based on the query.

[0005] The system also permits interactive analysis by tweaking or modifying the queried play to ask "what if... ?" types of questions. For example, with respect to FIGURE 9b, the user has exchanged one player (Brown) for another (Tucker) in the queried situation. The system updates the event probability accordingly, allowing for an efficient statistical comparison of how different players may affect a particular given game situation. Additionally, an embodiment further includes an interface permitting moving the location of one or more players, as shown in FIGURE 9d (Gortat being moved to a new location). Again, the statistical probability

information is updated based on the modified situation. More generally, the system permits the identification and analysis of plays that may be most effective against a particular opponent.

[0006] To achieve these results, an embodiment of the system includes three phases of operation: a) alignment of trajectories using multiple templates; b) discovery of a "playbook" (i.e., hash table) of plays directly from multi-agent trajectory and event data in an unsupervised manner; and c) using the playbook to obtain player and context-specific statistical information in response to input queries. Each of these phases is an improvement over existing systems, such as what is described in U.S. Patent Pub. No. 2016/0260015, to Lucey et al., which is incorporated by reference for all that it teaches. In an embodiment, the present system's use of multiple templates yields significantly improved alignment through "ensemble alignment" or "aligning and clustering." An embodiment also uses both player and ball trajectories, as well as event information, to construct the hash table of plays by using both a decision-tree framework using aligned data as well as a top-down hierarchical model which comprises pruning insignificant or non-predictive trajectories from plays.

[0007] Although the examples described herein relate specifically to the sports of basketball and soccer, the system is not limited to any particular sport, and can be applied to any sport or domain with fine-grain trajectory data (whether it be from optical tracking data (e.g., Sport VU) or wearable devices (e.g., RFID, GPS) or any other type of input (e.g., hand-drawn, annotated)).

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIGURE 1 illustrates an architectural environment for a system for interactive sports analytics, according to an embodiment of the present system;

[0009] FIGURE 2 is a diagram illustrating a general method for performing interactive sports analytics, according to an embodiment of the present system;

[0010] FIGURE 3 is a graphic diagram illustrating the use of alignment of positional sports player data, in accordance with an embodiment of the present system;

[0011] FIGURE 4 is a flow diagram illustrating a method of aligning sports plays using multiple templates, in accordance with an embodiment of the present system;

[0012] FIGURE 5 is a flow diagram illustrating a top down method of learning a decision- tree for sports plays, in accordance with an embodiment of the present system;

[0013] FIGURE 6 is an illustration of an application of a top down method of learning a decision-tree for sports plays, in accordance with an embodiment of the present system;

[0014] FIGURE 7 is a diagram illustrating an example application of an interactive sports analytics system, in accordance with an embodiment of the present system;

[0015] FIGURE 8 is an illustration of an interactive sports analytics interface, in accordance with an embodiment of the present system;

[0016] FIGURES 9a - 9d are illustrations of an interactive sports analytics interface, in accordance with an embodiment of the present system; and

[0017] FIGURES 10a - 10b are illustrations of an interactive sports analytics interface, in accordance with an embodiment of the present system. DETAILED DESCRIPTION OF THE INVENTION

[0018] Embodiments of the present system process large amounts of sports-related tracking data in an efficient manner, enabling the querying and retrieval of statistically similar sports plays and the generation of analytical statistical predictions for player and team behavior through an interactive visual interface.

[0019] A general overview of the context of the system is described with respect to FIGURE 1, in accordance with an embodiment. At a sporting event taking place at a venue 110, a tracking system 120 records the motions of all players on the playing surface, as well as any other objects of relevance (e.g., the ball, the referees, etc.). Tracking system 120 can be an optically-based system using, for example, a plurality of fixed cameras. Alternatively, tracking system 120 can be a radio-based system using, for example, RFID tags worn by players or embedded in objects to be tracked, or tracking system 120 can be another type of system that tracks moving objects. Preferably, tracking system 120 samples and records at a high frame rate (e.g., 25 frames per second) so as to minimize quantization, enabling expert humans to select the onset and offset of plays at precise times (i.e., frame-level), as well as particular players of interest. Tracking system 120 stores at least player identity and positional information (e.g., (x,y) position) for all players and objects on the playing surface for each frame in a game file 140. The game file 140 is preferably augmented with other event information corresponding to the frames, such as game event information (pass, made shot, turnover, etc.) and context information (current score, time remaining, etc.), and assembled into data store 150 comprising a large number of game files for the given sport (e.g., an entire season of the National Basketball Association games).

[0020] A preprocessing engine 160 processes the raw data from the data store 150 through multiple-template alignment and discriminative clustering, in accordance with embodiments described herein, and stores the results in a play database 170. A play database server 180 processes queries to the play database 160. A computing device 190 runs an interactive sports analytics interface and is communicatively connected to the play database server 180. Using the interactive sports analytics interface, a user can submit graphical representations of plays as queries to the play database server 180 and obtain results from the play database 170 that are situationally similar to the queried play, along with statistical information. The user can tweak or modify the query and obtain updated statistical results.

[0021] Turning to FIGURE 2, the general process of pre-processing and retrieving plays from a play database is shown, in accordance with an embodiment. In a pre-processing phase 210, the raw positional data 220 is aligned and clustered using multiple templates at a step 230. At a pre-processing step 240, the multiply-aligned plays are discriminatively clustered using a top-down hierarchical approach to form a hash table / decision tree that acts as a playbook, cataloging the plays for efficient retrieval by similarity and various classification metrics. Once the database has been pre-processed, an input query 250 to the database, consisting of a graphical representation of a play and other possible constraints or metrics, can be processed during a retrieval phase 255. The retrieval process begins by computing a hash value for the play at a step 260, and navigating the hash table / decision tree to fetch similar candidates at step 270. The candidates are ranked at step 280 and the top candidates are returned as the result at step 290.

[0022] Trajectory Alignment using Multiple Templates

[0023] In an embodiment, the alignment of plays is performed using multiple templates. An example of misalignment of tracking data is illustrated in FIGURE 3. FIGURE 3a indicates the locations of five basketball player positions in one quarter of a game. There is little

distinctiveness to any of the players, particularly on the offensive end of the court. FIGURE 3b illustrates player positions after aligning the plays based on their role (point guard, shooting guard, center, etc.) However, because the plays have been aligned to only single template, there is little variation shown between offensive and defensive positioning for each player role.

FIGURE 3c illustrates player positions after aligning plays with multiple-templates using a tree- based representation in accordance with an embodiment of the present system, showing greater definition and distinctiveness between a player's positioning on offense (left half) and defense (right half).

[0024] In an embodiment, the present system performs alignment using multiple templates. Given M agents (players, ball, and/or other objects to be tracked), and their two-dimensional continuous raw positions, the dataset of multi-agent behavior D consisting of length F frames is represented as a concatenated sequence of (x, y) points: where denotes the two-dimensional coordinates of the j agent at the i time instance and X _j is the representation of all M agents for the j frame.

[0025] Spatial alignment is performed by finding a set of permutation matrices with the objective of maximizing the similarity of the data. That is, a set of M permutation matrices, Φ = {Pi, . . . , P _M } is constructed such that the total similarity is maximized (or the total entropy is minimized). Given that the similarity between two frames of data can be measured as the negative Euclidean distance— 11 Xi— X _k 112, the objective is to maximize the following

[0026] The multiple template approach of the present system improves the alignment, maximizing the similarity of the data (or minimizing the reconstruction error when using the learned templates). In addition, the benefit of discovering multiple templates permits higher- level features or latent factors that can be used to personalize queries by matching specific contexts and conditions.

[0027] Turning to FIGURE 4, a method is described of finding multiple templates directly from raw trajectory data, in accordance with an embodiment. At step 410, a single exemplar play of interest is selected as the initial template. The initial template can be a random sample, or found using previous single template methods, such as those described in Lucey et al., "Representing and Discovering Adversarial Team Behaviors using Player Roles," Computer Vision and Pattern Recognition (2013), which is incorporated by reference for all that it teaches.

[0028] At step 420, all plays in the database are aligned to the initial template by calculating the cost matrix, which consists of finding the distance (such as L2 distance) between each trajectory in the template and each trajectory in the candidate play. The permutation matrix is calculated using known techniques (for example, the Hungarian algorithm), and the candidate play is accordingly permuted to align it to the template. [0029] At step 430, a value of K is chosen and a clustering algorithm (e.g., K-means, agglomerative clustering, affinity propagation) is used to assign each play of the database to one of K plays. The total reconstruction error is measured for the K clusters.

[0030] At step 440, the total reconstruction error for K is compared to a desired threshold value. If the total reconstruction error for K is less than the threshold, the K plays are used as the multiple templates, and the process terminates. Otherwise, at step 450, a new value for K is chosen, and the clustering algorithm is run again at step 430. Alternatively, a different threshold function is used, such as if K exceeds some number. Alternatively, the total reconstruction error for each iteration is compared, and the value of K yielding the minimal reconstruction error after some time period or number of iterations is selected.

[0031] Another alternative, as used in an embodiment, is a matching-pursuit type approach to find a suitable set of K templates. Beginning with K=l, the exemplar which can best represent the data is found. This exemplar is added it to the dictionary of exemplars, and K is incremented to K=2 to find the next best exemplar to represent the data. This process continues until some desired criterion is met.

[0032] The method of finding multiple templates described above with respect to FIGURE 4 has advantages over methods, such as hand-crafting features by an expert which encodes and discriminates the semantics of the behavior of interest (for example, if an expert were to code a "pick-and-roll" play in basketball). The expert-selection method is problematic as it is lossy (it only selects a few agents of interest), and creates an over-complete feature representation which would need to be reduced. In contrast, the method of FIGURE 4 corresponds to merely an added column in the database which corresponds to the best permutation, making clustering, retrieving and short-term prediction easy and interpretable.

[0033] Hash-Table/Playbook Learning

[0034] For retrieval tasks using large amounts of data, an embodiment of the system uses a hash-table is required by grouping similar plays together, such that when a query is made, only the "most-likely" candidates are retrieved. Comparisons can then be made locally amongst the candidates and each play in these groups are ranked in order of most similar. Previous systems attempted clustering plays into similar groups by using only one attribute, such as the trajectory of the ball. However, the semantics of a play are more accurately captured by using additional information, such as information about the players (e.g., identity, trajectory, etc.) and events (pass, dribble, shot, etc.), as well as contextual information (e.g., if team is winning or losing, how much time remaining, etc.). Thus, embodiments of the present system utilize information regarding the trajectories of the ball and the players, as well as game events and contexts, to create a hash-table, effectively learning a "playbook" of representative plays for a team or player's behavior. The playbook is learned by choosing a classification metric that is indicative of interesting or discriminative plays. Suitable classification metrics may include predicting the probability of scoring in soccer or basketball (e.g., expected point value ("EPV"), or expected goal value ("EGV"), as described in Miller et al. ("Factorized Point Process Intensities: A Spatial Analysis of Professional Basketball," in ICML, 2014) and Lucey, et al. ("Quality vs quantity": Improved shot prediction in soccer using strategic features from spatiotemporal data," in MIT Sloan Sports Analytics Conference, 2015), which are hereby incorporated by reference. Other predicted values can also be chosen for performance variables, such as probability of making a pass, probability of shooting, probability of moving in a certain direction/trajectory, or the probability of fatigue/injury of a player.

[0035] The classification metric is used to learn a decision-tree, which is a coarse-to-fine hierarchical method, where at each node a question is posed which splits the data into groups. A benefit of this approach is that it can be interpretable and is multi-layered, which can act as "latent factors."

[0036] Bottom-Up Approach

[0037] In an embodiment of the system, a bottom-up approach to learning the decision tree is used. Various features are used in succession to discriminate between plays (e.g., first use the ball, then the player who is closest to the ball, then the defender etc.). By aligning the

trajectories, there is a point of reference for trajectories relative to their current position. This permits more specific questions while remaining general (e.g., if a player is in the role of "point guard", what is the distance from his/her teammate in the role of "shooting guard", as well as the distance from the defender in the role of "point guard"). Using this approach avoids the need to exhaustively check all distances, which is enormous for both basketball and soccer.

[0038] Top-Down Approach [0039] In another embodiment of the system, a top-down approach to learning the decision tree is used. An example of the top-down approach is described with respect to FIGURE 5. At step 510, all the plays are aligned to the set of templates. From this initial set of templates, at step 520 the plays are assigned to a set of K groups (clusters), using all ball and player information, forming Layer 1 of the decision tree. Back propagation is then used at step 530 to prune out unimportant players and divide each cluster into sub-clusters (Layer 2). The approach continues at step 540 until the leaves of the tree represent a dictionary of plays which are predictive of a particular task - i.e., goal-scoring (Layer 3).

[0040] An example of applying the top-down approach is shown in FIGURE 6. Beginning with one exemplar template (Layer 1), all basketball plays are first aligned to the exemplar and clustered into K groups using all player and ball information, shown in Layer 2. Back- propogation is then used to prune out unimportant players and then to divide each cluster into sub-clusters, shown in Layer 3. This approach is continued until there is a "dictionary" of plays which are sufficiently predictive of a particular task, such as field goal-scoring, shown in Layer 4.

[0041] Personalization using Latent Factor Models

[0042] In addition to raw trajectory information, in embodiments of the system, the plays in the database are also associated with game event information and context information. The game events and contexts in the database for a play may be inferred directly from the raw positional tracking data (e.g., a made or missed basket), or may be manually entered. Role information for players (e.g., point guard, shooting guard, center) can also be either inferred from the positional tracking data or entered separately. In embodiments of the system, a model for the database can then be trained by crafting features which encode game specific information based on the positional and game data (e.g., distance from basket/goal, distance from defenders, particular events, etc.), and then calculating a prediction value (between 0 and 1) with respect to a classification metric (e.g., expected point value).

[0043] If there are a sufficient number of examples, the database model can be personalized for a particular player or game situation using those examples. In practice, however, a specific player or game situation may not be adequately represented by plays in the database. Thus, embodiments of the system find examples which are similar to the situation of interest - whether that be finding players who have similar characteristics or teams who play in a similar manner. A more general representation of a player and/or team is used, whereby instead of using the explicit team identity (i.e., James as a player, or Manchester United as a team), each player or team is represented as a distribution of specific attributes, in a manner such as described by Yue, et al. ("Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction," in ICDM, 2014), Miller et al. ("Factorized Point Process Intensities: A Spatial Analysis of Professional Basketball," in ICML, 2014) and Wei et al. ("Predicting serves in tennis using style priors," in KDD, 2015), which are hereby incorporated by reference.

[0044] Embodiments of the system use the plays in the hash-table/playbook that were learned through the distributive clustering processes described above. As an example, FIGURE 7 shows the results of one implementation where the game-context for a particular soccer team has been used with a database of soccer plays in accordance with an embodiment. In the example of FIGURE 7, the database of plays was aligned to multiple templates and distributively clustered on criteria including game-context (i.e., whether the team was winning, losing, or tied). Using the classifying metric of expected goal value, different goal scoring plays are retrieved from the database depending on the game-context. That is, when winning, the most likely shooting method is a counter-attack. When losing, the most likely shooting method is on a slower attack. The mean EGV for these situations is shown to be 14% when winning, 8% when losing, and 10% when tied. This information is useful for determining a team's strategy, and also allows performing more accurate match prediction generally by modeling a team's behavior for different match context depending on the strength/weaknesses of their attributes.

[0045] Turning to FIGURE 8, an example user interface 900 is shown for use in accordance with embodiments of the system to perform interactive analytics for a basketball game. A first interactive panel 910 displays the current play at issue, including graphical depictions of the players (circles) and their recent trajectories (tails), as well as the ball. Sliders 920 permit the user to select the play length window to be analyzed, and to see a timeline for the entire length of the play. The play is initially either loaded from the database (for example, as a result of a user search through a search panel 930 and/or a Game Select button 940), or drawn by the user using a touchscreen, stylus, keyboard or other input device (for example, after a user's selection of a Draw Play option 950). Once entered or selected, the interface uses this play as an input query to the database and retrieves similar plays to display in a retrieved play panel 960. In an embodiment, an additional panel on the interface is used to display actual video footage from the archived game, corresponding to a user's selection of one of the retrieved plays, synchronized to the graphical depiction. An additional player selection panel 970 is presented to allow the user to assign or manipulate team players to the graphical depictions in the first interactive panel 910.

[0046] Turning to FIGURE 9a, another example user interface is shown for use in interactive analysis of a basketball game, in accordance with an embodiment. In this example, the positions of the ten players are shown graphically, and identified by the players' names. Additionally, the offensive team players display the EPV given the current game configuration. Using the interactive panel, the user in this example has selected one player, "Brown," whose EPV in the situation is 0.34. In FIGURE 9b, the user selects one of the players, "Tucker", from a player selection panel, causing the interactive display panel to update using statistical information retrieved for similar plays from the database. In particular, the system updates the EPV for Tucker, to 0.28, indicating that Tucker is not expected to perform as well offensively in this situation as compared to Brown. In FIGURE 9c, the user has swapped Tucker with "Marshall" using the player selection panel, and Marshall's updated EPV of 0.44 is presented, based on the statistical information retrieved for similar plays from the database. Turning to FIGURE 9d, using the interface, the user has moved the position of the graphical depiction of one player, "Gortat," and the player's corresponding EPV is updated by the system from 0.40 to 0.34, indicating that Gortat is less likely to be offensively productive in this proposed position.

Alternatively, in addition to swapping a player, embodiments of the system calculate statistical values based on the swapping of the team surrounding the player. For example, if the same player now plays for a different team, the system can calculate but the team was different, how does the percentage change? This is important when we incorporate styles. As the most important part of the patent is the "swapping players" and seeing the change in probabilities, I think this is important.

[0047] Turning to FIGURE 10a, another example user interface 1100 is shown of use in interactive analysis of a basketball game, in accordance with an embodiment. A set of advanced searching options 1110 are presented, allowing the user to restrict the database search to particular subsets of plays. In this example, the user has restricted the "Green Team" (on offense here) to be the Warriors, while the "Blue Team" (on defense here) is unrestricted. Other restrictions might be based on the game context (e.g., less than 2 minutes left in game, first 5 minutes of game, time of day (day game vs. night game), particular player on the court, etc.). The retrieved similar plays 1115 from the database query are displayed on the right. An

Outcome Analysis section 1120 for the query displays statistical results for the retrieved plays, i.e., out of 19 results, 14 led to a shot, 2 to a turnover, and 3 to a foul. A Retrieval History section 1130 of the display indicates the statistical results from past queries of the situation. In this example, a previous result for limiting the query to the Cavaliers is shown (17 results), along with an unrestricted query of all similar plays (465 results).

[0048] In FIGURE 10b, the user has used the interface 1100 to manipulate the trajectory of the ball 1140 using a Draw option 1150. Whereas the original play had indicated a pass from Green Player 4 to Player 3 to Player 2, the new proposed play indicates a pass from Green Player 4 to Player 5. The interface queries the database accordingly to retrieve similar plays 1160, displayed on the right. Again, the Outcome Analysis section 1120 is updated according to the query results and can be compared to the Retrieval History section 1130, indicating that the proposed play increases the likelihood of taking a shot (from 71% to 73%) and the likelihood of being fouled (from 14% to 16%).

[0049] In addition to the examples of basketball and soccer described throughout this disclosure, embodiments of the system are not limited to these particular sports, and the system is suitable for use in a variety of other sports, including but not limited to, for example, rugby, volleyball and American football.

[0050] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0051] The use of the terms "a" and "an" and "the" and "at least one" and similar referents in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term "at least one" followed by a list of one or more items (for example, "at least one of A and B") is to be construed to mean one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to,") unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is

incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0052] Preferred embodiments of this invention are described herein, including the best mode known to the inventors for carrying out the invention. Variations of those preferred embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.