SYSTEMS AND/OR METHODS FOR AUGMENTING SPORTS IMAGING

Field of the invention

Embodiments of the invention relate to systems and/or methods for augmenting sports imaging.

Background of the invention and prior art known to the Applicant

In elite sports, individuals are these days monitored on the field of play by third parties assisted by the input of 3D position tags or sensors which determine their position of play throughout the game, the distance covered, their velocities for specific phases of play and their interactions with other players and articles associated with the sport in question, such as a ball or a bat. Often these vast accumulated statistics are reviewed by statisticians and coaches over a number of days in order to determine areas of potential improvement to feed back to individuals. Traditional systems, whilst handling large volumes of data fail generally to provide real-time feedback and are of limited use in live action. Furthermore, whilst only a few athletes are afforded the privilege to take part in specific sporting events, individuals on the side lines, aspiring athletes, or other third parties are generally deprived from the opportunity to practice their skills at the very top level in their particular sport of interest. A system is therefore sought after to provide more effective training to such individuals, particularly in the areas of decision-making. Improved systems and methods are required to provide virtual real-time feedback to third parties which may be on the side lines and to train such individuals to improve their decision-making as compared to active players.

Summary of the invention

In a first broad independent aspect, the invention provides a system for augmenting sports imaging, comprising: a camera configured to capture a sequence of images of a sporting event; a user interface for viewing the sequence of images; the images having a time stamp which is recorded in a data store; an augmentation processor configured to overlay the sequence of images with a plurality of pre-determined user selectable options corresponding to pre-determined individual sporting actions; the selectable options being selectable prior to their potential occurrence; a data store for recording the or each user selected option and its time stamp; a data store for recording completed actions undertaken by individual players of the sporting event and their time stamp; and a module for comparing whether a user inputted option corresponds to an actual completed action undertaken by the individual players; and for validating that the user inputted option has an earlier time stamp than the actual completed action.

This configuration is particularly advantageous as in certain embodiments it provides realtime training for a user in elite sport decision-making. It also provides a system for validating whether or not the user correctly predicts the action undertaken by the active player in question. In addition, it allows for these interactions to be processed in ‘real- time’ or near real-time whereby individuals on the side lines of the sporting event may be tuned to the action in advance of potentially taking part in it.

In a subsidiary aspect, the invention provides a system further comprising means for capturing the interaction between a user and the interface as a combination of at least one image and its time stamp and thereafter sending the interaction to the module for comparison. In recorded sports, this approach allows the user inputs to be compared with the actual action of particular players.

In certain embodiments, this is also particularly advantageous as it allows optical character recognition to be employed to rapidly compare user inputted images with images corresponding to player actions.

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of a direction relative to a point of an individual player.

This configuration is particularly advantageous as it allows a wide variety of potential options to be presented in order to more accurately determine a correlation between predicted actions and actual completed player actions.

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of an angle relative to the lengthwise direction of a playing field.

This configuration is also particularly advantageous as it provides the reference to the playing field in question and allows a wide range of options to be presented to users of the system.

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of the magnitude of the distance of a ball following the interaction with a player.

This provides the option to a user to not only select the kind of action which an elite sports person might choose, but also to provide a selection of a magnitude of the action in question. In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of the height which may be reached by a ball following interaction with a player.

This further level of user input provides for a more precise potential determination which may be followed by an accurate validation of the user’s choice.

In a further subsidiary aspect, the camera provides a live video feed and validation arises within a period of less than 30 seconds or less than 20 seconds or less than 10 seconds.

This configuration allows for ‘real-time’ or near real-time feedback to be obtained.

In a subsidiary aspect, the system further comprises an augmented reality headset configured to project the acquired sequences of images of a sporting event and the plurality of pre-determined user selectable options onto lenses of the augmented reality headset to form a single graphical view. This allows the user to be immersed into actual live playing action with the options being presented at the same time in order to refine his/her decision-making skills and to then obtain a validation of whether or not the selected course of action actually came to pass.

In a further subsidiary aspect, the system further comprises a storage of a plurality of different overlays and wherein the user selects the overlays corresponding to a particular player of the sequences of event; whereby the selectable options correspond to sporting actions of a particular individual and whereby the selectable actions are only selectable less than 10 seconds before that individual undertakes an action.

This allows a user to efficiently input choices so that these are recorded as relevant to a particular action phase. This minimizes input errors.

In a further subsidiary aspect, the user interface incorporates a microphone for receiving audio commands and the user selectable options are selectable by voice control. This allows a user to shout or issue brief vocal commands which can further assist in training decision-making.

In a further subsidiary aspect, the camera is secured to an individual and displaced by the individual relative to the sports field. This allows for the elite sports person to provide individual insight to their perspective prior to decision-making and to further allow the training of third-party users.

In a further subsidiary aspect, the camera incorporates a housing which is equipped with a gyroscope to determine the angular position of the housing. This allow the camera to be more accurately positioned for determining certain phases of action.

In a further subsidiary aspect, the user selectable options are selectable in advance of an event during a user or operator defined period; whereby a user or operator may select a variable period of activity for the selectable options. This will allow, for example, a user to further improve their decision-making over time by progressively reducing the time period available for inputting their selection.

In a further subsidiary aspect, the module for comparing employs optical character recognition to compare the image or a part of the image at the point of user input and the image at the point of realisation of an action. This configuration provides for rapid validation of the images and for ‘real- time’ or near real-time feedback to be obtained.

In a further subsidiary aspect, the system comprises a data store for historical inputs and results; and a computer executing software for comparing the user inputs both to “real” time player actions and previous player actions associated with a matching historical input and result.

The historical inputs include in preferred embodiments, the player's position on the pitch, the position and/or the velocity of the player, the position of other players relative to the player in question who is being tracked and the position and/or the velocity of the ball or other playing object such as a wheel, bat, javelin or puck. This comparison would allow a successful historical input to be compared to the current assessed sequence of events and the pre-event user input. This would allow a user to obtain a comparison against not only the live player but a prior player; whereby the performance may be assessed against both, relative to the same or a closely matching sequence of play. The performance of the user of the system may thus be assessed relative to both in order to further improve their ability to correctly predict the action and receive a validation.

In another broad independent aspect, the invention provides a method for augmenting sports imaging; comprising the steps of: capturing a sequence of images of a sporting event; viewing the sequence of images; the images having a time stamp which is recorded in a data store; augmenting the images by overlaying the sequence of images with a plurality of predetermined user selectable options corresponding to pre-determined individual sporting actions; the selectable options being selectable prior to their potential occurrence; recording the or each user selected option and its time stamp; recording completed actions undertaken by individual players of the sporting event and their time stamp; comparing whether a user inputted option corresponds to an actual completed action undertaken by the individual players; and validating that the user inputted option has an earlier time stamp than the actual completed action.

In a subsidiary aspect, the method further comprises the step of capturing the interaction between a user and the interface as a combination of at least one image and its time stamp and thereafter sending the interaction to the module for comparison.

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of a direction relative to a point of an individual player. In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of an angle relative to the lengthwise direction of a playing field.

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of the magnitude of the distance travelled by a ball following the interaction with a player (or its bat, club and other player related article as appropriate).

In a further subsidiary aspect, the pre-determined user selectable options include a user selectable digital button representative of the height which may be reached by a ball following interaction with a player.

In a further subsidiary aspect, the method comprises the steps of providing a live video feed and validating the prediction within a period of less than 30 seconds or less than 20 seconds or less than 10 seconds.

In a further subsidiary aspect, the method further comprises the step of projecting the acquired sequences of images of a sporting event and the plurality of pre-determined user selectable options onto lenses of an augmented reality headset to form a single graphical view.

In a further subsidiary aspect, the method further comprising the step of storing a plurality of different overlays and selecting the overlays corresponding to a particular player of the sequences of event; whereby the selectable options correspond to sporting actions of a particular individual and whereby the selectable actions are only selectable less than 10 seconds before that individual undertakes an action.

In a further subsidiary aspect, the said user interface incorporates a microphone for receiving audio commands and the user selectable options are selectable by voice control. In a further subsidiary aspect, the camera is secured to an individual and displaced by the individual relative to the sports field.

In a further subsidiary aspect, the camera incorporates a housing which is equipped with a gyroscope to determine the angular position of the housing.

In a further subsidiary aspect, the user selectable options are selectable in advance of an event during a user or operator defined period; whereby a user or operator may select a variable period of activity for the selectable options.

In a further subsidiary aspect, the method further comprises the step of employing optical character recognition to compare an image or a part of the image at the point of user input and the image at the point of realisation of an action.

In a further subsidiary aspect, the method further comprises a data store for historical inputs and results; and the method includes a computer implemented method which compares the user inputs with both the “real” time player action and the previous player actions associated with a matching historical input and result.

In a further broad independent aspect, the invention provides a system for processing sporting data, comprising: a camera configured to capture a sequence of images of a sporting event; a user interface equipped with a clock for providing a time stamp and allowing a user to input from a plurality of pre-determined user selectable options corresponding to predetermined individual sporting actions; the selectable options being selectable prior to their potential occurrence; a data store for recording the or each user selected option and its time stamp; a data store for recording completed actions undertaken by individual players of the sporting event and their time stamp; and a module for comparing whether a user inputted option corresponds to an actual completed action undertaken by the individual players; and for validating that the user inputted option has an earlier time stamp than the actual completed action. This configuration is particularly advantageous as it permits in-stadium user input without necessarily providing a screen or an image overlay.

In a further broad independent aspect, the invention provides a method for processing sporting data, comprising the steps of capturing a sequence of images of a sporting event; providing a time stamp and allowing a user to input from a plurality of pre-determined user selectable options corresponding to pre-determined individual sporting actions; the selectable options being selectable prior to their potential occurrence; recording in a data store the or each user selected option and its time stamp; recording in a data store completed actions undertaken by individual players of the sporting event and their time stamp; and comparing whether a user inputted option corresponds to an actual completed action undertaken by the individual players; and validating that the user inputted option has an earlier time stamp than the actual completed action.

Brief description of the figures

Figure 1 shows a flow diagram from a live image capturing step to the result step of the method in question.

Figure 2 shows the interaction between the user end and the server end of the system.

Figure 3 shows an embodiment which employs a virtual reality headset.

Figure 4 shows an example of a user interface where the elite sport in question is football.

Figure 5 shows a user interface in schematic form where the elite sport is cricket, from the perspective of a bowler in that sport. Figure 6 shows a schematic view of a user interface where the elite sport in question is cricket and the active player is a batsman.

Figure 7 shows a further embodiment of the interaction between the user end and the serve end of the system.

Detailed description of the figures

Embodiments of the invention often handle live sporting action and in this context images are captured by one or more cameras 1. These cameras are either secured to an individual and therefore may be displaced in the field of play or are secured independently to either a fixed structure or a mobile mount whereby they can accompany the action. In addition to the provision of a camera, the images 2 may be captured at a rate sufficient to produce a video and in association with the images and with the video a particular time stamp is recorded, which may for example comprise the minutes and seconds since the start of play or since the start of a particular phase of play such as a set in tennis. The images may be broadcast to a large number of outlets and in particular to individuals. Prior to being broadcast, these may be fed into an augmentation processor 3 which overlays a number of user selectable digital buttons. These may be representative of a particular action which a player may take, such as in the context of football: ‘Shoot’, ‘Cross’, ‘Dribble’, ‘Up’, ‘Down’. The displayed images are overlaid with these options as in step 4 of the flow diagram. These overlays may be substantially transparent in order to remain visible whilst at the same time allowing the live action to be observed through the overlay. Whilst this embodiment envisages specific fields or buttons for potential actions, further embodiments may provide a joystick overlay or a virtual joystick which facilitates an input by, for example, pressing an area of a touch sensitive pad.

Once a user, as in step 5 of the flow diagram, selects one of the pre-determined individual sporting actions presented in the overlay, the image and the time stamp are sent to optical recognition 6. During the same timeframe or shortly thereafter, once the player completes the action, the completed action is compared with the user predicted action in order to determine a result 7. As observers and users accurately predict what an elite sports person chooses, a percentage of accurate predictions may also be presented. The accuracy of the predictions may also be measured in terms of points. These points may be assigned to a particular user of the system and may be used against comparable other users who have likewise interacted with the system. The ranking of users may therefore be determined based on their percentage of accurate predictions or their point totals as appropriate. As users become more familiar with the interface and improve their decision-making to match the decision-making of elite sports individuals it is likely that their accuracy of prediction will also increase which when combined with an increase in physical attributes may allow them to perform better if selected as an active player in a subsequent phase of play.

In figure 2, the user end 8 and the server end 9 are compared as these must, in preferred embodiments, ultimately interact in order to provide meaningful real-time results. At the server end 9, it is envisaged that an augmentation processor 10 is employed to provide both the live images and the specific overlays relevant to a user selected individual who is to be tracked. In order to validate the results, a data store 11 is compiled for completed player actions. These may be obtained from a third-party statistical platform or by a sports network as appropriate.

In alternative embodiments, the augmentation processor is provided in the form of an application on the user device. Furthermore, the term server end may envisage multiple servers and locations in a digital network. Whilst multiple data stores are envisaged these may be integrated into a single data store in order to further improve the synchronisation of the user inputs and the data relating to the completed action. In preferred embodiments, the results are delivered in real time or in near real time. Preferably, the results are obtained in less than 30 seconds from the completed actions, more preferably the results are delivered in less than 20 seconds from the completed actions, more preferably still the results are delivered in less than 10 seconds from the completed actions.

In addition, the user inputs may be stored in a further data store 12 for evaluation in order to develop a user’s profile. A real-time comparison module 13 will also be provided in terms of a software application, which may contrast a completed player’s action with a predicted user input and provide a correct result or an incorrect result as appropriate. For accurate determination, the comparison module may employ an optical character recognition software which may compare images to determine that the prediction of a user arose prior to the completed action having taken place. In a particular embodiment, the optical character recognition software may be adapted to primarily contrast the specific time stamp of a user input with the time stamp of the corresponding completed player action. In this manner, the comparison may be in virtual ‘real-time’ in order to provide feedback to the user prior to the next potential phase of play for the individual player being tracked by the user. The comparison of time stamps also ensures that the user input is validated as having occurred prior to the completion of the action by the player being tracked. This module may also allow individual devices to be calibrated with respect to any video lag experienced due to network restrictions which may result in a delay of between 5 and 30 seconds between real live action (as it happens in the stadium) and the broadcast of the live action.

In a further embodiment, the software may be configured to compare the user inputs not only with the actual player actions but with a selection of successful prior player actions. In this context, the server end may employ a data store 33 of historical inputs and results. The system may therefore find matches or closely correlate data according to player positions of the tracked player and other players including opponents, the velocity of the players, the direction of the players, the velocity and position of the ball or other playing object in order to determine matches of the live action with historical results. In this context, it would be possible for the user to input a better selection than the live player provided they match the successful historical option.

At the user end 8, a wide variety of input devices are envisaged, such as a virtual reality headset 14, a touch sensitive pad 15 and a smart television 16. Each one of these may facilitate a combination of the live or near live sports broadcast in combination with an overlay which allows for a user to input. User selectable options may be input by physically pressing buttons on a pad or an area representative of a button on a touch- sensitive pad. Alternatively, user-selectable options may be selected by voice command recorded by the respective built-in microphones of the devices. The devices in question may advantageously be equipped with microphones to record the commands and the appropriate software to react to specific commands. In a further embodiment, the devices may be equipped with a gyroscope or other position sensing means, whereby a user input may take the form of an action of displacing the input device itself. For example, by a user holding his/her phone in their hand and abruptly displacing their hand in a given direction. The phone thus receiving the input by motion detection and comparing the recorded motion with the player's action. The inputted motion will then be validated against the actual player's action to confirm whether or not the prediction occurred prior to the realisation of the action.

A user interface may be provided in which the users select a particular player of interest, whereby the options are provided for that particular player as and when they are involved in the sporting action. This would, for example, facilitate the tracking of a specific player in a position of interest, such as a left back in football, whereby the user of the system may him- or herself be a left back or aspiring to follow the sequences of play of that particular position. The system also envisages the possibility of tracking more than one player during a particular game, which will be possible, since the results of the comparison may be obtained in ‘real-time’ or near real-time and the individuals tend to sequentially interact in the field of play.

In a specific embodiment as shown in figure 3, the user input device may take the form of a virtual reality headset 17 which may be equipped with lenses 18 for displaying an overlay of live action and user options. Furthermore, the virtual reality headset 17 may be incorporated with an integral microphone 19 for detecting voice commands. In addition, results may also be fed through a live feed and displayed on the virtual reality headset in order to inform in real time the wearer of the headset as to whether or not their latest prediction has been correct or not as the case may be.

In figure 4, a user interface 20 such as the screen of a mobile phone is shown in schematic form where an active player 21 shown for simplicity as a cylindrical object, appears in the live video action. In addition, a further non-active player is shown as player 22. A number of user selectable digital buttons are shown, such as button 23 which is a digital representation of the available options. In one embodiment, the area of the screen of the user interface corresponding pictorially to a digital button may be suitable to record a user input by pressing the area in question. In such embodiments, the screen may be touch sensitive. In this embodiment, touch sensitive buttons are provided, which correspond to the following actions, such as ‘Shoot’, ‘Cross’, ‘Dribble’, ‘Up’ and ‘Down’. Other more complex choices of options may be presented, dependent upon the level of training which the user is interested in receiving. Once the server side comparison has been completed and the inputted choice of user has been verified against a recorded completed action corresponding to the timeframe in question, a result feed may be displayed as shown in the user display at feedback window 24. In addition, the user interface displays the time stamp 25 corresponding to the live action in the form of minutes and seconds of play. In certain sports where these are not displayed, the processor itself may employ its internal clock to provide an indication of the time of input of a user’s selection.

Whilst the invention will have specific advantages for certain sports, it may be applicable to a wide variety of these, for example, it may be applicable to the field of cricket, as shown in figure 5 and figure 6 where the individual user selectable options are provided when, for example, in the case of figure 5 when the active player 27 is a bowler and in the case of figure 6 where the active player is batsman. In the context of the bowling options 28 of figure 5, a user may select a particular form of pitch or delivery in the timeframe leading up to the release of the ball by the active player. For example, as the player is running up to the wicket, the user may select a particular kind of delivery, such as the ‘Yorker’ and thereafter if the delivery corresponds to a ‘Yorker’, the system would deliver the real-time result validating that the prediction was correct.

In the case of figure 6, when the active player is a batsman, the user input interface may be displayed to provide batting options 29 in the form of a wheel 30 with specific segments, which a user may interact with to input their selection of stroke, shot or strike. For example, once the ball is delivered by the bowler, a brief window of opportunity for selection of the option arises whereby the user may for example select a straight drive and thereafter the system will compare whether or not the correct stroke selection was entered and deliver the result in real-time.

In certain sports an important variable is whether or not the correct power is selected in order to obtain an advantage over a particular opponent. For example, in tennis, in certain phases it is beneficial to play a low power stroke. In this context, a power input option may be provided, as power selection 31, to assist in the development of the appropriate decision-making as to when to opt for a low power stroke as opposed to a high power stroke. A power level may be displayed pictorially as the length of a power bar 32. A user may input the height of the power bar by swiping an area of the touch sensitive screen to a greater or lesser extent.

As shown in figure 6, a power selection option 31 may therefore be envisaged in addition to the other options already discussed.

In a further embodiment, the sport may be a motor sport such as Formula 1 (registered trademark) or Motor GP (registered trademark). In these applications, the image overlay may include options such as “overtake”, “pit”, “accelerate”, “brake”. In a particular embodiment, the display may include a wheel and the precise timing of the rotation of the wheel may be assessed together with the angle of rotation to determine whether or not the user is reacting in a similar manner to the actual driver. Whilst the preceding description refers at times to a player, this term may also be substituted by the term driver if appropriate.

Whilst certain preceding embodiments have discussed the process involved in reaching the results for the comparison between the user input and the completed player action, further embodiments may provide the results from not only one user but from multiple users and provide a ranking of accuracy of prediction of one user relative to at least one other user. A feed of multiple user results may therefore be simultaneously displayed on a user's interface as appropriate. In figure 7, the user end 34 and the server end 35 are compared as these must ultimately interact in order to provide meaningful real-time results. At the server end 35, it is envisaged that an augmentation processor 36 is employed to provide both the live images and specific user options relevant to a user selected individual who is to be tracked. In order to validate the results, a data store 37 is compiled for completed player actions. These may be obtained from a third-party statistical platform or by a sports network as appropriate. A data store 38 for user inputs may be provided. Optionally, a data store 33 (cf. figure 2 or 39 in figure 7) of historical inputs and results may also be provided. A “real time” comparison module 40 may be provided to execute computer executable software to determine whether the selection was correct or to determine whether the selection was closer to likely successful historical actions and results.

At the user end, a joystick 41 may be provided or a virtual joystick such as provided by an application on a smart phone 42, whereby a user may input the selection whilst viewing live action prior to the player completing an action. Thus, not necessarily providing a viewing screen for displaying the sporting action to the user. This could allow the system to be used in-stadium. One of the primary input from the user end will however remain the timing of their input which must by definition arise before the occurrence of the action. The action in this embodiment being associated precisely with the actual timing of the action. The validation would therefore take into account the stadium clock and the timing of the input by the user.