Field of the Invention

[1] This invention relates generally to autonomous computer scoring match tracking systems. More particularly, this invention relates to an autonomous speech signal recognition match scoring system.

Background of the Invention

[2] Scoring of sporting matches typically entails manual score keeping by an off-field scorer. However, scorers are not always available for all sporting events, especially nonprofessional sporting events.

[3] Whereas the umpire/referee of the sporting match may keep score, such is undesirous in that the umpire/referee generally cannot be distracted such as by take eyes off gameplay for input of data into a portable computing device, for example.

[4] In certain sporting events, such as Australian football league (AFL) and cricket, the umpire/referee may issue hand score signals but which again require interpretation by an off-field scorer.

[5] The present invention seeks to provide a scoring system, which will overcome or substantially ameliorate at least some of the deficiencies of the prior art, or to at least provide an alternative.

[6] It is to be understood that, if any prior art information is referred to herein, such reference does not constitute an admission that the information forms part of the common general knowledge in the art, in Australia or any other country.

Summary of the Disclosure

[7] There is provided herein a sports match scoring system for semiautonomous scoring of matches that uses speech signal recognition in a ways that allows the umpire/referee to score the match without being unduly distracted such as by having to take eyes off gameplay.

[8] In an embodiment, the system comprises an in-field device comprising an audio input which may receive and digitise umpire/referee speech signals and wirelessly transmit the speech signals using a wireless transmitter. The system may further comprise an off-field device comprising a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device.

[9] The off-field device comprises a signal acquisition controller for receiving and digitally sampling the speech signals and signal pre-processor and feature extraction controllers for preprocessing the signals and extracting features therefrom.

[10] The off-field device further comprises an automatic speech recognition (AS ) decoder for classifying patterns of the features as one or more spoken signals. [11] As such, the off-field device further comprises a scoring controller configured for calculating and updating an in-memory match score according to the one or more spoken signals classified by the ASR decoder.

[12] In embodiments, the system is configured for enhancing the classification accuracy of the ASR decoder.

[13] For example, in one embodiment, the off-field device may comprise a data dictionary of a finite of applicable spoken commands which, in embodiments, may be game type specific. As such, when classifying the spoken signals, the ASR decoder may classify the signals according to the finite set of available spoken commands from the dictionary, thereby enhancing the classification accuracy thereof.

[14] In further in embodiments, the off-field device may comprise a decision tree comprising a plurality of interconnected gameplay progress data nodes which may be used by the system to track gameplay progress in accordance with gameplay rules and events.

[15] The classification accuracy of the ASR decoder may be further enhanced with reference to the gameplay rules decision tree to filter out signal classifications which would be inapplicable according to the current gameplay progress.

[16] In further embodiments, the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary sensor data. For example, in one embodiment, the system may comprise an audio sensor and an associated audio analysis controller configured for detecting gameplay events according to audio signal patterns and updating the gameplay progress within the decision tree.

[17] In further embodiment, the classification accuracy of the ASR decoder and/or the scoring of the scoring controller may be further enhanced with auxiliary vision data (which may include charged couples device (CCD) image sensors, Light Detection and Ranging (LIDAR) sensing sensors and the like) wherein the system comprises a video analysis controller for processing vision data received therefrom for detecting certain match events and updating the gameplay progress within the decision tree accordingly.

[18] In embodiments, the wireless communication between the in-field device and the off-field device may be bidirectional wherein the off-field device is able to transmit data indicative of the correct recognition or the inability to correctly recognise spoken signals which may be conveyed to the umpire/referee by way of a feedback device such as a haptic feedback device.

[19] In further embodiments, the in-field device may comprise a user interface, such as a pushbutton interface so as to allow the umpire/referee to use pushbutton depresses for common scoring signals, such as "dot ball" signals in cricket. In embodiments, the in-field device may comprise an associated wirelessly connected handheld device having the pushbutton user interface thereon for the more convenient provision of these pushbutton inputs. In embodiments, the handheld device may comprise a pushbutton input for push-to-talk functionality.

[20] In further embodiments, the in-field device may comprise an accelerometer and an associated motion controller configured classifying hand signal gestures of the umpire/referee. The accelerometer may be either inbuilt within the in-field device or take the form of an electronic wristband in wireless communication with the in-field device so as to be able to detect hand motions of the umpire/referee, such as the "wide" and "out" hand signals of the umpire/referee in cricket.

[21] In accordance with this embodiment, the in-field device is further configured for transmitting acceleration data to the off-field device which uses the acceleration data for classifying the acceleration data according to a plurality of acceleration classification patents to identify hand gestures.

[22] In embodiments, the hand gestures may be used to further enhance the classification accuracy of the AS decoder.

[23] In accordance with one aspect, there is provided a speech signal recognition match scoring system comprising: an in-field device comprising: an audio input; a wireless transmitter for transmitting the speech signals received from the audio input; an off-field device comprising: a processor for processing digital data; a wireless receiver for receiving the speech signals from the wireless transmitter of the in-field device wireless receiver in operable communication with the processor across a system bus; a memory device for storing digital data, including computer program code instruction controllers and associated data, the memory device being in operable communication with the processor across the system bus wherein the controllers comprise: a signal acquisition controller for receiving digitised speech signals via the wireless receiver; signal preprocessor and feature extraction controllers for pre-processing the speech signals and extracting futures therefrom; an automated speech recognition (ASR) decoder for classifying patterns of the features as one or more spoken signals; and the scoring controller for calculating and updating an in- memory score according to the one or more spoken signals.

[24] The data may comprise a signal command dictionary comprising a finite set of signal commands and the ASR decoder may be configured for classifying the spoken signals according to the finite set of signal commands.

[25] The data may comprise a gameplay progress decision tree and the scoring controller may be configured for updating gameplay progress according to gameplay progress data nodes of the decision tree. [26] The off-field device AS decoder may be further configured for classifying the one or more spoken signals according to a current node position of decision tree.

[27] In use, the off-field device may be configured for calculating applicable spoken signals according to a current node position of the gameplay decision tree.

[28] The in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device and the off-field device may be configured for transmitting data indicative of a spoken signal being inapplicable to the current node position to the in-field device.

[29] The in-field device may further comprise a wireless receiver and the off-field device may further comprise a wireless transmitter configured for transmitting data from the off-field device to the in-field device.

[30] In use, the off-field device may be configured for transmitting feedback data indicative of at least one of successful classification of a speech signal and the unsuccessful classification of a speech signal.

[31] The in-field device may comprise a feedback device and the feedback device may be configured for providing feedback when receiving the feedback data.

[32] The in-field device may further comprise a user interface and the in-field device transmitter may be configured for transmitting user interface data to the off-field device receiver and the scoring controller may be configured for calculating and updating the in-memory score further in accordance with the user interface data.

[33] The system may further comprise an in-field accelerometer and the in-field device transmitter may be configured for transmitting acceleration data to the off-field device receiver.

[34] The off-field device may comprise a motion analyser configured for template-based classifying the acceleration data according to at least one gesture pattern.

[35] The ASR decoder may be configured for further classifying the one or more spoken signals in accordance with the at least one gesture pattern.

[36] The system may further comprise a wrist wearable comprising the accelerometer, the wrist wearable being in wireless short range communication with the in-field device.

[37] The system may further comprise a handheld device in operable communication with the infield device and the handheld device may comprise a user interface and the handheld device may be configured for transmitting user interface data to the in-field device.

[38] The handheld device may be in wireless communication with the in-field device.

[39] Other aspects of the invention are also disclosed. Brief Description of the Drawings

[40] Notwithstanding any other forms which may fall within the scope of the present invention, preferred embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:

[41] Figure 1 shows a speech signal recognition match scoring system in accordance with an embodiment.

Description of Embodiments

[42] Figure 1 shows a speech recognition scoring system 100 in accordance with an embodiment. The system 100 comprises an in-field device 102 comprising an audio input 110. In embodiments, the in-field device 102 may comprise an analogue-to-digital (A/D) converter 111 for digitising speech signals received from the audio input 110. The in-field device is used by the match umpire/referee within the sporting field/ring.

[43] The in-field device 102 further comprises a wireless transmitter 108 for transmitting the speech signals.

[44] The system 100 further comprises an off-field device comprising a processor 112 for processing digital data. The off-field device 100 may further comprises a wireless receiver 109 for receiving speech signals from the wireless transmitter 108 of the in-field device 102. The wireless receiver 109 is in operable communication with the processor 112 across a system bus 107.

[45] The off-field device 101 further comprises a memory device 106 for storing digital data, including computer program code instruction controllers and associated data. The memory device 106 is similarly in operable communication with the processor 112 across a system bus 107.

[46] The controllers comprise a signal acquisition controller 115 for receiving speech signals from the wireless receiver 109. For example, the A/D converter 112 may sample speech signals from the audio input 110 at 8 kHz with 12 bits resolution. An analogue filter may be utilised between the audio input 110 and the A/D converter 111.

[47] In embodiments, the transmitter 108 and the receiver 109 digitally transmit the digitised speech signals from the field. Alternatively, the in-field device 112 may transmit analogue signals to the off-field device 101 wherein sampling is performed by the off-field device 101.

[48] Whereas in embodiments, the transmitter 108 and the receiver 109 may be relatively short range of Bluetooth transmitters/receivers, in other embodiments, the transmitter 108 and receiver 109 are preferably at least intermediate range transmitters and receivers so as, for example, to be able to transmit the speech signals over more than 100 m in certain applications such as for cricket matches. [49] The off-field device 101 controllers may further comprise a signal pre-processor controller 116 and a feature extraction controller 117 for pre-processing the speech signals and extracting features therefrom.

[50] The pre-processor controller 116 may detect the beginning of a speech signal and remove DC harmonics, such as filtering 50 Hz soundcard interference ripples. The pre-processing controller 116 may further shift and normalise the speech signals.

[51] The feature extraction controller 117 may calculate a Fast Fourier Transform (FFT) of the signal an extract FFT coefficients between 100 Hz and 500 Hz for example.

[52] The controllers may further comprise an automated speech recognition (AS ) decoder 118 for classifying the features as one or more spoken signals. The ASR decoder 118 may implement a multilayer perceptron for pattern classification.

[53] The output of the ASR decoder 118 are recognised spoken signals.

[54] In embodiments, the functionality of at least one of the signal acquisition, preprocessor feature extraction and ASR decoding controllers 115 - 118 may be implemented by a commercially available speech recognition API such as the Google™ Speech API, Speechmatics™ ASR and the like.

[55] The controllers may further comprise a scoring controller 119 configured for calculating and updating an in-memory score 120 according to the one or more spoken words recognised by the ASR decoder 118.

[56] In embodiments, the off-field device 101 comprises a video output 131 for outputting a graphical representation of the calculated score on a digital display 128. Furthermore, the off-field device 101 may comprise a network interface card 130 for transmitting the score data across a data network 127, such as the Internet.

[57] In embodiments, the in-field device 102 is similarly a digital device comprising a processor 112 and a memory device 106 operably coupled thereto across a system bus 107, the memory device 106 similarly having computer program code instruction controllers therein for configuring the functionality of the digital in-field device 102. The audio device 110 may be a suitably located microphone either inbuilt or connected via an electrical lead to the lapel or the like of the referee so as to be suitably located for receiving spoken commands.

[58] Typically, the in-field device 102 may be a small form factor portable battery-powered electronic device.

[59] In embodiments, the ASR decoder 118 classifies spoken signals according to a finite set of signal commands from a data dictionary 124 in memory 106.

[60] Furthermore, the dictionary 124 may be further configured with a finite set of signal commands particular to a sports type. For example, for cricket, the dictionary 124 may be configured with a finite set of in play signal commands such as "dot ball", "out", "no-ball", "free hit", "wide", "dead ball" and the like and scoring signals such as "1 Run", "2 Runs", "3 Runs", "Four", "5 Runs", "Six", "Bye", "Leg bye, "Short run", "Television replay", "Penalty runs", "Last hour", "Revoke last signal", "New Ball", "Challenge System", "Power Play" and the like.

[61] By limiting the set of signal commands to a finite set of signal commands within the dictionary 124 including for a particular type of sport, the recognition accuracy of the ACR decoder 118 is increased.

[62] In further embodiments, the memory device 106 of the off-field device 101 comprises a gameplay progress decision tree 123. During gameplay, off-field device 101 may track gameplay progress by branching to various gameplay progress data nodes of the decision tree 123 according to a set of gameplay rules 122 and gameplay events, including detected gameplay events.

[63] For example, the decision tree 123 and gameplay rules 122 for a game of cricket may currently represent the gameplay progress of the current ball. When a spoken signal is received for the current ball (i.e. "one run", "no-ball" and the like), the current position within the decision tree 123 will move to the next ball position. In embodiments, the scoring controller 119 won't allow gameplay progress to progress up the decision tree 123 a signal is received for the current ball.

[64] In embodiments, the decision tree 123 may be utilised for enhancing the classification accuracy of the ASR decoder 118.

[65] For example, when classifying the features into one or more spoken signals, the ASR decoder 118 may reference the decision tree 123 to confine the applicable set of possible spoken signals applicable for the particular gameplay position and filter out inapplicable spoken signals.

[66] For example, should the gameplay progress of the decision tree 123 for a tennis match indicate that the current score is love-fifteen, the ASR decoder 118 may infer from the decision tree 123 that a subsequent classification of spoken signals of, for example "fifteen-thirty", "love-forty" and the like would be incorrect classifications of the signals.

[67] As such, by tracking gameplay progress with reference to the decision tree 123 and rules 122, the signal classification accuracy of the ASR decoder 118 is enhanced.

[68] The scoring controller 180 would also may also make reference to the gameplay progress of the decision tree 123 when calculating and updating the match score 120.

[69] For example, for cricket, when receiving the "bye" signal, the scoring controller 119 would update the gameplay progress of the decision tree 123 such that when receiving a subsequent "1 run" scoring signal, the scoring controller 118 would, by referencing the gameplay position of the decision tree, increment the team's total by one run but leave the current batsman's score unchanged. [70] In embodiments, wireless data transmission between the in-field device 102 and the off field device 101 may be bidirectional in that the off-field device 101 comprises a transmitter 108 for wirelessly transmitting data to a receiver 109 of the in-field device 102.

[71] In embodiments, off-field device 101 may be configured for transmitting data indicative of the correct recognition or the inability to recognise a spoken signal.

[72] Furthermore, the in-field device 100 to may comprise a feedback device 113, such as a vibrational haptic device. As such, a short pulse of the haptic device may be indicative to the umpire/referee of the correct recognition of the spoken signal whereas two short pulses may be indicative of the inability to recognise the spoken signal, prompting the umpire/referee to reissue the signal until such time that a short pulse is received.

[73] In embodiments, should the off-field device 101 determine that a signal is illogical in accordance with a current gameplay position of the decision tree 123, off-field device 101 may wirelessly transmit an indication of such to the in-field device 102. For example, the umpire may miscalculate the number of balls bowled in an over and allow seventh ball to be bowled. As such, the feedback device 103 may provide a long pulse indicative of the potential gameplay error prompting rectification by the umpire/referee oi possible.

[74] In embodiments, the in-field device 100 to may comprise a user interface 114, such as a pushbutton interface which may, for example, be used to input more commonly used signals, such as a "dot ball" signals in cricket. As such, as opposed to having to verbalise this signal for each ball bowled, the umpire may rather depress the button user interface 114.

[75] In embodiments, the in-field device 102 is further in operable communication with a handheld device 103. In embodiments, the handheld device 103 may be in wireless communication with the infield device 102 by way of a wireless data connection, such as a Bluetooth data connection.

[76] The handheld device 103 may comprise the user interface 114 in this embodiment and may take the form of a smaller more conveniently held handheld device more ergonomically used for depressing the pushbutton user interface 114.

[77] In embodiments, the handheld device 103 may comprise a push-to-talk button for selectively capturing speech signals.

[78] In embodiments, so as to further enhance the classification accuracy of the AS decoder 118 and/or the accuracy of the scoring controller 118, auxiliary data may be received and analysed by the off-field device 101 to detect certain signals and gameplay events.

[79] In one embodiment, the in-field device 102 comprises a Micro-electromechanical Systems (MEMS) accelerometer 112, such as one which is worn on the wrist of the umpire/referee, and an associated motion analyser controller 135. As such, in accordance with this embodiment, the accelerometer 112 monitors acceleration such that the motion analyser controller 135 is able to infer hand gestures from the motion of the wrist(s) of the umpire/referee.

[80] In embodiments, the motion analysis controller 135 employs template based matching wherein a plurality of acceleration classification patterns 133 store reference patterns for each class/gesture for matching against the acceleration data using similarity measurements, such as Euclidean distance matching. Alternatively, the motion analysis controller 135 may employ model- based methods based on the probabilistic interpretation of acceleration data such as by using Hidden Markov Models (HMM)

[81] For example, the acceleration classification patterns 133 may comprise a "wide" signal acceleration pattern indicative of the umpires wrist moving from side to side. Furthermore, the acceleration classification patterns 133 may comprise an "out" acceleration pattern indicative of the umpire's wrist moving upwardly.

[82] In one embodiment, the acceleration pattern classification may be utilised to enhance the detection accuracy of the AS decoder 118 wherein, for example, should the spoken command be inaudible and may potentially be classified as both "out" or "wide", the accelerometer data may be determinative for the correct classification.

[83] Furthermore, in embodiments, the utilisation of the accelerometer data may allow the umpire to issue spoken or hand gesture commands alternatively.

[84] In further embodiments, audio data may be analysed/classified wherein the system 100 comprises an in-field audio sensor 105 and audio analyser 125 for detecting/classifying audio signals received from the audio sensor 105 according to various well-known audio signal classification (ASC) techniques.

[85] For example, for cricket, the audio sensor 105 may be located near the batsmen so as to be able to classify audio signals indicative of a ball snick event. The audio analyser 125 is able to classify the "ball snick" signature which, if followed by a wicket-keeper "glove strike" or "crowd cheering" audio signature event, is determinative of an "out" signal. By way of further example, should the audio analyser 125 classify the "ball snick" signature and the umpire/referee/issue the "wide" verbal signal, the off-field device may provide the long pulse signal via the feedback device 113 indicative of the potential inaccuracy of the signal.

[86] In embodiments, the audio signature classification may be utilised to enhance the classification accuracy of the ASR decoder 118. For example, when classifying a "bat strike" audio signature, the gameplay progress of the decision tree 123 may be updated to indicate the successful hitting of the ball wherein, for a subsequent spoken signal from the umpire which could either be classified as "wide" or "1 run", the latter may be favoured as the correct classification by the ASR decoder 118.

[87] In further embodiments, vision data may be analysed wherein the system 100 comprises an image sensor 104 and/or LIDAR sensor 134 and an associated vision analyser 126.

[88] For example, the vision analyser 126 may implement motion detection of the vision signals received via the image sensor 104 to detect a "batsmen crossing the pitch" event and update the gameplay position within the decision tree 123 accordingly.

[89] Similarly, such an updated gameplay position may be used to subsequently enhance the detection accuracy of the ASR decoder 118.

[90] In embodiments, other sensors may be utilised such as boundary laser sensors, cricket wicked/tennis net accelerometers and the like.

[91] In embodiments, the scoring controller 119 may utilise match specific data 121 within memory 106. For example, as opposed to the umpire/referee having to memorise player names, player numbers may rather be provided on field. As such, the umpire may issue concatenated signals such as "player 2 - bye - 1 run".

[92] In embodiments, the signal dictionary 124 may comprise multilingual signals so as to allow the recognition of spoken signals in differing languages. In embodiments the ASR decoder 118 and the signal dictionary 124 may be configured for allowing bilingual or multilingual capabilities such that a single installation may recognise spoken command signals in differing languages alternately, such as for karate matches for receiving signal commands in both English and Japanese.

[93] Whereas the embodiment shown in figure 1 shows two devices 102, 101 in wireless communication with each other, it should be appreciated that, in other embodiments, a differing system configuration may be employed within the purposive scope of the embodiments described herein.

[94] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.