Field

The present invention or innovation relates to the capture of video images of an activity, particularly a sporting event, an entertainment event, or a training or educational event. In one field to which the invention relates to which the invention relates, different image capture locations are used simultaneously to create multiple video recordings of the same sequences of events in the activity. Very high speed camera optimisation systems are also described.

Background

The invention has been particularly developed for capturing video images of sporting events and it will be convenient to describe the background and the invention in relation to this particular application of the invention, but the invention is not limited to such use at sporting activities.

It is known in the sporting broadcast industry to use numerous image capture locations suiTounding a sporting field of play. A plurality of video cameras capture video images of the events occurring during the sporting activity and the cameras generate video data signals for storage and selective later display of the images of the events, as well as providing data signals for real time broadcasts or the like. Such video cameras include standard speed video cameras ("standard cameras") which, for conventional live broadcasts or for storage and later broadcast or other transmission, usually record at a frame rate of 25 frames per second (fps) in the case of Australia and many other countries, or at 30 fps for some countries such as USA.

Also known is the use of high speed video cameras ("slowmo cameras") provided at one or more locations and sited to capture images of the events in the sporting activity for later replay at a slowed frame rate enabling umpiring or refereeing decisions to be made concerning the events, and/or enabling enhanced information for commentators and entertainment for viewers. Slowmo cameras conventionally record at three times the standard camera frame capture rate, i.e. at 75 fps for Australia, or 90 fps for USA, etc.

The video data from standard cameras and slowmo cameras set up at a sporting venue are supplied in real time through data linlcs from the respective camera locations at the venue of the sporting activity, and the data signals are fed to a control location, such as an outside broadcast facility sited in the vicinity of the venue of the sporting event.

Even using multiple standard cameras and multiple slowmo cameras in numerous camera locations surrounding the field or arena of the sporting activity, commonly insufficient information is captured enabling clear visualisation of significant occurrences in the events. For example in Australian Rules Football, slowmo replays of incidents associated with scoring of a goal can frequently be inconclusive because even slowmo replays from cameras operating at three times the standard frame capture rate, critical details of occurrences may be incapable of being discerned. For example in Australian Rules Football, whether the ball was touched by a player within the field of play before crossing the goal line determines whether the score is a goal (six points) or a "behind" (one point). Use of a very high speed video camera capturing images at a frame capture rate significantly higher than a slowmo camera will enable better discrimination of such occurrences particularly if accurately synchronised with video replays from other camera locations. For example, a very high speed video camera may enable capture of video images to enable very fine determination of the exact instant that a ball was touched, whereas a standard camera video or slowmo camera video may be quite sufficient to determine the location of the ball in relation to the goal line at that same instant of the ball touch.

However, very high speed video cameras are not used widely in broadcast of sporting events because of a number of considerations including:

• the colour quality of known very high speed video cameras has not been equivalent to the colour quality of standard cameras and slowmo cameras,

• it has been considered that providing additional camera locations for very high speed video cameras would be inconvenient, would likely encroach upon spectator areas resulting in loss of some seating capacity at the venue, and would be expensive requiring substantially more configuration and set up work and provision of data links and control systems (e.g. because of the very high frame capture rate, there are large amounts of data to be transmitted and it has been considered necessary to locally buffer the data and download the data to the control location later, e.g. on demand five seconds after an incident of interest),

• use of very high speed video cameras has generally been considered unnecessary to resolve most questions surrounding incidents where use of slowmo camera images are used and so the use of very high speed video cameras has been considered unnecessary to add to or enhance the information or entertainment value of the video images captured and replayed. The above references to and descriptions of prior proposals or products are not intended to be, and are not to be construed as, statements or admissions of common general knowledge in the art in Australia or elsewhere.

Summary of the invention

It is an object of the present invention to provide useful systems for capturing video images of an activity to make effective use of video images from high speed cameras and particularly from very high speed video cameras, corresponding methods, and video image data sets generated by the systems or methods. According to a first aspect of the present invention there is provided a video image capture system for an activity where different image capture locations are used simultaneously to create multiple video recordings of the same sequences of events in the activity, the system including: a plurality of video cameras to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video cameras including:

a number of standard speed video cameras,

at least one high speed video camera, and

at least one very high speed video camera,

a plurality of camera locations where the video cameras are located and which provide different vantage locations for the capture of video images of events in the activity;

data links for transmission of video data from the cameras at the camera locations to a control location where the video image data are processed, the processing at the control location including storage in memory of video data from at least some of the cameras;

wherein the camera location for the or each said very high speed camera is the same as the camera location of a respective one of the standard video cameras so that video images of an event are captured by a standard camera and simultaneously captured by a very high speed camera from the same vantage location, and

wherein the video image data from the standard video camera and from the very high speed video camera that share the same vantage location are transmitted using data links from that location to the control location.

By siting a very high speed video camera at the same location as a standard camera, and using data links from that one location to the control location, advantages of enhanced information and/or entertainment from the very high speed video camera can be achieved without very substantial additional work, costs, or reduction of spectator capacity of the venue.

According to a second aspect of the invention there is provided a video image capture system for an activity where different image capture locations are used simultaneously to create multiple video recordings of the same sequences of events in the activity, the system including: a plurality of video cameras to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video cameras including at least two very high speed video cameras,

a plurality of camera locations where the video cameras are located and which provide different vantage locations for the capture of video images of events in the activity; data links for transmission of video data from the cameras at the camera locations to a control location where the video image data are processed, the processing at the control location including storage in memory of video data from the cameras;

wherein the camera locations for the said very high speed cameras are selected to provide different vantage locations all enabling capture of images of a target area within the larger field where the activity occurs, and wherein each very high speed camera is mounted at a respective one of the camera locations and is directed to capture images of substantially the entire target area whereby multiple very high speed cameras all capture from different vantage locations images of the same incidents of events occurring within the target area.

By using this second aspect it is possible to obtain very high speed image capture enabling very slow speed replays of events or incidents in a target area where critical incidents in the activity can be expected while maintaining good resolution.

According to a third aspect of the invention there is provided a video image capture system for an activity occurring within a field, the system including:

a camera set comprising at least three video cameras operative to capture video images of events occurring during the activity and to generate video data signals for storage and for selective later replay of the video images of the events,

a single camera location where the camera set is located and from which a target area within the field where the activity occurs is imaged by the video cameras,

wherein all the video cameras of the camera set are fixed in position at the single camera location and wherein each of the video cameras is positioned so as to be directed in a respective direction which is different from the directions of the other video cameras whereby each of the video cameras captures images of a respective smaller part of the target area.

This system enables video coverage of a large area without needing to pan across that area, and by unique siting at a ground level point beyond the dead ball line of a rugby field, can capture images enabling viewers to see if a touch of the ball to ground was made.

According to a fourth aspect of the invention there is provided a video image capture system for a sporting activity in which there are generated video recordings of sequences of events in the sporting activity, the system including:

a very high speed video camera to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video camera including a sensor array which generates the video image data; and

a data selection system operative to select for storage in memory and/or for visual presentation in a broadcast or in a video replay a subset of the possible maximum set of image data from the sensor array of the camera, the subset representing a selected subfield of the total field of view of the camera.

This fourth aspect can effectively use a very high speed video camera with a facility to select a smaller part of the full field of view of the camera for recording and super slow m otion replay without panning or zooming functionality of the camera itself while maintaining acceptable image resolution.

Brief description of the drawings

Possible and preferred features of the present invention can be further understood by making reference to the accompanying drawings. However it is to be understood that the features illustrated in and described with reference to the drawings are not to be construed as limiting on the scope o f the invention.

Fig. 1 of the drawings schematically illustrates a possible arrangement of standard, slowmo, and SS cameras at a sporting venue and which are located and configured according to the first aspect of the present invention.

Fig. 2 schematically illustrates a system, according to a second aspect of the invention comprising a possible set of locations for SS cameras used to capture video data for events in a Rugby or American football game in the vicinity of the try line at one end of the field of play.

Fig. 3 schematically illustrates a system of capturing video images of a rugby or similar game according to the third aspect of the invention.

Fig. 4 schematically illustrates a system of capturing video images of a sporting activity according to the fourth aspect of the invention.

Description of preferred features

The schematic illustration in Fig. 1 of the locations of the cameras is suitable for televising an Australian Rules Football match. The set up and operation of the system schematically illustrated in Fig. 1 of the drawing can be understood by referring to the following description and the annotations on the drawing.

The system preferably uses very high speed video cameras having frame capture rates of about 200 frames per second or more.

The preferred very high speed video camera used in the present invention has a frame capture rate of greater than about 200 fps, but can be as high as about 1000 fps or more. Such cameras are referred to as "Super Slowmo" cameras or "extreme" cameras and will be referred to herein as "SS cameras". Although frame capture rates for SS cameras suitable for commercial broadcasts can be as high as 1000 fps, and such rates are achievable with acceptable quality output in good lighting conditions, particularly daylight. At lower light levels, particularly at night, acceptable quality video image capture might be at a reduced frame rate of, say, 400-500 fps.

The light sensor of the SS camera used in the present invention is preferably provided with relatively large pixel sizes. Standard CMOS sensors use arrays of pixels each having dimensions of about 5x5 microns. Preferably the sensor array used in the SS camera in the present invention has large pixel sizes, e.g. each pixel sensor being about 10x10 microns.

The capture of video images in the SS camera may use either a rolling shutter for capturing of data from the pixel sensor array. A "rolling shutter" system samples pixels in sequence, such as a raster sequence throughout the area of the sensor array. Using a rolling shutter system provides less signal noise and is better used for lower light conditions such as at night.

However the SS camera may use a global shutter system for capturing the image data from the sensor array. A "global shutter" system samples all pixels of the sensor array

simultaneously. This can provide better quality data in daylight conditions.

The SS camera preferably has a sensor array with a Bayer colour filtering for optimising the video data.

Preferably the SS camera is mounted in association with the respective standard camera so that movement, such as panning, of the standard camera moves the SS camera identically whereby both cameras capture video images of substantially the same events.

Preferably the SS camera is mounted to the standard camera, e.g. on top of a housing or body of the standard camera.

The SS camera is adjustably mounted to the standard camera so that it can be sighted or aligned during a configuration or set up operation so that both the cameras are capturing images of substantially identical fields of view where events in the activity occur. For example, the SS camera may be mounted mechanically to the top of the body or housing of the standard camera. Any conventional mechanical means may be provided for adjusting the optical alignment of the line of sight of the SS camera relative to the line of sight of the standard camera including fine adjustments in the horizontal and vertical axes of the field of view.

Preferably the SS camera and the associated standard camera are synchronously operated by a local controller which may comprise a switching unit, the synchronisation of the two cameras including at least one of:

(a) the timing of the video image captures by the two cameras so that video images from the two cameras can later be synchronously replayed, with synchronous video display if desired, and (b) synchronous zooming of the two cameras. These synchronisations ensure the video captures by the two associated cameras exactly correspond.

The standard camera and the SS camera mounted to the standard camera can both utilise the same power supplies at the common vantage location. There is no need with this arrangement for provision at the site of the spoiling or other activity of a new camera location for siting of the SS camera or multiple SS cameras and with the attendant need for power supplies, data links, operator accommodation, and possible loss of spectator seating because of the additional space requirements.

Preferably the data links from the common vantage location for the associated SS camera and standard camera are shared for the transmission of video data signals from both of the cameras to the control location.

The data links conventionally used, for example at spoiling event venues, include broadband data links for the large volumes of data being transmitted. Fibre optic cables and/or co-axial cables are for example used for such data links. According to the present invention such data links can be shared by both the standard and the associated SS cameras.

The shared data link used by the associated SS camera preferably and standard camera preferably utilises high level network technology. Such high level networking technology can be of the kind used for example in internet routers or nodes, such as supplied by Cisco for broadband high data volume switching and transmission.

Preferably the video data signals relating to the video images captured by the SS camera are transmitted in real time to the control location using the data links from the associated standard camera location.

The invention in its first aspect also provides a method of creating multiple video recordings of sequences of events in an activity using different image capture locations, the method including the steps of:

providing a plurality of video cameras to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video cameras including:

a number of standard speed video cameras,

at least one high speed video camera, and

at least one very high speed video camera,

locating the video cameras at a plurality of camera locations so as to provide different vantage locations for the capture of video images of events in the activity; transmitting of video data from the cameras at the camera locations via data links to a control location where the video image data are processed, the processing at the control location including storage in memory of video data from at least some of the cameras;

locating the or each said very high speed camera at the same camera location as a respective one of the standard video cameras so that video images are captured by a standard camera and simultaneously captured by a very high speed camera from the same vantage location, and

transmitting the video image data from the standard video camera and from the very high speed video camera that share the same vantage location using data links from that location to the control location.

wherein the video image data from the standard video camera and from the very high speed video camera that share the same vantage location are transmitted using data links from that location to the control location.

The invention in its first aspect also provides a recorded data set comprising data for video images of events recorded during an activity and enabling selective later replay of the images of the events, the data set comprising stored video data generated by the system or the method according to the first aspect of the invention.

In addition to (or even as an alternative to) one or more SS cameras being located at and associated with a respective standard camera for simultaneous and synchronous capture of video images of substantially the identical events in the sporting activity, there may be one or more SS cameras likewise located at and associated with respective ones of the slowmo cameras. Such SS cameras can enhance the data quality (information content) compared to the associated slowmo camera, by capturing between two and about ten times the quantity of image frames as the associated slowmo camera, but the incremental advantage of such SS cameras associated with slowmo cameras is not so marked as the incremental advantage of the image data from SS cameras over the image data associated with standard cameras.

It is also to be understood that there may be additional SS cameras located at respective unique camera locations providing additional video footage for information, decision, education, entertainment purposes.

Fig. 2 of the drawings schematically illustrates a possible set of locations for SS cameras used to capture video data for events in a Rugby or American football game in the vicinity of the try line at one end of the field of play. Similar locations of cameras at the opposite end of the field would also be provided. The three SS cameras in Fig. 2 are positioned so that all three have the entire try line in their fields of view. The three cameras may all be fixed so that their fields of view remain the same throughout the game, ensuring that critical incidents near the try line determining scores can be captured by SS cameras, preferably by all three but sometimes by only two or even by one only if there are obstructions to clear lines of sight of the incident (such as participants in the game).

The invention in its second aspect also provides a method of creating multiple video recordings of sequences of events in an activity using different image capture locations, the method including the steps of:

providing a plurality of video cameras to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video cameras including at least two very high speed video cameras,

locating the video cameras at a plurality of camera locations so as to provide different vantage locations for the capture of video images of events in the activity;

transmitting video data from the cameras at the camera locations via data links to a control location where the video image data are processed, the processing at the control location including storage in memory of video data from the cameras;

locating the said very high speed cameras at respective camera locations which are selected to provide different vantage locations all enabling capture of images of a target area within the larger field where the activity occurs, and mounting each very high speed camera at a respective one of the camera locations and directing it to capture images of substantially the entire target area whereby multiple very high speed cameras all capture from different vantage locations images of the same incidents of events occurring within the target area.

The invention in its second aspect also provides a recorded data set comprising data for video images of events recorded during an activity and enabling selective later replay of the images of the events, the data set comprising stored video data generated by the system according to the second aspect, or generated by the operation of the method according to the second aspect.

Fig. 3 of the drawings schematically illustrates a video image capture system for an activity occurring within a field (shown as a rugby field), the system including a camera set comprising four video cameras operative to capture video images of events occurring during the activity and to generate video data signals for storage and for selective later replay of the video images of the events, and a single camera location where the camera set is located and from which a target area within the field where the activity occurs is imaged by the video cameras. All the video cameras of the camera set are fixed in position at the single camera location and each of the video cameras is positioned so as to be directed in a respective direction which is different from the directions of the other video cameras whereby each of the video cameras captures images of a respective smaller part of the target area.

Each of the smaller parts is contiguous with at least one other smaller part and the smaller parts cumulatively cover at least the entire target area. Preferably the video cameras include types of cameras selected from high speed video cameras and very high speed video cameras. Preferably also all of the video cameras of the camera set are identical types of video cameras, e.g. SS cameras.

Fig. 3 illustrates the system in use where the activity is a game of rugby or a variant or similar game and the target area comprises the try line of the field of play or pitch. The camera set is provided and operated at the single camera location sited beyond the try line, particularly beyond the dead ball line, and the video cameras are directed towards the field of play or pitch to thereby cumulatively capture images along the entire length of the try line. The camera location is substantially at or slightly above ground level thereby enabling the video cameras to capture images showing whether the ball used in the game is touched to the ground or not.

In use, the system of Fig. 3 provides capturing and enabling slow motion replay of images shortly before an incident in which a player attempts to touch the ball to the ground beyond the try line, as well as at the moment of such contact being made, and a short time thereafter. The multiple video images can be used also for information, adjudication and/or entertainment by replaying longer sequences of play from the vantage position of the camera set, namely at ground level. The multiple video image sequences from the multiple cameras can be stitched together electronically so that the multiple images provide a panoramic single view of the entire try line from the ground level vantage of the camera set. Selecting subsets from the stitched panoramic image (e.g. as later described in relation to the fourth aspect) can enable display of the rugby game leading up to the attempted try, including display of movement across the field of play without the associated jitter caused by having moving cameras tracking the game play. It will also be understood that it is possible to select either a subset from the stitched panoramic view from the multiple camera images or alternatively video image sequences from a single one of the multiple cameras for replay of the incident of interest, particularly the grounding of the ball in an attempted try.

The capture of images from ground level will not enable determination from those captured images whether the ball had been touched to the ground before or beyond the try line. However this is not the purpose of the use of multiple video cameras forming a camera set at the ground level location. Whether the ball has been touched to the ground before or beyond the try line can be determined from other conventional elevated camera locations. In particular, synchronous display of slow motion replays from conventional high level camera locations together with the ground level captured images will enable determination of the location of the ball touching the ground in relation to the try line as well as whether there was actual successful touching of the ball to the ground determined from the ground level camera set images.

When installed and set up preparatory to capturing images of a rugby game, the video cameras of the camera set can be adjusted to allow for the different distances of the respective cameras from their imaged smaller parts of the length of the try line. In particular, with the camera set sited substantially on the longitudinal centre line of the rugby pitch at a location beyond the dead ball line, the outer video cameras of the camera set will be adjusted during set up to have slightly longer focal distances because they are optimally imaging the outermost parts of the try line furthest from the location o f the camera set. The centre video camera (if three cameras are used) or central two cameras (if four cameras are used as illustrated) will be set up to have shorter focal distances.

The invention also provides a recorded data set comprising data for video images of events recorded during an activity and enabling selective later replay of the images of the events, the data set comprising stored video data generated by the system according to the third aspect.

Fig. 4 of the drawings schematically illustrates a system of capturing video images of a sporting activity according to the fourth aspect of the invention. This system uses a video image capture system for a sporting activity in which there are generated video recordings of sequences of events in the sporting activity, the system including:

a very high speed vi deo camera to capture video images of events during the activity and to generate video data signals for storage and for selective later replay of the images of the events, the video camera including a sensor array which generates the video image data; and

a data selection system operative to select for storage in memory and/or for visual presentation in a broadcast or in a video replay a subset of the possible maximum set of image data from the sensor array of the camera, the subset representing a selected subfield of the total field of view of the camera. This system can be used as "stand alone" system for optimising the value of the very high speed camera data or can be used in the multi-camera systems of the first, second and third aspects (Figs. 1 , 2 and 3).

In a possible embodiment of the fourth aspect of the invention, the SS camera may have a "6 megapixel" specification, e.g. having a sensor array of 2832 x 2128 sensors, generating image frames each composed of 6,026,496 pixels if all sensors are actively read for each frame.

However by reading say only a subset of sensors selected to include image data for a target subfield of the total field of view in which an incident of interest occurs, a faster frame capture rate from the SS camera will be achieved. With a suitable size of the selected subset relative to the sensor density of the sensor array, little or no discernible loss of resolution of the image quality needs to result from the utilisation of only a subset of the maximum available number of pixels from the camera's sensor array.

Illustrating this method of utilising image data subsets, if a subset of 1420 x 1080 sensors of the total 2832 x 2128 sensor array is selected from which image data is read, images of 1.5 megapixels can be generated representing approximately one quarter of the field of view of the SS camera. Such 1.5 megapixel images may be quite adequate for super slowmo replays of the incident of interest. The capture and storage of such subsets of image data can be much faster than utilising the entire sensor array so much slower slow motion replays are possible without excessively interrupted or stepwise motion of the captured action degrading the value or experience for viewers (whether game officials, broadcasters, commentators, spectators and viewers).

Another available SS camera has a 12 megapixel sensor array so using this camera enables a wider total available field view for equivalent resolution, or greater resolution if set up for an equivalent field of view.

The subset of image data selected for use may be selectively variable in position and/or size compared to the total available image data from the camera's sensor. For example the selected subset may be shifted anywhere within the total available field of image capture under operator control. This may be done in real time during the spoiling event by an operator viewing images presented in real time and produced from data being fed from the SS camera and/or from the associated standard camera, and by the operator controlling the location of the sampled or selected subfield of view for which the image data subset is required. Fig. 4 illustrates schematically an operator in real time viewing images from, say, a standard video camera and, by input means (e.g. keyboard, mouse, joystick, touch pad, touch screen, voice recognition controller, etc.), selecting a subfield of the total field of view for generation of the image frames from the SS camera associated with the standard camera. The operator can therefore ensure video images of incidents in the activity that are of particular interest are being captured at high speed by the SS camera.

By selecting a subfield shifted laterally and/or vertically from the centre of the total field imaged, the operator is effectively panning the SS camera partially independently the associated standard camera. Of course, as the standard camera is panned by its operator, the field of view of the SS camera also moves so the central operator may decide to move the selected subfield to maintain the desired subfield image. Also the system may include functionality enabling the operator to change the dimensions of the selected subset of image data from the SS camera, thus effectively zooming into incidents in the activity or outwards to give a wider field of view of the SS camera output. The minimum size of the subset can be limited to avoid excessive degradation of the resolution.

In the case of fixed SS cameras, such as in the Fig. 2 or Fig. 3 embodiment for rugby, or a fixed SS camera mounted by a goal post in Australian Rules Football, or a fixed SS camera imaging a region including stumps and batsman in cricket, the selection of a subset of image data effectively provides images from a target part of the total field of view of the camera. Because the SS camera is fixed the total available field of view is known and the operator can use the input means to zoom into and/or pan within that field to select a target subfield where incidents of interest in the activity are occurring. This can be done in real time during the event to enable the capture of images at a faster frame rate while maintaining the details of the incident of interest in view.

In one embodiment the system provides functionality to at least partially automatically select the subset of image data from the SS camera being recorded in response to the operator indicating by pointing or touching where in the total field of view the action of interest is occurring, e.g. where the ball in the sports game is located in the field of view. By continually pointing or screen touching in the total field of view of the SS camera, a target subfield of view is automatically determined by the system software surrounding the point indicated by the operator. To avoid the appearance of discontinuous camera movement or the imaged field jumping suddenly to another location, the selected subfield may be automatically progressively approached and reached from the previous selected subfield, thus simulating smooth panning of the camera to follow the action in the game.

It is to be understood that various alterations, modifications and/or additions may be made to the features of the possible and preferred embodiment(s) of the three aspects of the invention as herein described without departing from the spirit and scope of the invention.