Device and method for rendering of moving images and set of time coded data containers

The current invention is concerned with a device for rendering of moving images and with a set of time coded data containers. The current invention is further concerned with a method for rendering of moving images.

Background

Current moving image contents tend more and more to be composed of image sections shot by a camera and image sections comprising computer graphics.

For instance, it is common to shoot full images of movements of an actor in a studio in front of a monochromatic background and to add background image content afterwards. For doing so, garbage matte data are generated serving as a mask for separating the full images into a foreground layer to-be-preserved and a background layer to-be-discarded or to-be-replaced. Generating matte generation is also known as keying.

The computer graphics is commonly generated using computer graphic assets. Using the assets and camera position data comprising camera position in 3D plus pan angle and tilt angle of a current shot, computer graphics background can be generated for combination with foreground extracted from the current shot. Time required for computer graphics generation is varying in dependency on the complexity of the computer graphics.

Summary of the invention

Due to the variability of time required for computer graphics generation, computer generated content is made available asynchronous with availability of the image data. The inventors recognized that despite said lack of synchronicity real-time rendering and preview of the entire content would become possible if generation of computer generated content could be done a little ahead. To do so, the inventors propose to predict a future camera position at a future point in time and to generate computer generated content for said future camera position.

That is, it is proposed a device according to claim 1 and a method according to claim 5 for rendering moving images which comprise image sections shot by a camera entrained by a time code generator and image sections comprising computer graphics. Furthermore, a data container according to claim 9 is proposed.

According to an aspect of the invention, the device comprises, for receiving time codes, a time code interface for connection with the time code generator. For receiving time coded full image data and for receiving time coded camera tracking data, the device comprises an image and camera data interface for connection with the camera. Further, the device comprises means for generating time coded garbage matte data using the full image data. The device also comprises means for predicting future camera position data of a future point in time using current camera tracking data of a current point in time and, for transmitting at least the future camera tracking data prediction and for receiving computer generated content of the future point in time, a rendering engine interface for connection with a rendering device. And, means are comprised for generating a time coded preview using at least the image data, the garbage matte data and the computer generated content.

In an embodiment, the future point in time is selected such far ahead in the future that the computer generated content can be rendered before. Thus, computer generated content can be made available together with garbage mattes and full image data and real-time preview becomes possible.

In an embodiment, the device can comprise a data management device for generating time coded data containers, the data container of a respective time code at least comprising the respective time code, the camera tracking data of the respective time code, the full image data of the respective time code and the garbage matte data of the respective time code. In an embodiment, the device can comprise a storage device interface for retrieving computer graphics assets from a storage device wherein the rendering engine interface is adapted for transmitting the computer graphic assets to the rendering device.

The method according to the invention comprises steps of, receiving time codes from the time code generator; receiving time coded full image data and receiving time coded camera tracking data from the camera; generating time coded garbage matte data using the full image data; predicting future camera position data of a future point in time using current camera tracking data of a current point in time; transmitting at least the predicted future camera tracking data to a rendering engine and receiving time coded computer generated content for the future point in time from the rendering device; and generating a time coded preview using at least the image data, the garbage matte data and the computer generated content.

In an embodiment of the method the future point in time is selected such far ahead in the future that the computer generated content can be rendered before. In an embodiment, the method can comprise generating time coded data containers, the data container of a respective time code at least comprising the respective time code, the camera tracking data of the respective time code, the full image data of the respective time code and the garbage matte data of the respective time code.

In an embodiment, the method can comprise retrieving computer graphics assets from a storage device and transmitting the computer graphic assets to the rendering device. In a set of time coded data containers according to the invention, each data container at least comprises a respective time code, camera tracking data of the respective time code, full image data of the respective time code and garbage matte data of the respective time code. In an embodiment, each data container further comprises a time coded preview.

Further advantageous embodiments of the invention are described dependent claims and the detailed description. Drawings

Exemplary embodiments of the invention will be described in detail below using the figures. It shows:

Figure 1 an exemplary system in which the invention can be used;

Figure 2 schematically and exemplarily how the exemplary system of Figure 1 can be used on set;

Figure 3 modules comprised in exemplary embodiments of the inventive device; and

Figure 4 details of the modules depicted in figure 3. Embodiments of the invention

Figure 1 shows an exemplary system in which the invention can be used. For wireless connectivity the device comprises antenna 100 allowing for Wi-Fi, GPRS or ETL connection such that video signals can be received and/or transmitted.

An HD-SDI-Monitor 200 allows for watching video content. For remotely controlling a camera rig, the depicted device comprises a joystick 400 but other means for remote control of the camera are possible. The system further comprises means 500 for receiving user inputs via, e.g., a keyboard, a touch pad and/or a mouse. For recording video content a HD-Recorder 800 is comprised. Data generated is stored in data storage 1000, for instance realized as disk array. For processing the system comprises two CPUs 600, 900, each with an associated system disk. Further, a remote control 3 for a keying module and a power supply 1200 is comprised. A connectivity panel 1 100 enables connection to other systems or devices. The HD-SDI-Monitor 200 allows for displaying video content composed of live action currently recorded on set and static and/or animated digital elements in a preview mode in real-time and for displaying GUI for controlling and parametrizing the different modules, hard and software.

Fig. 2 depicts schematically and exemplarily how the exemplary system can be used on set. A camera CAM captures image frames at a frame rate, preferably but not necessarily, digitally and in high resolution. The camera is entrained by a time code generator TCG. The time code generated by the time code generator TCG is made available via an interface to an exemplary embodiment of the inventive device diVERT. The interface can be implemented such that the time code generator TCG is a plug-in of the device diVERT.

The device further receives, via a further interface, camera data gathered from a tracking system TRK and processed by tracking software TKS. The camera data at least comprises pan angle and tilt angle, zoom and focus. This is sufficient for a tripod mounted camera. In an embodiment, camera data further comprises 3D position data of the camera CAM and camera motion data as required for servo- robotic motion control cameras.

Servo-robotic motion control cameras comprise motion control, for instance, by a fly- by-wire system for camera movements. A motion control camera is equipped with a system for repeating camera operations identically by computer control. It is often used in cinema and other productions that require complex multiple compositions. Compatibility of the camera data with the industry standard, like, e.g., MAYA CG software makes shooting more efficient by enabling advance simulation of the robotic motion. The ease of reproducing identical camera operations reduces the time required for the multiple and CG compositions of VFX productions. In the exemplary system of Fig. 2, camera CAM is mounted on a crane or dolly which is moved by electric motors using joystick 400.

The tracking software TKS exemplarily consists of a man-machine interface (MMI) and a robot controller that generates run commands for the crane. The crane position for each axis is selected using the joystick 400. The positional control data are then inputted to the MMI via analog I/O. The motion trajectory (spline curve) is generated from the starting point to the finish of robotic motion once all positional data have been inputted to the MMI for each given key frame.

When a run command for motion control is sent from the MMI, the data pass to the robot controller for conversion into the control data of each frame. The motion controller inside the robot controller sends the command signal to each motor driver in the control box and each servomotor of the crane moves accordingly to achieve crane operation. Alternatively to remote control, the rig may be operated manually in which case the controller servos will be switched to passive. Positional camera data is transferred to the 3D-CG software format for each frame. The system can, therefore, simulate the robotic motion in 3D on the interface PC before the shooting takes place.

The device diVERT of this exemplary embodiment uses this data to achieve perfect geometrical and optical coherency between a real and a virtual camera. This will be done by encoders on the which will provide continuous data about the rotation on the pan axis (or horizontal panoramic), on the tilt axis (or vertical panoramic), on the zoom and on the focus rings. This is done by the encoders being geared to each axis of rotation, and the controller which gathers information and transmits it to the tracking CPU over a LAN connection.

The image content captured by the camera CAM is either directly provided to the device diVERT through a digital video interface, if captured digitally, or is provided via a film scanner FSC and through another interface, if captured analogue.

The device diVERT generates, using a data management module, for each frame captured, a data container comprising a time code of the frame and image data of the frame. In addition thereto, garbage matte data for the image data comprised is further comprised in the container. Optionally, the container can further comprise a preview of the time code, computer graphic assets and/or computer graphic rendering output of the time code.

The exemplary embodiment of the device diVERT depicted in Figure 2 further comprises optional interfaces for interfacing with storage SSD, 3D software 3DS, compositing software COM and on-set preview PVW, respectively.

If present, interface allows for retrieving computer graphic assets from storage SSD.

If present, interface for interfacing with 3D software 3DS is bidirectional and allows for transmitting at least camera data from the device diVERT to 3D software 3DS as well as for transmitting 3D rendered output from the 3D software 3DS to the device diVERT. For accessing computer graphic assets, the 3D software 3DS may interface with storage SSD directly, or receive, via said interface, the computer graphic assets retrieved by the device diVERT from storage SSD via interface and provided to the 3D software 3DS together with garbage matte data and camera data.

If present, interface allows the composite software CMS to import data containers for further processing, post-processing and/or final rendering.

If present, interface can be used to provide a preview to a display.

For instance, plug-ins for selected 3D and compositing software packages can be provided which allow direct 1 -click import of all linked data.

Figure 3 exemplarily depicts modules comprised in exemplary embodiments of the inventive device diVERT. Exemplary tracking module TM registers, synchronized by the time code received via interface, camera data comprising camera tracking data such as position and orientation of the camera in 3D.

Tracking Module TM comprised in device diVERT uses the raw data provided by the encoded rig to compute all virtual 3D camera parameters so that the virtual image will fit the geometric features of the actual image.

The format in which the data then is stored is preferably one which allows handling by known 3D software packages such as Autodesk Maya, 3D MAX, or the like. For example the data is stored in FBX and/or COLLADA file format. This compatibility is eases further workflow.

Motion control system preferably uses a reserved Ethernet network for data transfer. This is beneficial for avoiding unwanted interaction with other unknown devices and to preserve bandwidth.

The synchronized camera data is made available, together with the corresponding image data, to keying module KM which determines garbage matte data and foreground image content for the image data.

The Keying Module KM is adapted to separate, in the content captured in each frame, foreground (FG) from a uniform, e.g. green or blue, background (BG) and to generate garbage mattes which are stored for post-production. In an embodiment, the keying module KM is further adapted to composite the determined foreground image content with computer generated background content generated by Preview Module PVM. Preferably, keying is implemented as integrate circuits in order to maximize performance, but software based solutions are possible, too. The Keying module KM of an exemplary embodiment comprises

Dual Link 4:4:4:4 RGBA input and/or output (Foreground (FG) In, Background (BG) In and FG+BG Out) for best compositing results, bypassing of preprocessing stages and delivering of unaltered image to the main processing stages. To increase maximum flexibility, it can be made possible to individually select the FG Input, the BG Input or the FG+BG Output to be 4:4:4:4 or 4:2:2.

Depending on the embodiment, the Alpha (Matte or Key) Input could be delivered as part of the Dual Link signal, and/or could be a separate input. Similarly, the Alpha out could be part of the Dual Link signal, and/or as a separate output.

The Keying module KM of an exemplary embodiment is adapted for receiving 4:2:2 (Y,Cb,Cr) signals and for artifact correction in the 4:2:2 (Y,Cb,Cr) signals in which objects with high contrast and sharp transitions may exhibit an unwanted "ringing" artifact in blue/green screen compositing.

The 4:2:2 artifact correction will eliminate or reduce the ringing artifacts in the transition region of FG objects to the backing (the screen), while maintaining the original image within the opaque FG object. No FG object detail is lost in this process. Only the ringing element is removed from the transition regions while maintaining every strand of hair or semitransparent object.

Digital image capturing devices like CCD or CMOS are inherently noisy and don't exhibit uniform sensitivity to all wavelengths of the visible spectrum. This results in higher noise levels in the red and the blue channels of an RGB image, compared to the green channel. In blue or green screen image compositing, all three channels of the FG contribute equally to the processing of the original image. Therefore, the noisiest channel (Blu) will have an impact on the least noisy channel (Green). The Keying module KM can be further adapted for significantly reducing the noise levels in the FG red and blue channels, with no visible reduction in the fine detail information of the image (all controls at default settings). The green channel will retain its inherent noise level (e.g.: based on camera gain settings), as no noise reduction is applied to this channel.

The keying module KM at least uses one of the following keying methods: Luma keying, difference keying, chroma keying, HLS keying Color difference keying, and 3D keying. As each method has its advantages and drawbacks. For instance, due to its color blindness, luma keying is not useful for the majority of keying jobs, though it may be helpful for special keying jobs like text extraction (black text on white background), and difference keying or chroma keying quality declines with background pixel value variation always present in the images.

Therefore, in an embodiment the keying module KM combines two or more of these keying methods.

That is, keying module KM separates the image data into foreground layer image data and background image data, the former showing captured image content to- be-preserved, the latter showing image content to be discarded or to-be-replaced by, e.g., computer generated content or content captured at a different time or from a different position. This is done in a synchronized way such that the garbage matte data can be provided with time code.

A data streaming management module DSM ensures synchronization of operations during recording. Optionally, the module DSM also scales the image data, up or down. Data streaming management module DSM can further create "containers" which hold all data related to one take of a specific shot. Then, all files will be linked to a specific shot/take. This will be done by using meta-data in addition to the essence, the metadata being included in the file, and a database enabling a user to access information needed through a file-browser, which will group the relevant data, based on the meta-data. The meta-data at least comprises time code. Time code is the most important piece of information, used throughout the production: from shooting, to editing and finally post production.

The time code is used to identify shots, determine their length etc. Additionally, metadata can comprise Production / Shot / Take; Shot "good" or "bad" and/or Roll Number (when shooting with analogue media). Time code can be provided from various sources. Therefore, it is preferable if time code can even be selected. Among time codes from which selection may be possible there is Vertical interval time code (VI TC) according to SMPTE 12M-1- 2008; Ancillary Time Code (ATC, formerly known as DV!TC) according to SMPTE 12M-2-2008; Linear time code (LTC) according to SMPTE 12M-1-2008 and Time code from the application controlling the MXF encoder (e.g. real-time recording device or software encoder). The generic data container comprises Essence with time code in the essence stream. The Essence comprises the image data and may further comprise sound data, compound data and/or system data. The metadata linking the containers into source packages indicates up to n time tracks which may contain discontinuities. The metadata linking the containers into material packages indicates up to one time tracks which may not contain discontinuities.

Data streaming management module DSM outputs time code such that a default output time code shall be equal to the control time code wherein advanced connected devices may optionally allow the output time code to be configured as preset time code or source time code stored in the essence container that is described by the played package (i.e. the material package of an OP1 a, OP2a or OP3a file or a source package of a file of arbitrary operational pattern).

As several modules are running in parallel it need to be ensure that all modules generate data, which has the same IN -and OUT point, based on the same time code. This means that the modules need to be synchronized when the recording starts and stops.

In an embodiment the device diVERT uses a protocol based on master-slave methodology, such as, e.g., a Video Disk Control Protocol also known as "Sony

Protocol" where, in a point-to-point topology, a controlling device takes the initiative in communications between the controlling automation device and the controlled device. The garbage matte data and the image data are then saved by data management module DMM together with the time code and the camera data in a common container. Appropriate attributes may be added, too. The container may contain the camera data as a file in the format in which it is stored by tracking module TM. The container is stored in a storage device SSD, e.g. a solid state video recorder with two or more slots for SSDs enabling "hot-swap" during operation and shooting. Data is linked using metadata included in the individual files and an internal database, based on SMPTE time code. The container may be then be used for generating the final output FO.

In addition thereto, a preview module PVM is present which generates an on-set real-time preview using the containers generated. The preview can be added to the container which is useful for further processing or future preview.

The preview module PVM comprises, as depicted in Figure 4, a 3D rendering engine RND outputting a visual output FO. The preview module PVM at least receives, as input, camera position data. The main task of the Real Time 3D Render Engine RND is to display the virtual 3D model including animation with the right perspective, based on the input (XYZ-coordinates, rotation, zoom, focus) from the tracking module TM. The visual output FO is the result of this operation, which can be viewed, re-viewed and stored. The preview module PVM is connected bidirectional to keying module KM which receives a background image of computer generated content from the 3D rendering engine RND of the preview module and returns a composite image composed of the received background image and the foreground image extracted from the full image data received from the camera. The preview module PVM receives, from the keying module KM, the composite image and generates the final visual output FO which can be previewed and/or encoded for storage. For instance the visual output can be hardware encoded according to H.264 video compression standard and stored in storage device SSD either as part of the data containers or individually.

The device diVERT of an exemplary embodiment of the invention is adapted to provide the preview module not with camera position data according to the current camera position but with data of a predicted camera position. To do so, the device of the embodiment comprises a camera position prediction module. The module determines a prediction of a future camera position for a future point in time using the current camera position and/or the current camera motion information. The future point in time is chosen such that prediction uncertainty remains minimal but the rendering engine having sufficient time for rendering a background image of computer generated content ere to the future point of time being reached. Then captured image and computer generated content are available synchronous and can be rendered in real-time.

In an exemplary hardware realization of the invention, a master clock CPU synchronizes the system by generating a time code generated using, e g, a black & burst signal (bi-level sync.) outputted from a sync-generator with the host Time Code Generator being locked to the black & burst signal, while the HD camera and clapper are synchronized to HD Sync (tri-level sync.) outputted by HD sync- generator. The slate or clapper and the motion control system operate on the basis of that timeline wherein the slate assists synchronization of picture and sound and allows for designating and marking of particular scenes during recording.

A real-time rendering CPU renders all digital elements (animation and/or static background) according to tracking data provided by Tracking CPU and time code. Keying CPU renders the keying or garbage mattes for separating background from the elements to be preserved. Through the garbage mattes integration/compilation of preserved elements with the digital elements becomes possible. This is done in VFX studio where the final image is generated using computer graphics elements. Real-time preview is enabled on-set.

In an embodiment a rendering engine is implemented which supports high-level shaders and multi texturing and handles local and global multi passing.

The implemented rendering engine has a data storage which is performance optimized. Static geometry data can be stored directly on the graphics device, e.g. in an OpenGL vertex buffer-object or display list. The implemented rendering engine further has an effect framework where standard effects like normal mapping and shadow mapping can be chosen from and new effects can easily be integrated (for rapid prototyping). This effect framework is flexible in terms of number of lights and number of used textures. It allows for adding additional effects to all objects in a scene, which in turn should merge with the effects an object already contains. Additionally, it easily allows for getting a depth-only version of an effect. The implemented rendering engine makes use of Cg, GLSL or HLSL programs for rendering and provides a post-processing effect framework to be used for tone mapping, blooming and other effects, which operate on fragments of the frame-buffer.

The implemented rendering engine provides a way of specifying more than one render target and thus supports rendering techniques like deferred rendering. Supporte file formats are at least Maya file format, supported image formats comprise bmp, JPEG, OpenEXR and png. The implemented engine shadowing techniques like stencil shadow volumes and shadow maps are handled like any other rendering effect. The implemented engine supports effects that need local or global multipassing by handling the number of passes needed for applying an effect. The engine makes use of culling of the scene and sorting of renderable objects. Scenegraph algorithms are comprised which provide an easy way to access scenegraph data, as well as enable for traversal of a given scenegraph through development of a new scenegraph algorithm. Developers can easily set complex render states. The engine is scriptable using a common script language like Lua, Python or JavaScript. The rendering engine is graphics API independent and, e.g. OpenGL or Direct3D can be used for rendering. For accelerated rendering the engine provides hardware occlusion queries. And a simple editor allows for creating a scenegraph and applying effects to objects.

To group the functionality of the rendering engine by topic and software level, modules organized as layers are introduced. These modules are called Core, Graphics, GraphicsGL and GraphicsD3D.

The Core module works at the lowest software level of the engine and provides submodules like system functions, a general application framework and common utility functions. The Graphics module is built on top of the Core module and provides an interface to 2D and 3D graphics.

Among other submodules, Graphics contains a graphics-API-independent rendering interface, an effect framework, a scenegraph and importers for several 3D model file formats.

Modules GraphicsGL and GraphicsD3D implement the graphics-API-independent rendering interface provided by module Graphics. This implementation builds the bridge between the rendering interface and

OpenGL respectively Direct3D.

The engine framework implements an actor concept where an actor is a class, which can be instanced by just providing its type ID and its properties are configured by a data chunk. This means that by providing an XML-representation of a chunk, an actor and all its children can be instanced and configured. The actor class is also the base class for all published classes, e.g. classes which should be accessible from a scripting language or which should be serializable. There are some predefined actors for loading XML description files and plugins.

The scenegraph is used to organize the objects of a scene in a hierarchical way. The user can create different kinds of nodes to arrange the meshes, lights and cameras of a scene. The rendering effects are not directly applied to the meshes but are distributed to the scene by creating instances of special node classes.

With every update of the scenegraph, its renderable objects are synchronized with the underlying drawgraph. Renderable objects in this context consist of a geometry object along with an effect defining the way this geometry is rendered.

CgFX effect files is supported. Thus, used effect code will replace the Cg framework program during rendering. Effect files are a convenient way for fast prototyping of new effects.

The engine allows for concatenating small code fragments to build the desired shader code at a level than source code thereby enabling reordering and/or disabling of unneeded fragments. Using the Cg interface feature, DIVERT does not have to care about variable names, register allocations or other code breaking problems. Additionally, the shader fragments (called technique parts in this context) can include render states, which are merged and applied automatically by the system. Abstract Shade Trees are designed to generate the source code for one rendering effect and are used to solve dependencies on the source code level.

The scenegraph of the rendering engine does not make direct graphics API calls. The scenegraph itself with its nodes, which are connected to each other using a parent-child relation, is only kept to allow the user to organize the objects in a hierarchical structured way. Algorithms running on this graph are not supposed to directly reference leaf nodes but find them by traversing from the root to the leaf. But with scenegraphs containing hundreds of thousands of objects grouped together in thousands of groups (e.g. a scenegraph modeling a city), the traversal leads to non-negligible processing overheads. The least traversal overhead is achieved, if all nodes are direct children of the root node which does not allow the user to build logically or spatially organized groups. To let the user organize the nodes in a structured way and simultaneously allow fast traversals, a Scene Database is introduced providing algorithms direct access to the data of scene nodes, e.g. to a list of all meshes in the scene; to structural context data of any node for direct look-up; to a list of all emitted variables and effect in the scenegraph; to a list of all culling objects; to a list of all controllers and to a list of all renderables.

Thus, the scene database is just another view of the data of the scene - it provides the same data as the scenegraph but in a flat and fast way. Therefore, if a controller changes data in the scenegraph, these changes are committed to the scene database to update its internal state.

To synchronize the data view of the scenegraph with the scene database, an event-based system is used. The processing sequence from a registered change in the scenegraph to an update in the scene database may have the following steps:

• Generating a list of tasks which have to be performed on the scene database once data of a node in the scenegraph has changed.

• Sending, from the node, an event to the global scenegraph context containing the event ID indicating what type of data has been changed and the node itself as sender object.

• Processing all raised events at the beginning of a frame of the global scenegraph context.

• Generating, for every event, tasks and collecting the generated tasks in a list.

• Searching the list of all tasks for duplicate tasks and sorting the lists. For example, the world transformation matrices of the nodes need to be updated before their bounding volumes in world space can be recalculated and the task addressing the former is sorted before a task addressing the latter.

• Executing the list of tasks.

Using this concept, redundant work per scenegraph node is prevented. This is achieved by the elimination of duplicate tasks as listed above. Another benefit is that a task is only performed on nodes, which need to update their data. For example, the system only updates the structural context of those nodes, which are influenced by a data change in the scenegraph. To generate a list of tasks for a sent event, an interface is introduced where every implementation class represents one event ID. Every concrete class generates only the tasks that are necessary to process the event it represents. The generated task is only a class instance which holds a task code as listed above along with a list of nodes. When the task is executed the operations of the task are only applied to that list of nodes.

Every task code has a corresponding class instance, which can only execute one specific task. These execution classes are all supporting a corresponding interface. Thus, when the global scenegraph context executes a specific task, the latter is searching for an object which can handle the code of the task. Then the Execute()-method of the returned object is called. This method gets the list of nodes as parameter on which the executer should operate. Both the task generator and the task executer concept are following the command design pattern.

The visitor pattern implemented using double dispatch is also integrated into the scenegraph. This enables support of algorithms of other scenegraph frameworks and easy porting to the scenegraph according to the rendering engine implemented. The task executer for updating the structural context of nodes is using the visitor pattern to collect the data from the scenegraph.

Furthermore, the concept of a resource manager is introduced.