The invention relates to a video composition device.

In the post-production domain, the use of computer type common computing platforms is widespread, to the detriment of dedicated platforms, composed of specialised hardware.

The computing platforms lead in fact to greater flexibility and benefit from a mass market, and thus lower prices.

In the real-time production world, the use of dedicated platforms is further privileged due specifically to insufficient bandwidth in the computing platforms available on the market.

In fact, to compose video in real-time is extremely costly in terms of calculation resources and bandwidth on the bus, and the actual components and architecture are under-dimensioned to respond to these constraints. It is currently impossible to convey more than two HD streams on an express PCI bus and this thus limits the composition possibilities.

A system example according to the prior art is shown in figure 1. This system comprises a motherboard card 9 connected via an express PCI bus 5 to a video acquisition card 10 and via an express PCI bus 6 to a graphic card 2. This latter is connected to display means 3 via the intermediary of a connection of type S-video, HDMI or DVI. The display means can be remote means interfaced by broadcasting means, for example tele-visual. The cards 2, 9 and 10 are preferably cards that can be inserted in a personal computer (PC).

The graphic card 3 comprises at least a graphics processor (GPU) 210 as well as a graphics memory 220 of RAM (Random Access Memory) type or equivalent.

The motherboard card 9 comprises at least a processor (CPU) 101 , commercialised for example by the company Intel of "Pentium" type and its two associated components 103 (northbhdge) and 104 (southbridge). These two components are for example respectively commercialised under the references 82975X MCH and 82801 GR ICH7R of the company Intel.

The motherboard card 9 also comprises a memory 901 of RAM type intended to record at least the video data received from the acquisition card 10 and to contain the code instructions suitable to operate the processor 101 as well as the variables and constants required to run this programme as well as a memory 902 of ROM type (Read Only Memory) intended to contain the instructions required during the start-up of the processor 101.

A user interface 8 is also connected both to the motherboard card 9 as well as to the graphics card 2. This is a diagrammatic view. The user interface is in fact a programme for which the code is recorded in the memory 901. This user interface is preferably displayed on a screen (not shown) that is for example the personal computer screen comprising the cards 2, 9 and 10. A user controls then this interface and transmits commands and data to the graphics card 3.

The video acquisition card 10 comprises an acquisition module 402 that receives at input a plurality of video streams (video 1 , ...video n), and emits at output a plurality of video streams transmitted via the intermediary of the bus 5 to the motherboard card 9. This card fulfils essentially the role of synchronization of incoming video streams with the computing system that takes control of them via the express PCI buses. This module is constituted of FIFO (First In First Out) type memory, for which the input is at the rhythm of the input video clock and the output is at the rhythm of the express PCI bus clock.

When the video streams are of High Definition (HD) type, it is impossible to transfer more than two streams between the acquisition card 10 and the motherboard card 9 in real-time. Likewise between the motherboard card 9 and the graphics card 2. Thus, such a system does not enable the graphics card 2 to compose a video output comprising a plurality of high definition videos, or of more than two high definition videos. In fact, a high definition image requires a bitrate of 3Gb/s. In a mixing application as implemented on the graphics card, it is desirable to be able to mix up to 6 videos, which could imply bitrates of 18 Gbit/s. The internal buses do not enable such a bitrate. A possible solution is to compress the video streams on the acquisition card. However, this introduces heavy processing monopolizing calculation resources, introduces additional delays and degrades the quality of the video signals. The invention proposes to overcome this lack of resources by proposing a video composing device comprising:

- rendering means intended to compose a video from a plurality of videos received,

- interface means receiving information relating to the composition of said videos and transmitting a first command to the rendering means.

According to the invention, the device comprises

- reformatting means receiving at input said plurality of videos and intended to reformat said plurality of videos, and transmit said plurality of reformatted videos to said rendering means via at least one data bus,

said interface means transmitting a second command to the reformatting means, said second command being intended to control the reformatting means of said plurality of videos so as to reduce the quantity of data transmitted on the at least one data bus.

Thus, the use of a standard computing platform can enable transport of video at a bitrate compatible with the requirements of video composition and advantageously when it involves real-time video composition.

Contrary to previously received ideas, advantageously the invention enables the graphics processor to be discharged. In fact, the prior art advantageously profits from the capacities of the graphics processor to assign to it all of the composition tasks and this must as a consequence convey significant volumes of data on the computing buses. The inventors, contrary to received ideas, have inventively thought that a processing of the video prior to the composition would enable the bitrate required on the computing buses to be reduced and would thus enable them to be able to convey more video streams. This runs contrary to preconceived ideas where the capacities of graphics processors are fully exploited, by making other modules carry out some tasks.

Advantageously, said second command comprises an item of information relating to the size of each video in the recomposed video, - said at least one data bus having a bandwidth known to the reformatting means, the reformatting means reformatting said plurality of videos according to said information and a recommended (setting) bitrate of said bus.

Preferably, the reformatting means comprise filtering means.

Preferably, the reformatting means comprise cropping means.

Preferably, the cropping means are controlled via the interface means.

According to a preferred embodiment, the cropping means are controlled automatically via detection means of salient areas of each video of the plurality of videos. Advantageously, the first command comprises information relating to the final size of each input video in the video composed by the rendering means.

Advantageously, the first command comprises instructions relating to the enlarging or the reduction of at least one of the input videos.

The invention will be better understood and illustrated by means of embodiments and advantageous implementations, by no means limiting, with reference to the figures in the appendix, wherein: figure 1 shows a device according to an embodiment of the prior art,

figure 2 shows a device according to a preferred embodiment of the invention,

figure 3 shows a reformatting means according to a preferred embodiment of the invention,

figure 4 shows a filter used in the reformatting means according to a preferred embodiment of the invention, figure 5 shows the method implemented in the reformatting means, according to a preferred embodiment, figure 6 shows a video image obtained according to a preferred embodiment of the invention.

Figure 2 shows a device 1 according to a preferred embodiment of the invention wherein the invention is advantageously implemented in a personal computer type electronic platform and based on electronic components commercialized by the Intel company.

This embodiment is in no way limiting and the invention can also be implemented in other platform types based on different components.

Elements common from one figure to another have the same references. This system comprises a motherboard card 1 connected via an express

PCI bus 5 to a video acquisition card 4 and via an express PCI bus 6 to a graphics card 2. This latter is connected to display means 3 via the intermediary of a connection of type S-video, HDMI or DVI. The display means can be remote means interfaced by broadcasting means, for example tele-visual. The cards 1 , 2 and 4 are preferably cards that can be inserted in a personal computer (PC).

The graphics card 2 comprises at least a graphics processor (GPU) 210 as well as a graphics memory 230 of RAM type.

The motherboard card 1 comprises at least a processor (CPU) 101 , commercialised for example by the company Intel of "Pentium" type and its two associated components 103 (northbridge) and 104 (southbridge). These two components are for example respectively commercialised under the references 82975X MCH and 82801 GR ICH7R of the company Intel.

The motherboard card 1 also comprises a memory 102 of RAM type intended to record at least the video data received from the acquisition card 4 and to contain the code instructions suitable to operate the processor 101 as well as the variables and constants required to run this programme as well as a memory 105 of ROM type (Read Only Memory) intended to contain the instructions required during the start-up of the CPU 101. A user interface 7 is also connected both to the motherboard card 1 as well as to the graphics card 2 and the video acquisition card 4. This is a diagrammatic view. The user interface is in fact a programme for which the code is recorded in the memory 105. This user interface is preferably displayed on a screen (not shown) that is for example the personal computer screen comprising the cards 1 , 2 and 4. A user controls then this interface and transmits commands and data to the graphics card 3 and especially to the video acquisition card 4.

The graphics interface transmits notably a first command to the graphics card 2 and a second command to the graphics card 4.

The first command is relative to the final rendering of the plurality of input videos and provides at least one item of information relating to the final size of each input video in the final display. Notably, this command may contain instructions relating to the re-dimensioning, to the enlargement as well as to the reduction, of at least one of the input videos. This may for example be the case when an input video is in simple format definition (for example 720 pixels by 576 lines or 720 pixels by 480 lines) and there is a desire to display it in high definition (1080 pixels by 1920 pixels) in the final display. In this case, the graphics processor performs an extrapolation and this avoids transferring on the internal buses 5 and 6 videos requiring a bandwidth that is too large.

The second command relates to an intermediary rendering, enabling the quantity of data on the internal buses 5 and 6 to be reduced. It enables the size of incoming videos to be modified and more specifically to be reduced, particularly when they are in high definition format, in a way to obtain an intermediary size between the size at input from the acquisition card 4 and the size at output from the graphics card 2. The video acquisition card 4 comprises an acquisition module 420 and a reformatting module 410. It also receives at input a plurality of video streams (video 1 ,...,video n), and emits at output a plurality of reformatted video streams transmitted via the intermediary of the bus 5 to the motherboard card 1. The invention is particularly implemented in the reformatting means 410 of the acquisition card 4.

The reformatting means receive at input a plurality of video streams, of which 5 are shown as a non-restrictive example in figure 2.

The acquisition card 4 also receives from the user interface a second command, as indicated previously, enabling the control of the reformatting means 410. This user interface enables the parameters of the size of the videos and the centring or cropping of these videos to be set by the intermediary of this second command so that they require less bandwidth on the express PCI bus 5.

The reformatting means are described non-restrictively in reference to figures 3 and 4.

According to figure 3, the reformatting means 410 comprise a set of filters 4101 to 410n, a filter being necessary for each video input. The output of each filter is connected to an input of a multiplexor 41 1. Each filter receives at input the commands from the user interface 7.

The input video can be viewed by the operator on control screens generally available in the studio infrastructure, but the user interface may also integrate a monitoring application of each input video. The composition being prepared by the operator is generally viewed via a specialised composition path, called "pre-visualisation", for which a screen is dedicated and that enables the operator to adjust his composition before broadcasting. This pre- visualisation can also be integrated in the user interface and use the computer screen.

The filters 4101 to 410n are bidirectional filters described in reference to figure 4.

Each filter comprises a buffer memory 41 1 1 at input intended to receive videos at input. This buffer memory has a capacity for at least one line of the full resolution, high definition video image.

Each two-dimensional filter is broken down into two cascading independent filters. The first is purely vertical and enables the re-dimensioning of the image in its height, it is constituted of delay lines 4130, 4140, 4150, 4160 (shown), of multiplexors 4131 , 4141 , 4151 , 4161 , of adders 4122, 4132, 4142, 4152 and memory 41 13 for the sub-sampling. The second is purely horizontal and enable the re-dimensioning of the image in its width, it is constituted of delay registers 4170, 4180, 4190 (shown), of multipliers 41 16, 4171 , 41 17, 4181 , 4191 and adders 4172, 41 18, 4182, 4192. The pixel sub- sampling is integrated into the cropping module itself comprising a storage buffer. Each line exiting the buffer 41 1 1 is transmitted to a first delay line 4120, to a succession of delay lines 4130, 4140, 4150, 4160 (shown) and to a plurality of delay lines not shown.

The output of each delay line 4120, 4130, 4140, 4150, 4160 is respectively connected to a multiplier for which the multiplication coefficient is from the control module 4114.

The control module receives the commands from the user interface 7 shown in figure 2.

When the user interface transmits a command requesting a reduction in the resolution of the input video image, the control module 41 14 transmits a command to the various multipliers 4121 , 4131 , 4141 , 4151 , 4161.

Let us take for example a reduction in the size of the image by 4.

When for example, the user requests a reduction in the size of the image by 4, he indicates it via the user interface to the control module 41 14. This control module will then transmit to the multipliers 4121 , 4131 , 4141 , 4151 , 4161 carrying out a line filtering, the order to select one line in two per image line. Then the control module will also control the column filtering and transmit to the multipliers 41 16, 4171 , 4117, 4181 , 4191 , the order to select one pixel in two in each selected line. The adders 4122, 4132, 4142, 4152, 4162 at the output of the multipliers 4121 , 4131 , 4141 , 4151 , 4161 enable the field as well as the adders 4172, 41 18, 4182, 4192 at the output of the multipliers 4116, 4171 , 41 17, 4181 , 4191 to be reconstituted. Moreover, a buffer memory 41 13 enables the data at output from the adders 4122, 4132, 4142, 4152, 4162 to be delayed before the column filtering.

The quality of filtering desired will determine the width of filters to be implemented. A filter width implicating ten horizontal and vertical coefficients enables a good filtering quality to be obtained suitable for professional applications. This implies nine delay lines, nine registers, twenty multipliers and 18 adders.

The control module 4114 also controls the cropping means 41 15. The cropping means enable just a part of the image to be retained. Cropping is compatible with filtering. It is possible that the user transmits both a filtering command (for example a reduction by 4 of the image size) and a cropping command, per selection, of graphic type for example, of a part of the image. It is also possible that the user does not specify a cropping command. In this latter case, the multiplexers let all the pixels of the image pass.

The cropping command can be made manually via the graphical interface. However it should be noted that in the production domain, different special effects are used and the producers favour manual selection of areas of the image to be conserved.

The user selects then manually an area of the image that he wants to see displayed. This is easily done, for example by using a mouse controlling the graphical interface and outlining an area of the image. This manipulation can be carried out iteratively on the different input videos.

However, in other advantageous embodiments, an automatic selection of areas to be retained can be envisaged. To do this, the control means 41 14 can receive a cropping command from the user but this command does not specify the area of the image to be retained. This command can also specify the desired size, in terms of length by width. The cropping means are then responsible for selecting a part of the image corresponding to this size. Advantageously, this area is centred on the areas of interest or salience. Different salience area detection algorithms are known to those skilled in the art and for example described in the European patent applications EP 1695288 filed in the name of the company Thomson Licensing and published June 30, 2005. The filtering and control means 41 14 comprise then the salience area detection means (not shown).

According to the bandwidth admissible on the bus 5 or recommended (setting) bitrate, the user is constrained to respect certain reformatting rules and the system does not authorise reformatting commands giving too great a bitrate to be transmitted on the bus 5. These constraints are naturally also conditioned by the number of videos at input.

Hence, when the user selects three videos, the system authorises more bitrate for each of these videos than when he selects 5 or 6 videos. The user interface thus has its parameters set, and is limited by the performances desired in terms of bitrate.

If the user transmits a filtering and cropping command to the video that he has selected and these commands lead to a bitrate less than the recommended bitrate of the bus, he is informed and can reduce the severity of the filtering and/or cropping.

Conversely, if the bitrate is greater than the recommended bitrate, he can not transmit the commands.

It is noted thus that the reformatting depends on the commands transmitted for each input video but also on the recommended bitrate of the bus.

The video output of each filtering means 4101 , 4102,...41 On, is connected to a multiplexer 41 1 , that transmits the different reformatted video streams to the motherboard card 1 via the intermediary of the express PCI bus 5.

This multiplexing uses the exchange protocol of the standard Express PCI, each filter output 4101 , 4102,...41 On occupies alternatively the express PCI bus 5 to constitute a temporal multiplexing.

The videos thus reduced are transmitted from the acquisition card 4 to the motherboard card 1 and recorded in the memory 102 of the motherboard cardi .

The data are then transferred from the memory 102 to the graphics card 2 via the express PCI bus 6 and transferred into the RAM type graphics memory 230. The graphics processor (GPU) 210 carries out the display of the video for the display. To perform the composition of the scene, the graphics processor 210 uses a graph of the scene describing a scene composed of surfaces representing the arrangement of video streams. For each of the surfaces corresponding to a video stream, a specific node of the graph is added containing information enabling the stream to be located in the graphics memory, as well as processing the video stream. The video streams are placed on the surfaces of the scene composed by the graphics processor. Figure 5 shows a method according to a preferred embodiment of the invention.

During a step E1 , a video acquisition is made. This video acquisition is for example made via a camera. The video can also be received via a network.

During a step E2, a user transmits reformatting commands. These reformatting commands can introduce filtering, cropping, each separately, one or the other or a combination of the two.

During a step E3, each stream is reformatted according to the command received.

Each stream can receive a different command. In fact, it is possible that one video is more important for the image compositor, and that he wants to give it more importance filtering it less or giving it a more important place in the final composition.

During a step E4, the reformatted streams are multiplexed to be transmitted to the motherboard card 1 during a step E5 via the intermediary of the express PCI bus 5.

Then, during a step E6, the multiplexed streams are registered in the memory 102 of the motherboard card 1. These streams are then transmitted to the graphics card 2 during a step E7 and registered in the graphics memory 230. During a step E8, the graphics processor 210 carries out the graphical composition and the final display according to the commands of a user

(director) transmitted via the intermediary of the graphical interface 7.

To perform the composition of the scene, the graphics processor 210 uses a graph of the scene describing a scene composed of surfaces representing the arrangement of video streams. For each of the surfaces corresponding to a video stream, a specific node of the graph is added containing information enabling the stream to be located in the graphics memory, as well as processing the video stream. The video streams are placed on the surfaces of the scene composed by the graphics processor. During a step E9, the final display is transmitted to the display means 3 that may moreover be remote.

Figure 6 shows a diagrammatic view of the final display comprising three reformatted videos V"1 , V"2, V"3. The final compositor carries out the rendering as explained above choosing the emplacement of different videos in the final display. He can for example, as indicated in this figure, encrust these three videos in an image comprising a television news presenter. The invention is not limited to the embodiments previously described.

In particular, it also relates to a device in which the graphics card is integrated in the motherboard card. For reasons of compatibility with existing platforms, the invention is described in the framework of a standard personal computer type architecture comprising a motherboard card.

Moreover, the graphics card 2 constitutes what is commonly called the display means. This graphic card effectively enables, in addition, the rendering of data intended for the visualisation means.