This invention relates to controlling images displayed by adaptive backdrops such as LED walls or projection screens.

When video is being captured, for example for movie filming or current affairs broadcasts, it is becoming increasingly common to display a changing backdrop behind a subject. The subject may be an actor or a presenter. The backdrop may represent a set or a location where the subject is to be depicted. The backdrop can be displayed using a display wall (e.g. formed of multiple LED display panels) or using a projection screen. One advantage of this approach is that it avoids the need for the subject to be in the said location, and allows the location to be a fantasy or historic location. In comparison to the green screen method, in which the backdrop is inserted in video post-processing, it reduces the need to for such post-processing and makes it easier to use the technique for live events.

It is desirable for the image displayed on the backdrop to appear realistic from the point of view of the camera that is being used to capture the video. One known way to accomplish this is to store a three-dimensional model of the scene that is to be the backdrop, to sense the location of the camera relative to the wall or screen on which the backdrop will be displayed and then to form an image of the scene as it would appear on the wall/screen from the location of the camera. The image is then passed to the wall for display, or to a projector for projection onto the screen.

One problem with this approach is that it takes some time for the image to be formed once the location of the camera is known. When the camera is moving, this time delay causes a lag in the change of the image once the camera has moved to a new location. This can be perceived by a viewer and can make the backdrop seem less convincing. It would be desirable to have an improved mechanism for forming adaptive backdrops so that camera motion may be better accommodated.

According to the present invention there is provided apparatus and methods as set out in the accompanying claims.

The display controller may comprise one or more processors configured to execute code stored in non-transient form to execute the steps it is configured to perform. There may be provided a data carrier storing in non-transient form such code.

The rendering engine may comprise one or more processors configured to execute code stored in non-transient form to execute the steps it is configured to perform. There may be provided a data carrier storing in non-transient form such code.

The present invention will now be described by way of example with reference to the accompanying drawing.

In the drawing:

Figure 1 shows a system for capturing video of a subject against a backdrop.

Figure 2 shows an oversized image and a subset of that image of a size appropriate for display on a display wall.

Figure 1 shows a system that could be implemented to capture video of a subject 1 against a background displayed on a display wall 2. The subject could, for example, be one or more actors or presenters or an inanimate object. The display wall is a structure extending in two dimensions that can display a desired image. It could be a matrix of individual display units, for example LED display panels. A camera 3 is provided for capturing the video. The camera is located so that from the point of view of the camera the display wall 2 appears behind the subject 1 . The camera is mounted, e.g. on a wheeled tripod 4, dolly or articulated boom, so that it can be moved when video is being captured.

A positioning system is provided so that the location of the camera can be estimated. Any suitable positioning system can be used. In one example, a sensor 5 mounted on the camera senses the position of markers or transmitters 6 in the studio, and thereby estimates the location of the camera. The positioning system may, for example, be as described in EP 2 962 284. The position of the camera may be estimated at the camera or remotely from the camera.

In the example of figure 1 , the sensor 5 estimates the position of the camera and transmits that position to a rendering unit 6. The rendering unit 6 comprises a processor 7 and has access to a memory 8 storing in non-transient form code executable by the processor to cause the rendering engine to operate as described herein. The rendering engine also has access to a scene memory 9. The scene memory stores data defining a scene in such a way that from that data the scene can be portrayed from multiple locations. Conveniently, the scene may be defined by data that defines the three-dimensional structure and appearance of objects in the scene. Such data allows it to be determined how those objects would appear relative to each other from different locations. The rendering engine may implement a rendering engine such as Unreal Engine. The rendering engine receives the camera’s position. The rendering engine has previously been provided with information defining the location of the display wall 2. With that information the rendering engine can determine what image needs to be displayed on the wall so that the backdrop will present the scene accurately from the point of view of the camera. This may, for example, be done by tracing rays from the camera position through the wall to the scene as it is defined in an imaginary space behind the wall, and thereby determining the colour, brightness etc. of each element (e.g. pixel) that can be displayed on the wall. Once that image has been determined it is passed to a display controller 10 local to the wall. The display controller controls the wall to display the desired image. If instead of a display wall a projection screen is used, the display controller may be local to a projector which projects the image on to the screen.

The camera 3 captures video data which is stored in a video data store 11 . From there it can be edited and/or post-processed and broadcast if required.

As indicated above, it would be desirable for the system to be able to respond quickly to changes in the position of the camera 3.

One way in which this may be done will now be described. The rendering unit 6 is configured to form an image that extends beyond the edges of the wall 2 at the scale at which the image is to be displayed. This may be done by tracing rays from the camera position through points that lie beyond the edges of the wall so as to determine what colour, brightness etc, would be represented in the scene at those locations. This is illustrated in figure 2. Boundary 20 indicates the margin of the image that is generated. Boundary 21 indicates how much of that image can be displayed on the wall at the desired scale. It will be seen that there are regions within boundary 20 but not within boundary 21 . Depending on the complexity of the scene and the processing power of the rendering unit, it may take some perceivable time to generate the image. Successive images may be generated several times per second, for example 30 or 60 times per second. The frequency may be selected depending on the frame rate of the camera, the frame rate of the wall and the expected speed of motion of the camera.

When the oversized image has been formed by the rendering engine it is transmitted to the display controller 10. The display controller also receives the position of the camera. Due to lag in generating the oversized image and transmitting it to the display controller the position of the camera may have changed since the oversized image was generated. In dependence on the latest position of the camera that it has received, the display controller selects a subset of the oversized image and optionally applies a geometric transformation so as to form an adapted image. The display controller then causes the wall to display the adapted image. The framing of the oversized image and any geometric transformation can be quicker operations than that of forming the oversized image. In this way, the display controller can cause the image that is displayed to be adapted promptly to the position of the camera. This can help to reduce any perception that the background to the subject 1 is unnatural.

The display controller 10 has a processor 12 and a memory 13. The memory 13 stores in non-transient form code executable by the processor 12 so that the display controller can perform its functions.

The adaptation of the oversized image will now be described.

In one example, the rendering engine provides the oversized image to the display controller together with an indication of the camera position for which the oversized image was formed. The display controller can then apply an algorithm to frame (i.e. select a subset of) and optionally apply a geometric transformation to the oversized image (or the selected subset of it). For example, if the camera has traversed horizontally by a certain amount, then the display controller may select a subset of the display that is correspondingly offset horizontally from the centre of the oversized image. Or if the camera has traversed vertically by a certain amount, then the display controller may select a subset of the display that is correspondingly offset vertically from the centre of the oversized image. The resulting image may be an imperfect rendering of the scene from the current point of view of the camera, but if the camera is moving at a reasonable speed compared to the frame rate of the display system, any errors can be expected to be minor.

In the discussion above, the rendering engine and the display controller receive the position of the camera. They may also receive the direction of the camera (e.g. the direction of the central axis of the camera’s lens). They may also receive information about the properties of the camera’s lens. They may also receive information about the state of the camera’s lens, for example its zoom and/or aperture setting. Any one or more of these factors may be employed by the rendering engine to form the image and/or by the display controller to adapt that image. Information about the camera other than its position that was used to generate the image may be transmitted by the rendering engine to the display controller to permit the display controller to adapt for changes in the camera’s set-up since the image was formed.

The display controller may have information about the location of the screen so that it can establish the location of the camera relative to the screen. That information may also be used to affect the adaptation of the oversized image. For example, when the camera has undergone translation, the amount by which the centre of the adapted image is shifted from the centre of the oversized image may vary depending on the distance of the camera from the wall.

In one example, the camera may have rotated between the image being formed and it being processed by the display controller. In that situation the display controller may apply a suitable trapezoidal or other geometric transformation to the image or part of it to form the adapted image for display.

In some situations the image as formed by the rendering engine may be stretched by the display controller to form the image for display. Then, if desired, the display controller may employ an algorithm to interpolate between pixels of the oversized image to form the adapted image.

It has been found that the present system is especially valuable for accommodating translation of the camera relative to the wall or screen.

The rendering engine may provide the display controller information about the depth of objects depicted at respective locations in the image. This may, for example be provided as a depth map in which for each pixel or block of pixels in the image a value is specified indicating the depth or aggregate or average depth of locations depicted in that pixel or block. The display controller may apply a transformation to parts of the image in dependence on the indicated depth information. For example, a greater shift responsive to camera motion may be applied for pixels or blocks at a greater depth. Interpolation and/or culling of pixels and/or inpainting algorithms can be applied as appropriate to maintain a desired pixel density in the adapted image.

The display controller could be local to the display wall or projector or remote from it. It could be integrated into the wall or to one or more display panels. It could be integrated with the rendering engine.

The camera position sensor could signal the camera’s location directly to both the display controller and the rendering engine as illustrated in figure 1 . Alternatively it could signal the camera’s location to one of those entities, which could then forward it to the other. Other information about the state of the camera, such as its direction and the state of its lens, could be signalled in the same way.

The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

The phrase "configured to" or “arranged to” followed by a term defining a condition or function is used herein to indicate that the object of the phrase is in a state in which it has that condition, or is able to perform that function, without that object being modified or further configured.