Field of invention

This invention relates to a system, apparatus and method for encoding, decoding and transmitting video.

Background

Creating a consistent high quality visual experience across all modern devices has been a difficult task - in particular for 3D real-time graphics. In general, a user is required to install an application simply to interact with the experience which users are often not willing to do. Furthermore, even after installing a bespoke application, the output quality may vary greatly in dependence on the user's device. An improved solution is therefore required.

The present invention

According to one aspect of the invention there is provided a method of streaming graphics, the method comprising: transmitting a graphics stream from a server to a client device; receiving feedback relating to the rendering of the graphics stream; and modifying said graphics stream in dependence on said feedback; wherein said graphics stream and said feedback are transmitted via a bidirectional transport socket. In such a way, a fast, efficient and graphics stream with which a user can interact in real-time is provided.

Optionally, the feedback comprises user input. Optionally, the graphics comprise 3D graphics. 3D graphics can be relatively more data-intensive than other types of graphics so the method described herein is particularly well suited to 3D graphics. Optionally, the graphics comprise interactive computer generated graphics. Optionally, the user input relating to the rendering of the graphics comprises adjustments to the presentation of the graphics. In such a way the user can interact with the graphics. So as to provide a simple and effective user experience, the user input may comprise a user device orientation, an accelerometer measurement, user interacting with a touchscreen, and/or a user interacting with a mouse / keyboard. For efficiency, the bidirectional transport socket may be kept open for the duration of the transmission. For efficiency of data transfer, the method may further comprise transmitting metadata relating to the graphics stream via said bidirectional transport prior to transmitting said graphics stream. For efficiency of data transfer, the method may further comprise the step of encoding the graphics stream prior to transmission. Optionally, the encoding process does not reference future frames. For efficiency of data transfer, the graphics stream may comprise a single header for a streaming session. For efficiency of data transfer, the method may further comprise detecting a change in streaming parameters and transmitting a further header relating to said new streaming parameters. For efficiency of data transfer, the method may further comprise the step of packetizing the data stream prior to transmission. Optionally, said transmitted packets are reassembled by the client device and played in sequence. For interoperability, said transmission may utilise the Hypertext Transfer Protocol (HTTP).

For interoperability and ease of use, the graphics stream may be provided to the client device via a web browser. So as to enable performance maintenance, the feedback may comprise data relating to transmission performance. According to another aspect of the present invention there is provided a method of streaming graphics, the method comprising: transmitting a graphics stream from a server to a client device via a bidirectional transport socket; receiving data relating to transmission performance via a bidirectional transport socket; and modifying at least one property of said transmission based on said data relating to transmission performance.

In such a way, properties of the graphics stream can be dynamically adjusted Όη- the-fly' so as to maintain optimum performance for current network conditions. Optionally, said modifying comprises adjusting encoding of the graphics stream. So as to easily adjust the data rate of the stream, adjusting the encoding comprises modifying a number of frames per packet, modifying the output resolution, modifying the stream bitrate, modifying the keyframe interval, and/or modifying the client side buffer.

Optionally, the method further comprises monitoring the performance of said transmission. So as to determine initial conditions, the method may further comprise measuring the bandwidth of a network; and determining initial transmission properties based on said measured bandwidth.

Optionally, the method may further comprise transmitting a header relating to the transmission performance.

According to another aspect of the present invention there is provided a method for streaming graphics comprising; determining the presence of an augmented reality marker; determining a projection matrix relating to the augmented reality marker; transmitting said projection matrix via said bidirectional transport socket; and receiving an augmented reality graphics stream via said bidirectional transport socket.

Optionally, the step of determining the presence of an augmented reality marker comprises analysing an image. So as to enable augmented reality over video, the step of determining the presence of an augmented reality marker may comprise analysing a video feed.

For efficiency of computation, the method may comprise transmitting said detected augmented reality marker via said bidirectional transport socket for analysis. In such a way, a device with greater computational power may be used to analyse the augmented reality marker. Furthermore, the augmented reality marker may comprise transmitting data related to a portion of a video or image.

For an improved user experience, the method may further comprise amending the received augmented reality graphics stream. Optionally, said amending comprises removing virtual green screen elements; optionally removing the virtual green screen elements comprises chroma keying.

According to another aspect of the present invention there is provided a method of streaming graphics, the method comprising: receiving a projection matrix relating to an augmented reality marker via a bidirectional transport socket; generating, using said projection matrix, an augmented reality graphics stream; and transmitting the augmented reality graphics stream via said bidirectional transport socket.

Optionally, the step of generating, using said projection matrix, an augmented reality graphics stream may comprise parsing the projection matrix into a 3D application and display model.

For ease of use and/or efficiency, the method may further comprise generating a virtual green screen to apply the received augmented reality marker to. For efficiency, the method may further comprise receiving a detected augmented reality marker for analysis. For efficiency, the method may further comprise analysing said received augmented reality marker so as to generate information identifying the presence of the augmented reality marker So as to ensure alignment of the augmented reality stream, the projection matrix may comprise information relation to the orientation of the augmented reality marker. Optionally, the orientation of the augmented reality marker may comprise a relative orientation of the augmented reality marker with respect to a camera on a device. Optionally, the orientation of the augmented reality marker may comprise a relative orientation of the augmented reality marker with respect to a virtual camera of the graphics stream.

The method may further comprise the steps of: connecting a first and second user; and wherein the transmitted graphics stream comprises a first user's session.

According to another aspect of the present invention there is provided a method of streaming graphics, the method comprising: connecting a first and second user; transmitting a first user's session to the second user; wherein said transmission occurs via a bidirectional transport socket.

In such a way, a first user may share their screen session with a second user in a fast, efficient and user-friendly manner. Optionally, said first and second user are connected via a server. In such a way, a server can control the streaming settings. For ease of use, the method may further comprise allowing remote control of the first user's session to the second user. So as to enable second user control, the method may comprise receiving input from second user, via said bidirectional transport socket, said user input controlling the first user's session. For security, the step of connecting a first and second user may comprise providing a unique session identification from a first user to a second user. For ease of use, said unique session identification may comprise a link. For efficiency, the method may further comprise selecting an application server for transmitting the first user's session to the second user, the application server being selected based on geographical distance to the first user. For efficiency, the method may further comprise modifying the transmission of the first user's session in dependence on the device being used by the second user. For efficiency, the method may further comprise modifying the transmission of the first user's session in dependence on the network connection to the second user.

According to another aspect of the present invention there is provided system for streaming graphics, the system comprising: means for transmitting a graphics stream from a server to a client device; and means for receiving user input relating to the rendering of the graphics stream; and means for modifying said graphics stream in dependence on said user input; wherein said graphics stream and said user input are transmitted via a bidirectional transport socket. According to another aspect of the present invention there is provided a system for streaming graphics, the system comprising: means for transmitting a graphics stream from a server to a client device via a bidirectional transport socket; means for receiving data relating to transmission performance via a bidirectional transport socket; and means for modifying at least one property of said transmission based on said data relating to transmission performance.

According to another aspect of the present invention there is provided a system for streaming graphics, the system comprising: means for determining the presence of an augmented reality marker; means for determining a projection matrix relating to the augmented reality marker; means for transmitting said projection matrix via a bidirectional transport socket; and means for receiving the augmented reality graphics stream via said bidirectional transport socket.

According to another aspect of the present invention there is provided a system for streaming graphics, the system comprising: means for receiving a projection matrix relating to an augmented reality marker via a bidirectional transport socket; means for generating, using said projection matrix, an augmented reality graphics stream; and means for transmitting the augmented reality graphics stream via said bidirectional transport socket.

According to another aspect of the present invention there is provided a system of streaming graphics, the system comprising: means for connecting a first and second user; means for transmitting a first user's session to the second user; wherein said transmission occurs via a bidirectional transport socket.

Optionally, the system further comprises: a first user client device; and a second user client device, and optionally further comprising a server.

The invention extends to a client or user device in the form of a telecommunications device or handset such as a smartphone, tablet or other, preferably mobile, computing device adapted to carry out the methods described herein.

According to another aspect of the present invention there is provided a system for streaming graphics, the system comprising: a server; and at least one client device; wherein the system is adapted to carry out the method as described herein. The invention extends to any novel aspects or features described and/or illustrated herein. Further features of the invention are characterised by the other independent and dependent claims. Any feature in one aspect of the invention may be applied to other aspects of the invention, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa.

Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly. Any apparatus feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory. It should also be appreciated that particular combinations of the various features described and defined in any aspects of the invention can be implemented and/or supplied and/or used independently. According to another aspect there is provided a computer program and a computer program product comprising software code adapted, when executed on a data processing apparatus, to perform any of the methods described herein, including any or all of their component steps.

According to another aspect there is provided a computer program and a computer program product comprising software code which, when executed on a data processing apparatus, comprises any of the apparatus features described herein. According to another aspect there is provided a computer program and a computer program product having an operating system which supports a computer program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein. According to another aspect there is provided a computer readable medium having stored thereon the computer program as aforesaid. According to another aspect there is provided a signal carrying the computer program as aforesaid, and a method of transmitting such a signal. The invention extends to methods and/or apparatus substantially as herein described and as illustrated in the accompanying drawings.

As used herein, the term 'graphics' or 'digital graphics' may mean computer- generated (or computer-augmented) visual data. For example, three dimensional (3D) graphics are 3D models to be presented to a user who may interact with the model by changing the viewing angle (for example). In one example, the graphics being streamed are a video.

Many devices are not able to directly present 3D graphics, but use 3D graphics so as to render 2D images which are compatible with the presentation device (such as a 2D screen).

The present invention aims to provide 'console quality' interactive 3D visual experiences into any modern browser on any modern device, with no need for installing plugins or applications. Thus, the invention may be described as "browser agnostic". In such a way, 3D applications can be embedded into a website as easily as embedding a video. This technology may also allow streaming of non-graphically intensive applications to allow an application / desktop replication solution for any intensive application.

The invention will now be described by way of at least one example, with references to the accompanying drawings, in which:

Figure 1 shows a networked system including a server and a client device;

Figure 2 is a circular flow-diagram of a graphics streaming method;

Figure 3 shows a flow diagram of initiating a transmission stream;

Figure 4 shows a method of dynamic maintenance of streaming quality;

Figure 5 shows a flow diagram for transmitting an augmented reality stream;

Figure 6 shows a representation of an augmented reality stream overlaid on a camera feed;

Figure 7 shows a method of streaming graphics corresponding to a first user's session to second user; and

Figure 8 shows a method of a first user providing control of a session to a second user.

Detailed description

Figure 1 shows a system comprising a server 100, a client device 102 connected via a network connection 104.

The client device 102 may be in the form of a personal computer (e.g. a desktop or laptop), a portable device such as a mobile (cellular) telephone or tablet, or any suitable, preferably mobile, computing device. The client device 102 has a user input module 106 which enables a user to interact with the client device via one or more user input devices. Examples of user input devices include, but are not limited to: mouse, keyboard, touchscreen, accelerometer (e.g. enabling a user to interact by physically moving the client device), motion sensor, proximity sensor, camera, and microphone.

The client device 102 further comprises a decoder 108 for decoding a received graphics stream and a display 1 10 for displaying the graphics stream to a user. A camera 11 1 may also be provided, for example for enabling augmented reality graphics. The client device 102 also comprises a processor 112 and associated memory 114. The server 100 comprises an encoder 116 and a packetizer 118 which encode and prepare the graphics stream for transmission. These processes are described in more detail below.

The server 100 further comprises a user input module 120 and a network performance module 122. The user input module 120 receives data relating to the user input from the client device 102 via an application 121 "sitting" on the sever 100. The network performance module 122 receives data relating to transmission performance. The server 100 also comprises a processor 124 and associated memory 126.

The network connection 104, in one example, comprises the internet 120 and may comprise a wired or wireless connection (or combination thereof). For example if the client device 102 is a desktop computer, the network connection 104 may comprise an Ethernet connection to the internet. If the client device 102 is a portable device such as a mobile (cellular) telephone, the network connection 104 may be via WiFi™ or via a cellular network such as General Packet Radio Service (GPRS), 3 ^rd Generation Partnership Project (3GGP), Long term evolution (LTE), 4 ^th Generation (4G), or 5 ^th Generation (5G).

The network connection 104 enables a graphics stream to be transmitted to the client; this is performed by negotiating and setting up a connection between a particular socket on the server and client. This reduces the amount of overhead required, decreasing latency and increasing efficiency, as will be discussed in more detail below.

In order for the graphics stream to be modified on the basis of real-time data relating to the graphics stream, data relating to user input and/or network performance must also be transmitted to the server.

It has been found that using a single transport socket for both directions of communication produces a surprisingly effective and efficient system. A second transport socket may be negotiated to carry the data in the reverse direction which provides similar advantages at a marginal cost of an increased setup burden. The term 'bidirectional transport socket' is used herein for two-way communication between server and client which utilises a single, or pair of, transport sockets (communication nodes) for transmission and receipt of data. This is in contrast to, for example using HTTP calls to send control signals back to the server; such a method adds significant overhead to mouse click, movement or keypress (for example), as each message goes through the latency-adding process of negotiating a connection to the server.

When looking at all of possible scenarios for sending data across the web from a web browser to a server and back with the lowest possible delay one must the consider multiple stages that add up to form latency. The first stage is connection time: in the case of using typical HTTP requests, each small piece of information required by the client has a corresponding simple HTTP request. This means for each packet received a separate connection would have to be negotiated before receiving any data, and again for each command to be send (like a mouse click event). This adds an extra overhead to each piece of data being transmitted or received.

An alternative could involve a single direction long lasting socket from the server meaning that the connection only need be negotiated on the initial connection and then maintained whist the client is active. However each control command would still need to be transmitted over a single HTTP call, this would of course reduce the number of connection negotiations by at least half, but it is still not as efficient as one would like.

The solution described herein is to use a single bidirectional long lasting socket, which allows for a single connection negotiation as the beginning of the user's session, and no further connections to be made (baring any network errors, reconnection, etc.). Negotiation of a bidirectional long lasting socket is more difficult than HTTP requests and can add to initial connection time which is one reason why HTTP requests are often preferred. However, overall latency for many applications - in particular the real-time streaming of graphics - can be reduced.

Example latency times for different connection types are provided below:

• No long-lasting sockets:

o Connection time to receive one frame 30ms,

o Connection time to send one frame of input 30ms.

o Single round trip this can add 60ms of latency on top of all of the other points in the system where latency exists. • Single bidirectional long-lasting socket:

o Essentially 0ms receive connection

o Essentially 0ms send connection.

Figure 2 shows a simplified flow diagram for transmitting graphics to a user, with which the user can then interact. In overview, there is provided a loop of transmitting a stream 200, receiving feedback in the form of user input 202, and modifying the graphics stream 204. This loop operates continuously in real-time during the transmission of the stream.

The feedback in the form of user input 202 may comprise the user requesting adjustment to the presentation of the graphics, for example manipulating a 3D object, or 'walking through' a virtual environment. The user input could, for example, take the form of: a user changing the orientation of their device (e.g. tilting to rotate an object), an accelerometer measurement, a user selecting a colour, a user selecting a link (e.g. to move the 'camera' to a different location), and the user interacting with a touch screen, with a mouse, joystick or keyboard. The feedback in the form of user input is sent back to the server side application over a bidirectional transport socket. Simpler inputs such as mouse and keyboard input may be implemented into a server side application without substantive modification. Examples of user input modification would be a user modifying the 'camera angle' or 'zoom' of a 3D video, for example, by virtually walking through a 3D model of a building. Such user interaction could take the form of the user using the arrow keys of the keyboard, or moving the mouse, to indicate how they wish to 'move'.

More advanced input such as gyroscopic/accelerometer device positioning and multi- touch digital input may require some application modification to receive messaging directly from the application streaming server. In the case of a gyroscopic/accelerometer device, the streamed application would need to be able to position the camera based on the gyroscopic/accelerometer input. This may present some user experience challenges that one may not experience with a simpler mouse controlled camera. For example, when using a mouse it is possible to inhibit the camera from pointing in undesirable locations, which may not be possible when controlling the camera via a gyroscopic/accelerometer device. Alongside the video data being transmitted from the server side to the client, with some modification and application can send application data to allow for client side Ul manipulation and other supplemental data to go along with the application stream. The overall use of this solution is to stream real-time applications that could not otherwise be reproduced on the client device without installing specific applications or browser plugins. This means that an application can render 'console quality' 3D visuals using a device with far lower processing power, such as a smartphone. This also allows for additional features to be added to the application experience as the application is streamed from an accessible cloud location; for example multiple users could see the same output and collaborate in real time, or Application Elements can be fused with HTML Website content creating a 'hybrid' application paradigm where some parts are rendered locally and others in the cloud where both platforms strengths can be utilised while avoiding the weaknesses of each platform.

Figure 3 shows a flow diagram of initiating a transmission stream.

The first step 300 is for an initial metadata packet to be created and sent to the client. Sending an initial package with metadata reduces the size of subsequent packages as it avoids the need to include headers. Once an initial metadata packet is received at the client, the client assumes that all further packets received via that socket relate to the same stream. Receipt of the metadata package initiates rendering of a window to present the graphics to the user. A 'loading' image, bar, hourglass or similar may be presented to the user so as to inform them that graphics will shortly start to be streamed.

The next step 302 is for the graphics to be encoded. This step may include capturing data from the client (for example, device orientation etc.) prior to encoding. At encoding time, only data from previous frames (P-Frames) is used to create each sequential frame, as opposed to the widely used 'B-Frame' functionality of many Motion Compensation codecs as we wish to avoid playback buffer that would have to occur as B-Frames reference both previous and future frame data. The graphics are preferably encoded using a standardised video encoding protocol, such as H264 or H265, for example. However, given the real-time nature of the transmission stream, comparison to future frames (i.e. bi-directional frames (B-frames)) are not used, as data relating to previous frames are available; thus l-frames and P-frames may be used. The encoding parameters used may vary on network performance as is described in more detail below in the 'Dynamic maintenance' section.

The next step 304 is to parse and packetize the encoded graphics. In one example each package contains a single encoded frame as a standard container, such as MPEG-4. The process of standardising the package may be termed 'genericizing'. By providing the packages as standard containers ensures maximum compatibility with client devices. It is possible for each packet to comprise more than one frame - such situations will be described in more detail below in the 'Dynamic maintenance' section.

The packetized graphics are then sent to the client via the transport socket 306 where the data within the packet is parsed at step 308. This process determines the format of the graphics (e.g. which encoding process has been utilised). The packetized data is sent down a bi-directional socket connection, which remains open between the client and the server for the duration of the session.

Depending on the capabilities / preferences of the client device, the graphics may be decoded and presented to the user in different ways. One method of decoding and presentation to the user 310 comprises the parsed data being passed directly to the native media source extension to allow native video playback using hardware decoding. The graphics are then rendered to the native HTML5 Video tag at step 314.

Another method of decoding and presentation to the user 312 comprises decoding the frames using software running in a browser on the client device. The software may include WebGL (Web Graphics Library) which is JavaScript™ application program interface (API) for rendering interactive 3D and 2D graphics within any compatible web browser without the use of plug-ins. WebGL does so by introducing an API that closely conforms to OpenGL ES 2.0 that can be used in HTML5 <canvas> elements.

The choice of decoding methods is typically constrained by the capabilities of the device/browser being used. The main advantages of using the built-in hardware decoder are improved performance, battery life and quality/efficiency. Using programmatic client side decoding incurs a high CPU cost, which can effect battery life and efficiency, Some platforms are limited to lower resolutions when using client side decoding as they don't allow fast enough processing to decode all pixels for high resolutions. However, the client side decoding method will work across almost any modern device/browser.

The graphics are then presented to a user via a web browser where it is rendered to the HTML Canvas DOM element at stage 316.

Dynamic maintenance

The initial connection to the server 100 starts with a connection benchmark; this is a quick bandwidth test simulating traffic as expected over the same transport socket that is used for streaming.

This test provides an initial snapshot of the network performance, from which the optimal video profile to stream to the client can be selected.

A knowledge graph may also be used to select some of the streaming parameters based on the device, and software on the device to which we are streaming. For example, using a lower keyframe interval on Android™ devices, as the built in media playback component provides much more stable and smooth stream at low latency with this parameter; this causes a degree of visual fidelity degradation, but improves the overall performance.

In order to maintain performance in a dynamic fashion, feedback relating to the rendering of the graphics stream may comprise data relating to transmission performance. Based on this data, the transmission stream can be modified so as to maintain a consistently smooth and responsive user experience for the streaming of the application. This may involve reducing the quality of the video data being streamed in a number of ways, increasing the server side latency by changing paketization strategy, increasing the client side video buffer etc.

These modifications can lower the visual fidelity or slightly increase the experienced input latency, but allow a better overall experience by keeping a consistently smooth and responsive application user interface (Ul), which may be considered more important in many applications.

It is also important to recognise when it is possible to improve the visual fidelity or lower the experienced input latency (e.g. when detecting that the stream is performing optimally) so as to improve the quality of the video data streamed from the server side.

Figure 4 shows a method of dynamic maintenance of streaming quality. The process is effectively a loop comprising transmitting a graphics stream 400; receiving data relating to the transmission performance 402; and modifying a property of the transmission based on the data relating to transmission performance (transmission metrics) 404. Once the stream is in progress, a number of metrics on the client side are measured including (but not limited to): client-side video buffer (in Milliseconds), Packets per second, data throughput, video playback frames per second, dropped frames per second. If these metrics drop below a variable threshold, the server is requested to reduce overall quality to maintain a consistent streaming experience; in such a case one or more of the following parameters may be reduced: Bitrate, resolution, keyframe interval. It would also be possible to increase the number of frames per packet should we detect that general bandwidth fluctuations are causing too much variability in received packets. Further to this on the client side if it is detected that too many frames are being dropped even though they have been received at the client side, the client buffer may be increased so as to reduce frame dropping, as its likely that the video decoder is unable to keep up with the shorter buffer. The parameters for the variable threshold may be set per decoding method, and can be continually 'tweaked' so as to improve overall performance.

If the measured variables are at or above an upper threshold (i.e. the stream is operating essentially perfectly) bandwidth, resolution and overall quality and/or latency of the stream may be slowly improved, whilst continuing to monitor all the way up to the maximum quality profile.

This solution has proven especially successful on highly changeable networks such as 3G/4G to allow better quality when the network allows, but continuing the keep the experience working where connectivity can be achieved.

This system differs substantially from other systems such as HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (MPEG-Dash) which encode at a few different profiles corresponding to different quality levels Offline'. This means that it is possible to select an encoding profile which closest matches the requirements for your device and network situation, however with the present Online' encoder, it is possible to tailor the video profile to perfectly suit the situation, and to adapt as the situation changes on the fly.

Whilst streaming to the client, continuous performance analysis of the stream playback is maintained. The data is sent back to the application server over the bidirectional transport socket. This data is fed back into the packetiser whereby to adjust the number of frames per packet, output resolution, stream bitrate and/or keyframe interval to maintain a good user experience if required, for instance when resolution is changed or large jump in bitrate a new header packet is generated and sent before the new packets start streaming. The effect is an overall better visual quality where possible and a much more consistent experience across the board, where you might see a 'spool up' scenario with HLS or the like where it will start on the lowest quality and then progressively enhance the quality, we are able to start out at a quality that suits the device and network conditions you are in and change parameters on a frame by frame basis to continue to give you the best performance and quality possible.

Augmented reality

The aim of augmented reality (AR) streaming is to allow consistently high quality 3D rendering/animation of assets to be overlaid onto a local live camera feed so as to augment reality. At the heart of the solution are AAA Game engines that can run graphically intensive 3D visuals that can then be overlaid into the real world using the users webcam. Figure 5 shows a flow diagram for transmitting an augmented reality stream - from the perspective of both the client and the server.

The first step 500 is to determine the presence of an augmented reality marker. A marker may be in the form of a mark, object or image. For example, a chair may be identified as a marker and be augmented with graphics such as a character sitting on it. The marker may be determined by image and/or video processing techniques so as to recognise certain patterns or shapes. Determining the presence of a marker may be computed either on the client device 102 or alternatively data relating to the marker (such as portions of images or videos, or derivative data therefrom) may be transmitted 502 to the server 100 where greater processing power may be exploited to analyse the data relating to the AR marker 504. The 'real time' nature of the transmission described herein and the low latency of the transmission allows for efficient Outsourcing' of such processor-intensive tasks. In such an example, the server 100 receives the data relating to the AR marker 504, analyses this data 506 so as to generate information identifying the presence of the AR marker, and transmit this to the client 102 where it is received 508.

Once the presence of an augmented reality marker is determined, the client device 100 determines a projection matrix relating to the augmented reality marker in step 510. The projection matrix defines the relative position of the client device to the augmented reality marker, for example device orientation and distance.

In step 512, the projection matrix is transmitted to the server via the same transport socket as the graphics are received. The projection matrix is recalculated on a regular basis, which may be defined by a regular interval or prompted by a change in device orientation (or a mixture thereof). It will be appreciated that the frequency of updates may be increased or decreased in dependence on transmission properties as described above in the section relating to 'dynamic maintenance'.

The next step 514 is for the server to produce an augmented reality graphics stream, which is then transmitted to the client device via the bidirectional transport socket at step 516 and received at step 518.

Various steps will now be discussed in more detail. Marker tracking takes place via a browser on the client device 102, and is implemented to quickly recognise AR markers and transmit the data down the bidirectional transport socket. The projection matrix sent by the client device gives precise information regarding the position, rotation and scale or the VR Marker in relation to the camera 1 1 1 on the client device 102.

Once the Transform Matrix is received at the server 100, the target object is rendered onto a 'Virtual Green Screen' at the requested position, rotation and scale. This vital data is then broadcast using the platform back to the client 102 where chroma keying solution is used to remove the 'keying green screen' displaying just the target object at the location of the AR marker thereby effectively adding an object to be rendered into the scene from the user's perspective.

In one example the Chroma keying occurs in javascript™ using one of a number of techniques depending of the device/browsers abilities where by a sample is taken of the 'virtual green screen' from an unused corner, and then that sample used to define a profile of which pixels should be removed by colour range. Taking this colour range it is possible to use WebGL (if available in the browser) to remove the background with a shader. If WebGL is not available it is possible to use a per pixel iteration to remove the background.

The method described herein thus allows a much higher level of fidelity for 'in- browser', or Όη-mobile' AR. Even when WebGL is not available it is possible to render 3D in the cloud and overlay in the browser to achieve a better result. In such a way full 3D animated renders can be streamed to the browser in real-time and displayed in place within the local environment.

Figure 6 shows a representation of an augmented reality stream overlaid on a camera feed. Camera feed 550 is shown - in this case with the image of a chair 555 which is defined as a marker. A marker recognition layer 560 is shown (which may not be presented to the user) which identifies the location of the marker (shown as a box 565 labelled ). There may be multiple different markers in a single camera feed, but for simplicity Figure 6 only shows a single marker. The final layer is the render from the server 570 overlaid on the camera feed 550. In this example, a cube 575 is rendered in a position defined by the marker 565. It is worth noting that the marker 565 and the rendered object 575 do not have to coincide; the marker 565 may simply define where the object 575 should be displayed (e.g. a certain displacement from the marker). Co-browsing

The methods and techniques for real-time, low latency transmission of graphics described above can also be used to share a user's screen session with another user. Trying to explain what you are seeing or experiencing on your computer screen can be difficult; current desktop replication solutions requires software installation and allows access to your local machine which may present a security risk - which a user is unlikely to want to do in a simple sales scenario, or when asking a simple question about a product (for example). An improved solution is therefore required.

Figure 7 shows a method of streaming graphics corresponding to a first user's (the presenter's) screen session to second user (the viewer). In general, a first user and a second user are connected, and the first user's session is transmitted to the second user via a bidirectional transport socket (via a server).

As Figure 7 shows, a server 100 essentially sits in-between the presenter 102-1 and the viewer 102-2. In use, the presenter 102-1 connects to the server 100 and shares their session over a bi-directional transport socket. This session may then be forwarded over a separate bidirectional transport socket to the viewer 102-2.

The server 100 is preferably geographically near both users so as to reduce latency. When setting up the session, an appropriate server may be selected based on the users' geographical locations - for example from their IP (Internet Protocol) addresses. If the users are very distant from one-another, selecting a server that is closer to the first user reduces latency as the transmission from first user to server is the first (and key) step in the process.

The first step 600 is for the presenter 102-1 to connect to the server 100. This comprises negotiating a bidirectional transport stream between the presenter 102-1 and the sever 100 and initiating a transport stream as described above with reference to Figure 2. A 'co-browsing' application is loaded on the server 100 at step 602 and streamed, with the connection optionally being dynamically adjusted for the device and connection at step 604.

The presenter 102-1 then initiates the viewing session at step 606. This initiates the sharing of a UUSI (Universally Unique Session ID) at step 608. In a normal session, the user initiates the application which is then streamed to only them, and input is gathered only from their PC. However with Co-Browsing, after the first user 102-1 (the presenter) initiates their session, they are able to allow access to others by sharing a link, the initial user session works exactly as usual on the Application Streaming Platform, however when the second user 102-2 (the viewer) connects, a number of different events occur. Sharing the UUSI can be achieved in a number of ways, a QR code may be generated and ready to be captured by a mobile device, or a simple link can be emailed, or a code can be shared that will allow access when entered at a viewing portal.

In order to verify the UUSI, it is looked within a session database so as to connect the Viewer to the same server as the Presenter. The local server session ID is also queried so as to determine the correct session. With this information it is then possible to connect the viewer 102-2 to the server 100 via the bidirectional transport socket that the application that the presenter started is on at step 610. Instead of sending the usual start application command the viewer 102-2 can simply ask the server 100 to stream the same application as the presenter. The viewer than views the same graphics as the presenter at step 614.

The session to be shared (defined herein as the first user's session, or presenter's, session) is then encoded in real-time and transmitted to a second user 102-2 (the viewer). This transmission occurs over a bidirectional transport socket so as to reduce the overhead and latency when transmitting data over HTML (for example), but also so as to be able to receive feedback relating to the rendering of the graphics stream so as to enable the second user to interact with the first user's session (described below with reference to Figure 8).

As with normal one to one sessions, both the presenter and the viewer have server streams optimised for their respective devices and connection capabilities (steps 604 and 612). A further optional functionality added here is that the presenter can enable remote control of their session, meaning that input can then be streamed from the viewer's device into the presenter's session, allowing both the presenter and the viewer to control and interact with the same application instance.

Figure 8 shows a method of a first user providing control of a session to a second user.

The first step 700 is for the presenter 102-1 to initiate 'remote control'. This may be via a button on a user interface. The viewer 102-2 is notified of this at step 602, which they may either accept or deny. The viewer 102-2 then is able to modify the presenter's stream at step 704 - for example interacting with the graphics being displayed, highlighting text, selecting different parts of programs etc. This user input is then used to modify the graphics stream at step 706 - as described above with reference to Figure 2.

The modified stream is then transmitted to both users at step 708.

Alternatives and modifications Various other modifications will be apparent to those skilled in the art for example the server 100 and one of the first 102-1 or second users 102-2 may be contained within a single device.

It will be understood that the present invention has been described above purely by way of example, and modifications of detail can be made within the scope of the invention. Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.