A SHARED AUGMENTED REALITY SESSION

FIELD The present application relates in general to computer games and, in particular, to interactive computer games that establish shared Augmented Reality (AR) sessions.

BACKGROUND

Augmented Reality (AR) was develop as a series of technologies aimed at overlaying computer-generated virtual images onto a user’s view of the real-world. The widespread use of Global Positioning System (GPS) chips, digital compasses and accelerometers in mobile devices such as smart phones has led to a growth in mobile AR applications. While such mobile devices typically have far less processing power than personal computers, their portability has been a catalyst for the proliferation of mobile AR applications. Shared AR is the concept of two or more devices (typically mobile devices) running the same AR application in the same physical space experiencing a substantially identical virtual world from different perspectives in real-time. The automated and seamless establishment of shared AR experiences between mobile devices poses a number of technical problems, particularly with respect to the physical detection of devices and the subsequent data exchange requirements for the synchronization of coordinate space.

More specifically, few communication technologies are able to provide both the proximity detection capabilities that allow mobile users to establish shared AR sessions in a seamless (e.g. requires as little user intervention as possible) and flexible manner, and the high levels of data exchange required in real-time connections and the establishment of coordinate space synchronization.

These technical problems are compounded by the need to provide gaming experiences across multiple mobile device platforms, each platform having unique technologies for direct communication between devices, and often being specifically designed to prevent direct communication with devices of other platforms.

Accordingly, there is a need for improved simple and effective methods and systems for establishing proximity-based cross-platform shared AR sessions between mobile devices.

SUMMARY

The present disclosure relates to, amongst other things, methods, systems and mobile device applications configured to provide proximity device discovery and communication using a first data communication technology, real-time networking using a second data communication technology and coordinate space synchronization using real-time networking.

According to a first aspect, there is provided a method of establishing a shared augmented reality session between a first computing device and a second computing device. The method comprises the step of performing proximity device discovery between the first computing device and the second computing device using a first data communication technology. The method further comprises the step of performing real time network communication between the first computing device and the second computing device using a second data communication technology. The method further comprises the step of performing coordinate space synchronization of the first computing device and the second computing device using the second data communication technology.

According to another aspect, there is provided a non-transitory processor-readable storage medium, having processor-executable instructions stored thereon, which, when executed by a processor, cause a first computing device comprising the processor and a camera to implement an augmented reality application. The augmented reality application is configured to perform proximity device discovery between the first computing device and a second computing device using a first data communication technology. The augmented reality application is further configured to perform real-time network communication between the first computing device and the second computing device using a second data communication technology. The augmented reality application is further configured to perform coordinate space synchronization of the first computing device and the second computing device using the second data communication technology.

According to yet another aspect, there is provided a system for establishing a shared augmented reality session. The system comprises a proximity device discovery module configured to perform proximity device discovery between a first computing device and a second computing device using a first data communication technology. The system also comprises a real-time network communication module configured to establish real time network communication between the first computing device and the second computing device using a second data communication technology. The system also comprises a coordinate space synchronization module configured to perform coordinate space synchronization of the first computing device and the second computing device using the second communication technology.

BRIEF DESCRIPTION ON THE DRAWINGS

These and other aspects and features of various embodiments will become apparent from the following description with reference to the accompanying drawings, in which:

Figure 1 is a diagram illustrating an embodiment of the system described herein; Figure 2 is a flowchart illustrating steps in a method for establishing a shared AR session in accordance with a described embodiment;

Figure 3 is a flowchart illustrating steps in a method for establishing a real-time network session in accordance with a described embodiment;

Figure 4 is a diagram illustrating the general data structure of a Bluetooth Low Energy (BLE) packet;

Figure 5 is a diagram illustrating the general data structure of the Broadcast Data portion of a Bluetooth Low Energy (BLE) packet; and Figure 6 is a diagram illustrating the data structure of the Broadcast Data portion of a Bluetooth Low Energy (BLE) packet in accordance with a described embodiment.

The aforementioned drawings are provided as non-limiting examples and illustrations of various aspects and features described in the following description. DESCRIPTION

With reference to Figure 1 , in order to establish a shared AR experience between a first device 101 of a first platform (e.g. iOS®) and a second device 100 of a second platform (e.g. Android®), the system uses a first wireless communication technology 102 for proximity device discovery and the establishment of real-time network communications, and a second communication technologies 103, 104 for real-time network communications.

As defined herein, a communication technology is a technology that allow two or more devices to enter into data-communication. Communication technologies can implement communication protocols, which define the rules, syntax, semantics, synchronization and error handling of communications, and may be implemented by hardware, software or a combination thereof.

In some embodiments, the first communication technology 102 uses discovery Application Program Interfaces (APIs) that are common to the different platforms. For example, Bluetooth Low Energy (BLE) and Wi-Fi have common discovery APIs for both Android® and iOS® platforms. BLE is a wireless personal area network directed to providing reduced power consumption and cost with a similar range, and was introduced as part of the Bluetooth 4.0 Core Specification. Advantageously, BLE is natively supported by iOS®, Android®, Windows Phone®, BlackBerry®, macOS®, Linux® and Windows 10® operating systems. Moreover, BLE is a radio technology that does not require user intervention to transmit data between devices (e.g. BLE does not require pairing, unlike traditional Bluetooth®), and has a transmission range of between 20m-30m (obstructed) and 60m (line of sight). Accordingly, in some embodiments, a first communication technology 102 is used for proximity device discovery and for establishment of a real-time network connection between devices 100, 101. For example, in some embodiments, the functionality afforded by the Generic Access Profile (GAP) layer in the BLE technology stack, which functionality includes device discovery, link establishment, link termination, initiation of security features and device configuration, may be used for proximity device discovery and for establishing a real-time network connection.

Typically, in accordance with BLE technology protocols, when a link is established at the GAP layer, data communication (and associated functionality) is subsequently handled at the BLE Generic Attribute Profile (GATT) layer. In order to avoid the disadvantages associated with BLE Generic Attribute Profile (GATT) connections, however, some embodiments of the systems and methods described herein use BLE Advertising and Scanning exclusively for proximity detection and in assisting with the establishment of a real-time network connection (via a second communication technology) between devices 100, 101. As will be appreciated by the skilled reader, other cross-platform communication technologies can be used to ensure device proximity detection, and to assist in establishing a real-time network connection.

As described briefly above, in some embodiments, the AR application service broadcasts as a BLE service. Thus, in such embodiments, devices running the AR application service can scan exclusively for BLE advertisements for the AR application service.

In some embodiments, custom data sent in the BLE advertising packet includes a unique ID used for peer-to-peer resolution by the gaming server 106, as described in more detail below. The gaming server generates this ID upon user account creation, which may, in some embodiments, simply require incrementing an unsigned integer and assigning it to each new user.

Once a scanning device detects an advertisement, the AR application service resolves the unique ID through the gaming server 106 to retrieve the peer information (e.g. username, status, etc.). The AR application service then displays this to the user and notifies them that they may join a shared AR session with the peer, as described above. Should the user accept, or later decide to join their peer (assuming the peer is still within proximity) the AR application service then sends a push notification via the gaming server to the peer requesting their consent to establish a shared AR session. Once all parties involved have consented to participate in the shared AR session, a real-time network connection is established with the gaming server 106. As defined herein, a real-time network connection is any data-communication connection capable of enabling high frequency, low latency updates between devices. As such, real-time network connections ensure as accurate and smooth an AR experience as possible for users. In some embodiments, the real-time network connection may involve a relay server. In some embodiments, the described system may use the gaming server 106 to host the real-time network session.

As mentioned above, in some embodiments, BLE is used for proximity detection. This method is particularly advantageous because it does not require user intervention to transmit data between devices (e.g. it does not require device pairing).

In some embodiments, BLE Advertising packets are used to transmit data between devices. Figure 4 represents a diagram of a BLE Advertising packet, which includes 31 bytes for Advertising and Scan Response data. A more detailed structure of the 31 bytes of Advertising and Scan Response data is shown in Figure 5. Figure 6 shows a detailed structure of the 31 bytes of Advertising and Scan Response data (i.e. Broadcast data), as used in accordance with an embodiment of the presently disclosed system, which structure includes Service UUID (16 byte UUID + 2 byte overhead) having 16, 32, or 128-bit Universally Unique Identifier (UUID) identifying the BLE service. The structure also includes Flags (1 byte flag + 2 byte overhead) and LocalName (remaining 8 bytes + 2 byte overhead).

Accordingly, by using base64 encoding, it is possible to insert 6 bytes of configurable data into the LocalName field for use by the AR application service. Thus, a unique user identifier and information capable of establishing a real-time network connection may be encoded into 48 bits of the LocalName field. In some embodiments, the device network IP address and port, together with the unique identifier, are encoded into the 48 bits. In other embodiments, the BLE MAC address (which also serves as a unique identifier that can subsequently be linked to a user’s unique identifier by way of the gaming server 106) is encoded into the 48 bits. In yet other embodiments, a user ID that can be generated and resolved by way of a relay server (such as Gamesparks®) can be encoded into the 48 bits. As will be appreciated, other data configuration arrangements may be possible.

Once the real-time network session is established, coordinate space synchronization between device 101 and 100 is sought, as described in detail below. As will be appreciated, each device running the AR application service has its own virtual representation and understanding of the real world. This includes the world coordinate space, which the AR application service (or an API) creates based on the device position and orientation when the user starts the application. In order to synchronize coordinate space between participating devices, the AR application service can use a third-party coordinate space synchronization system, such as Google Cloud Anchors™.

Thus, coordinate space synchronization can be provided by a cloud-based anchor system such as that provided by Google’s ARCore™ Cloud Anchor API service. In order to provide shared AR experiences, such cloud-based systems host and resolve synchronization anchors through real-time network connections. Synchronization anchors are shared reference points in 3D space that allow two or more devices/users to establish coordinate space synchronization, thereby allowing shared AR experiences.

Typically, hosting and resolving synchronization anchors involves having a user create a local anchor in their environment (i.e. surrounding 3D space). During hosting, the user’s mobile device, through an API service, uploads 3D feature map data relating to the anchor to the cloud-based system, which returns a unique ID corresponding to that anchor. The AR application running on the mobile device can then distribute the unique ID to other devices/users. Using the unique ID, other AR applications can then recreate the same anchor by having the cloud-based anchor system resolve the anchor through the API service.

In some embodiments, synchronization of the coordinate space between device 101 and 100 may be performed as follows. Device 101 requests an anchor from the coordinate space synchronization server 107 at some position and orientation in the world.

The position and orientation can be obtained by some form of user intervention. In some embodiments, a ray trace against AR planes is made from the user tapping the screen, and the hit location is where the anchor (e.g. a virtual object) is placed. In other embodiments, the anchor can be placed by scanning a real-world object. The request uploads the surrounding point cloud to the coordinate space synchronization server 107 which then generates an anchor based on this information. The coordinate space synchronization server 107 provides device 101 with a unique anchor ID for the anchor.

Then, device 101 sends the unique anchor ID to device 100. Device 100 then scans the area where device 101 placed the anchor and requests that the coordinate space synchronization server 107 resolve the anchor ID. The resolution request uploads the surrounding point cloud to the coordinate space synchronization server 107, which then attempts to resolve an anchor based on this information. Finally, the coordinate space synchronization server 107 provides device 100 with the position and orientation of the anchor, at which point device 100 and device 101 share a synchronized coordinate space.

In some embodiments, the AR application service encourages or prompts the user to map a large section of the anchor placement area before placing the anchor in order to allow for easier resolution of the anchor by the coordinate space synchronization server 107 from varying angles and distances afterwards.

Once an anchor is resolved, the AR application service replicates all position and orientation data in relation to the anchor coordinate space. This can include, for example,“parenting” virtual objects to the anchor or transforming coordinates into the anchor local space before transmitting to peer devices.

Figure 2 shows a method 200 of establishing a shared AR real-time network session in accordance with an embodiment. The method comprises the process of accepting an invitation to join a shared AR session (steps 204 to 209) and the process of initiating an invitation to join a shared AR session (steps 210 to 217). It will be appreciated that each of these processes are independent, and therefore need not occur together.

At step 201 , an AR application is launched on a mobile device 101. Examples of mobile devices include, but are not limited to, smartphones, tablets, wearable devices (such as Google Glass®) and laptop computers. At step 202, the AR application running on the mobile device 101 starts a BLE Advertising broadcast with a unique peer ID, as described below. Then at step 203, the AR application starts BLE Scanning for broadcast of the AR application service. It will be appreciated by the skilled reader that steps 202 and 203 need not occur sequentially, nor need they occur in any particular order.

With reference to steps 204 to 209 of Figure 2, the process of accepting an invitation to join a shared AR session is described below. In the example of Figure 2, another user of a second mobile device 100 may have received and/or otherwise detected the BLE Advertising broadcast sent by mobile device 101 , and may have performed a method of instigating a shared AR session as, for example, described in more detail below with reference to steps 210 to 215. In particular, the method described with reference to steps 210 to 215 results in the second mobile device 100 inviting mobile device 101 (via, for example, push notification through gaming server 106) to join a pre-established AR session having a unique ID. In other embodiments, any other type of notification can be used to notify the user of the mobile device 101 of an invitation to join a shared AR session. In some embodiments, the invitation includes the unique ID.

If, at step 204, the mobile device 101 receives a push notification (or other notification) from the gaming server 106 to join a shared AR session, and the user of the mobile device 101 accepts the invitation at step 205, the method proceeds to steps 206 to 208, which aim to have the coordinate space synchronization server 107 resolve the synchronization anchor relating to the unique ID. If, on the other hand, no invitation is received (step 204), or indeed the invitation is refused by the user of mobile device 101 (step 205), then the AR application of mobile device 101 continues to perform BLE Scanning for broadcast of the AR application service.

At step 206, once the user of mobile device 101 has accepted the invitation, the AR application sends a request to the coordinate space synchronization server 107 to resolve the anchor, by way of the API service. At step 207, the AR application and/or the coordinate space synchronization server 107 may request that the user of the mobile device 101 provide input in order to aid in the resolution of the synchronization anchor, and perform some degree of mapping in order to generate the required 3D feature map data. Such mapping can, for example, include moving the mobile device around the anchor point in 3D space in order to capture 3D feature map data (e.g. the anchor’s relative position with respect to other objects in the 3D space surrounding the anchor, assessed from multiple angles). In some embodiments, the AR application can generate instructions and/or other visual indicators through the User Interface (Ul) in order to assist the user in capturing 3D feature map data. In some embodiments in which the disclosed system and method is at least partially implemented using the ARCore™ Cloud Anchor API, step 206 can be implemented using the resolveCloudAnchor(cloudAnchorld) resolve request.

If the synchronization anchor is resolved successfully at step 208, the mobile device 101 joins the real-time network session, by way of communication technology 103. In some embodiments, communication technology 103 allows mobile device 101 to communicate over the Internet, such as, WiFi or broadband cellular network technologies (e.g. 4G, 5G, etc.).

As will be appreciated, in some embodiments, communication technologies 103 and 104 can comprise a plurality of interoperable communication technologies. For example, mobile device 101 could be connected to a broadband cellular network access point and mobile device 100 could be connected to a WiFi access point, the broadband cellular network access point and the WiFi access point being interconnected by various elements of networking infrastructure.

If, on the other hand, the coordinate space synchronization server 107 is unable to successfully resolve the anchor at step 208, the user may again be asked to provide further input at step 207 to aid in resolving the synchronization anchor. Alternatively (not shown), the method may be ended and the user may be informed that the coordinate space synchronization server 107 was unable to resolve the synchronization anchor.

With reference to steps 210 to 217 of Figure 2, and Figure 3, the process of inviting a peer to join a real-time network session will now be described. In the present example, mobile device 100 may broadcasting an AR application service by way of BLE Advertising, as described in more detail above.

Then, at step 210, mobile device 100 broadcasting the AR application service is detected by mobile device 101. In some embodiments, the BLE Advertising packet will include Peer ID information allowing mobile device 101 to identify mobile device 100 (or the user of mobile device 100) by way of peer information, as described in more detail below. In some embodiments, the BLE Advertising packet will include other information (such as a unique identifier) which will allow mobile device 101 to send a request to the gaming server 106 to resolve the Peer ID of mobile device 100 (step 211 ).

At step 212, peer information related to the Peer ID is displayed to the user of mobile device 101. As will be appreciated, peer information includes, but is not limited to, any information allowing a the user of mobile device 101 to identify mobile device 100 and/or the user of mobile device 100, and/or indeed an Internet Identity (I ID), such as a pseudonym used online by gamers, or other identities used in internet forums, online chats, social media, etc. At step 213, the user of mobile device 101 can choose to instigate a shared AR session with the detected peer associated with the Peer ID, or continue scanning for broadcasts of the AR application service. If the user of mobile device 101 instigates a shared AR session with the detected peer at step 213, a determination is made at step 214, by gaming server 106, for example, as to whether a shared AR session has already been established by, for example, mobile device 100.

If a shared AR session has already been established, the mobile device 100 and/or the gaming server 106 invites the peer (the user of mobile device 101 ), via push notification through the backend, for example, to join the established AR session. In some embodiments, the invitation also includes an anchor ID associated with the synchronization anchor established for the shared AR session. If the peer (i.e. the user of mobile device 101 ) then accepts the invitation at step 216, the peer joins the real-time network session. If, at step 214, it is determined that a shared AR session has not already been established, then a shared AR session can be established by the user of mobile device 100 (or the user of mobile device 101 ) as described, for example, with reference to Figure 3.

Figure 3 shows an embodiment of a method 300 for establishing a shared AR session. While the process described below relates to the establishment of a synchronization anchor 108 by the user of mobile device 101 , it will be appreciated that the same process can be used by mobile device 100 to create a synchronization anchor.

At step 301 , a user creates a synchronization anchor 108 using the AR application running on mobile device 100. Creation of the anchor typically involves a user placing a virtual object/anchor at a point in three dimensional space (step 302), as seen from the camera of the mobile device. Then, the user obtains a 3D feature map of the environment in and around the point from different viewing angles and positions in a certain period of time prior to the request being submitted using the API service. In some embodiments, the process of generating the 3D feature map can be initiated by the user. In other embodiments, the process of generating a 3D feature map can be initiated automatically. In some embodiments, in order to reduce the amount of time required for a user to collect 3D feature map data by moving the camera around a particular point in 3D space, the process can be initiated automatically, and without user notification. For example, in some embodiments, the AR application service may begin mapping an area in order to collect data for anchor resolution as soon as it receives an invitation at step 204 or as soon as the peer ID is resolved via the backend at step 21 1. In these embodiments, the anchor ID can be exchanged, and the anchor resolution can be achieved prior to the user being asked whether they wish to accept an invitation to a shared AR session at steps 205 or 216. Then, after the user places the synchronization anchor at step 302, the AR application service requests an anchor via the coordinate space synchronization server 107 at step 303. In some embodiments in which the disclosed systems and methods are at least partially implemented using the ARCore™ Cloud Anchor API, step 303 can be implemented using the hostCloudAnchor(anchor) hosting request. If it is determine at step 304 that the synchronization anchor has not been created successfully at step 304, then the AR application requests further user input to create a synchronization anchor at step 301. In some embodiments, the request for further input can include a request to reposition the anchor in another location. In some embodiments, the request for further input can include a request to obtain further 3D feature map data of the environment in and around the point from different viewing angles and positions.

Once it is determined at step 304 that the synchronization anchor has been created successfully, a shared AR session is created by the gaming server 106 at step 305.

Once a shared AR session has been established, the mobile device 101 and/or the gaming server 106 invites the peer (the user of mobile device 100), via push notification through the backend (e.g. gaming server 106), for example, to join the established AR session. In some embodiments, the invitation also includes an anchor ID associated with the synchronization anchor established for the shared AR session. If the peer (i.e. the user of mobile device 100) then accepts the invitation at step 216, the peer joins the real-time network session.

Those skilled in the art will appreciate that the above described systems and methods allow for simple cross-platform establishment of shared AR sessions. The description and drawings merely illustrate the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its scope, as defined in the appended claims. Furthermore, all examples recited herein are principally intended to aid the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited examples and conditions. For example, the present disclosure describes embodiments of the invention with reference to specific communication technologies. It will however be appreciated by the skilled reader that the present invention can also advantageously be used with other communication technologies that provide the same or similar capabilities.

Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass equivalents thereof. For example, the functions of the various elements shown in the figures, including any functional blocks labelled as "server",“network” or“application program interface” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.

Furthermore, while the aforementioned description refers to mobile devices, a person of skill in the art would readily recognize that steps of various above-described methods can be performed by any number of computing devices, such as video cameras, digital cameras, infrared cameras, desktop computers, laptop computers, tablets, smartphones, smart watches or other wearables. Herein, some embodiments are also intended to cover program storage devices, e.g., digital data storage media, which are, machine or computer readable and encode machine-executable or computer- executable programs of instructions, wherein said instructions perform some or all of the steps of the above-described methods. The embodiments are also intended to cover computers programmed to perform said steps of the above-described methods.

Any feature of any embodiment discussed herein may be combined with any feature of any other embodiment discussed herein in some examples of implementation. Certain additional elements that may be needed for operation of certain embodiments have not been described or illustrated as they are assumed to be within the purview of those of ordinary skill in the art. Moreover, certain embodiments may be free of, may lack and/or may function without any element that is not specifically disclosed herein.