REALITY DEVICES AND APPLICATIONS

CROSS REFERENCE TO RELATED APPLICATIONS

[001] This application claims priority to U.S. Provisional Application No.

62/649,082 filed March 28, 2018.

BACKGROUND

[002] Mixed reality or augmented reality display devices, such as head-mounted display devices, may be used in a variety of real-world environments and contexts. Such devices provide a view of a physical environment that is augmented by a virtual environment including virtual images, such as two-dimensional virtual objects and three- dimensional holographic objects, and/or other virtual reality information. Such devices may also include various sensors for collecting data from the surrounding environment.

[003] An augmented reality device may display virtual images that are interspersed with real-world physical objects to create a mixed reality environment. A user of the device may desire to interact with a virtual or physical object using the mixed reality device. However, conventional augmented reality devices may be limited to single-user applications, which may prevent collaborative applications of augmented reality devices from being realized.

BRIEF DESCRIPTION OF THE DRAWINGS

[004] FIG. 1 illustrates a system level block diagram for an augmented reality device, in accordance with an embodiment.

[005] FIG. 2 illustrates a system level block diagram for a system that includes networked augmented reality devices, in accordance with an embodiment.

[006] FIG. 3 is a diagram of an illustrative process flow that may be performed by an augmented reality device to load a virtual scene from memory, in accordance with an embodiment. [007] FIG. 4 is a diagram of an illustrative process flow that may be performed by an augmented reality device to join or host a network session or to begin a single- user session, and to manipulate a virtual scene of a session, in accordance with an embodiment.

[008] FIG. 5 shows illustrative placement of virtual markers in an augmented reality scene, in accordance with an embodiment.

[009] FIG. 6 shows illustrative placement of a virtual object in an augmented reality scene, in accordance with an embodiment.

[0010] FIG. 7 shows an illustrative process flow that may be performed by an augmented reality device to save a virtual scene of a session, in accordance with an embodiment.

[0011] FIG. 8 shows an illustrative process flow that may be performed by an augmented reality device to record a virtual scene of a session and to record user gestures and interactions with virtual objects within the virtual scene, in accordance with an embodiment.

DETAILED DESCRIPTION

[0012] The present invention will now be discussed in detail with regard to the attached drawing figures that were briefly described above. In the following description, numerous specific details are set forth illustrating the Applicant’s best mode for practicing the invention and enabling one of ordinary skill in the art to make and use the invention. It will be obvious, however, to one skilled in the art that the present invention may be practiced without many of these specific details. In other instances, well-known machines, structures, and method steps have not been described in particular detail in order to avoid unnecessarily obscuring the present invention. Unless otherwise indicated, like parts and method steps are referred to with like reference numerals.

[0013] As described above, conventional augmented reality devices may be limited to single-user environments and may therefore lack the ability to create a collaborative augmented reality environment in which multiple users. Embodiments of the present invention allow multiple augmented reality devices to be networked together and for the virtual portion of an augmented reality environment to be synced between the augmented reality devices of multiple users connected to a single network session over an electronic communications network. This networked mode of operation may be beneficially applied in classroom environments, where it may be desirable for a teacher and multiple students to view and interact with the same virtual objects displayed within an augmented reality scene on augmented reality devices worn or used by the students and the teacher. Furthermore, a content management system may be implemented, which may manage network sessions for augmented reality devices. This content management system may also include databases of models for the creation of 2D and 3D virtual objects. Memory of augmented reality devices and/or remote databases of the content management system may store user-created virtual scenes containing virtual objects. A given virtual object in a virtual scene may have corresponding metadata stored in respective data files of the database, which define the location, orientation, and other information about the given virtual object. Data corresponding to how a virtual scene changes and how users interact with a virtual scene may also be recorded and saved during the operation of an augmented reality device in a given session. Accordingly, embodiments of the present invention may allow for interactive and collaborative augmented reality environments to be created, saved, and accessed using networked augmented reality devices.

[0014] FIG. 1 is a block diagram depicting functional components of an illustrative augmented reality (AR) device 100 that may, for example, display a view of a physical scene that is augmented through the integration of a virtual scene that includes virtual images, such as two-dimensional (2D) and three-dimensional (3D) virtual objects. While shown as a block diagram here, the AR device 100 may be a smartphone, tablet device, head mounted display (FIMD) device, smart glasses, or any other applicable portable electronic device as may be readily understood by one of ordinary skill. For examples in which the AR device 100 is an FIMD device or smart glasses, the AR device 100 may be a holographic computer built into a headset having one or more semitransparent holographic lenses through which virtual objects may be displayed overlapping a physical scene so that a user perceives an augmented reality environment. For examples in which the AR device 100 is a smartphone or a tablet device, the AR device 100 may overlay captured video images of a physical scene with virtual objects to generate an augmented reality scene that may then be presented to a user via an electronic display. The AR device 100 may allow a user to see, hear, and interact with virtual objects displayed within a real-world environment such as a classroom, living room, or office space. For example, the augmented reality scene may display education content.

[0015] The AR device 100 may include a processor 102, a memory 104 that includes an AR component 110 and an operating system 106, input/output (I/O) devices 108, a network interface 112, cameras 120, a display device 130 and an accelerometer 140. Generally, the processor 102 may retrieve and execute programming instructions stored in the memory 104. The processor 102 may be a single CPU, multiple CPUs, a single CPU having multiple processing cores, GPUs having multiple execution paths, or any other applicable processing hardware and arrangement thereof. The memory 104 may represent both random access memory (RAM) and non-volatile memory (e.g., the non-volatile memory of one or more hard disk drives, solid state drives, flash memory, etc.) In some embodiments, the memory 104 may be considered to include memory physically located elsewhere (e.g., a computer in electronic communication with the AR device 100 via network interface 112). The operating system 106 included in memory 104 may control the execution of application programs on the AR device 100 (e.g., executed by processor 102). The I/O devices 108 may include a variety of input and output devices, including displays, keyboards, touchscreens, buttons, switches, and any other applicable input or output device. The network interface 112 may enable the AR device 100 to connect to and transmit and receive data over an electronic data communications network (e.g., via Ethernet, WiFi, Bluetooth, or any other applicable electronic data communications network).

[0016] The cameras 120 may include an outward facing camera that may detect movements within its field of view, such as gesture-based inputs or other movements performed by a user or by a person or physical object within the field of view. The outward facing camera may also capture two-dimensional image information and depth information from a physical scene and physical objects within the physical scene. For example, the outward facing camera may include a depth camera, a visible light camera, an infrared light camera, and/or a position tracking camera. The cameras 120 may also include one or more user-facing cameras, which may, among other applications, track eye movement of the user. For example, the AR component 110 could use such a user-facing camera of cameras 120 to determine which portion of the display device 130, and thereby which portion of the displayed AR scene, that the user is looking at. Generally, the accelerometer 140 is a device capable of measuring the physical (or proper) acceleration of the AR device 100. For example, the AR component 110 may use the accelerometer 140 to determine when the position of the AR device 100 is changing in order to track the location of the AR device 100. The cameras 120 may capture video data of the physical scene, such that activity of the user may be monitored and analyzed to identify actions (e.g., gestures and other movements of the user) indicative of desired interactions between the user and virtual objects within the virtual scene of the AR scene shown on the AR device 100.

[0017] Generally, the AR component 110 is configured to adjust the depiction of the physical scene viewed on or through the display device 130 via the display of virtual objects of a virtual scene, thereby producing an AR scene. These virtual objects have locations and orientations within the physical scene that are based on the locations and orientations of virtual markers that may be placed by a user's action. The collection of virtual components (e.g., the virtual objects, respective positions and orientations of the virtual objects, and optional state information for the virtual objects) in a given AR scene may be referred to herein as a "virtual scene." For instance, the AR component 110 may cause one or more virtual objects of a virtual scene to be displayed at one or more defined (e.g., predefined or defined by a user's action) locations within the physical scene to generate an AR scene. As the AR scene displayed via the display device 130 represents a three-dimensional space, the AR component 110 could determine an area of three-dimensional space for a given virtual object to occupy. Once a virtual object has been placed, the virtual object may be, for example, repositioned, rotated, resized, or otherwise manipulated by a user. A database of virtual objects may be stored in the memory 104 (e.g., as 3D models). When a user inputs a command (e.g., using one or more detectable gestures) to add (e.g., import) a virtual object to a virtual scene, the AR component 110 may retrieve the virtual object from this database in the memory 104 and may place the virtual object in the virtual scene. Each virtual object added to the virtual scene may have a location and orientation defined with respect to the virtual markers placed by the user. In some embodiments, other information about each virtual object (e.g., color, the present frame of the virtual object in an animation sequence, etc.) may be defined in memory. As will be described any given virtual scene that has been created using the AR component 110 of the AR device 100 may be saved in the memory 104 so that a user can later retrieve and load saved virtual scenes using the AR device 100. As virtual objects are added or removed from a virtual scene or otherwise changed within the virtual scene, the display 130 may be periodically refreshed to show these changes.

[0018] Metadata for each virtual object present in the virtual scene may be continuously updated during operation of the AR device 100, and may be stored in the memory 104 (e.g., as a data file such as a JavaScript Object Notation (JSON) file) when the corresponding virtual scene is saved. The metadata for a given virtual object may include, for example, the location and orientation of the given virtual object relative to each of the virtual markers placed by the user, a scaling factor representing the magnitude of any resizing that has been performed on the virtual object, and/or color information.

[0019] The metadata may further include a reference identifier for the virtual object to which the metadata corresponds. For example, each virtual object may be associated with an individual reference identifier. The reference identifier may, for example, be a string of alphanumeric characters. When retrieving a virtual object from memory to add the virtual object to a virtual scene, the AR component 110 may determine which virtual object to retrieve based on the reference identifier of that virtual object.

[0020] The metadata may further include state information or timestamp information for the virtual object. The state information may, for example, define which of a plurality of predefined animation sequences are being undergone by the virtual object. The timestamp information may define a frame of an animated sequence that is currently displayed via the virtual object.

[0021] In some embodiments, the AR component 110 may be configured to initialize a single-user session, to host a network session to which other networked AR devices may connect, or to connect the AR device 100 to a network session hosted by another AR device or host server. As used herein, a "network session" refers to an instantiation of a virtual scene (e.g., instantiated at a host AR device or a host server), which may be loaded onto (e.g., streamed to) an AR device connected to the session (e.g., communicatively connected to a host of the session) to generate an AR scene that includes the virtual scene overlaying a physical scene. As used herein, a "single- user session" refers to an instantiation of a virtual scene on a single AR device that is not accessible by other AR devices that may otherwise be in communication with the single AR device over, for example, an electronic data communications network. Network sessions may be accessed by AR device 100 via the network interface 112 and an electronic data communications network. As an example, a network session may be established by a separate host AR device or host server and may be joined by the AR device 100 and/or by additional networked AR devices. Alternatively, the AR device 100 may be configured to act as a host and may establish its own network session that is available for other AR devices to join via the electronic data communications network. The virtual scene of a network session may be loaded onto the AR device 100 such that the display 130 shows the same virtual scene as the other networked AR devices that have joined the network session, as will be described in more detail below.

[0022] FIG. 2 shows a network architecture 200 which may enable multiple network sessions to be established between networked AR devices and virtual reality (VR) devices. A detailed example of an AR device is described above in connection with AR device 100 of FIG. 1 , and it should be understood that VR devices may operate similarly to AR devices in that VR devices display virtual objects of a virtual scene. Flowever, while virtual objects are displayed over a physical scene with AR devices, VR devices display virtual objects in only a virtual scene without showing the physical environment. It therefore follows that VR devices may save and load virtual objects to and from a virtual scene similarly to how virtual objects are saved and loaded by AR devices, although it may not be necessary to calibrate the virtual scene of a VR device in the same way that the AR scene of an AR device must be calibrated. VR devices may interact with the content management system (CMS) databases described herein for the retrieval and storage of virtual objects in substantially the same way as AR devices, and may join and host network sessions managed by these CMS databases.

[0023] Two network sessions 204 and 208 may be established between AR devices 202-1 through 202-N and VR devices 206-1 through 206-M, respectively, where N represents the number of networked AR devices in the network session 204 and where M represents the number of networked VR devices in the network session 208. The networked AR and VR devices of each network session may communicate via one or more electronic data communications networks 210, which may include one or more of local area networks (LANs), wide area networks (WANs), the internet, or any other applicable network. For example, the network session 204 may be established by the AR device 202-1 , which may act as a host device in establishing network session 204. While network sessions 204 and 208 respectively include only AR devices and only VR devices, in some embodiments network sessions may be established that include both VR devices and AR devices.

[0024] Each of the network sessions 204 and 208 may be managed via one or more CMS databases 212. CMS databases 212 may, for example, maintain a list of open network sessions which, in the present example, includes the network sessions 204 and 208. The list of open network sessions may further include, for a given network session (e.g., network session 204), identifying information (e.g., IP addresses, MAC addresses, device names, etc.) corresponding to networked AR and/or VR devices connected in the given network session (e.g., AR devices 202). A new network session may be added to the list of open network sessions maintained by the CMS databases 212 in response to a host AR or VR device initiating the new network session. An open network session may be removed from the list of open network sessions maintained by the CMS databases 212 in response to a host AR or VR device for the open network session closing the open network session. The CMS databases 212 may, for example, further include one or more databases of virtual objects that may be retrieved an AR or VR device connected to CMS databases 212 (e.g., using AR devices 202 or 206 connected to CMS databases 212 via networks 210) and loaded into the virtual scene running on the AR or VR device or that may be downloaded to the memory of an AR device for future use in generating virtual scenes.

[0025] Turning now to FIG. 3, an illustrative method 300 is shown by which a virtual scene stored in memory may be loaded onto an AR device (e.g., the AR device 100 shown in FIG. 1 ). For example, the method 300 may be performed by executing, with a processor (e.g., processor 102 shown in FIG. 1 ), instructions (e.g., part of the augmented reality component 110 shown in FIG. 1 ) stored in memory (e.g., the memory 104 shown in FIG. 1 ) of the AR device. While method 300 is described in connection with an AR device, it should be readily understood that method 300 may instead be performed using a VR device.

[0026] At step 302, the AR device may start a network session as a host, may start a single-user session, or may join a network session already in progress. For example, a prompt may be shown via a display of the AR device from which the user may select the mode of operation of the AR device, where the selectable modes include hosting a network session, joining a network session in progress, and beginning a single-user session.

[0027] At step 304, the user of the AR device adds first and second virtual markers to the virtual scene at first and second locations of the physical scene in order to calibrate the AR scene displayed by the AR device. The locations and orientations of virtual objects in the virtual scene will be defined in terms of these virtual markers. In some embodiments, the user may be prompted via the display of the AR device to place the virtual markers in opposing corners of a room. In other embodiments the virtual markers may be automatically placed based on the location of physical objects detected by the AR device. For example, the AR device may detect QR codes that are present in the physical scene and, may be placed the virtual markers at locations corresponding to these QR codes. In this way, calibration may be standardized across networked AR devices operating within the same room or space. It should be noted that the use of first and second virtual markers in step 304 is intended to be an illustrative, non-limiting example. If desired, any applicable number of virtual markers (e.g., one, three, or more) may be added to a virtual scene to calibrate an AR scene displayed on an AR device.

[0028] At step 306, the AR device receives a command from the user to load a selected virtual scene stored in memory. For example, the user may select a virtual scene from a list of available virtual scenes shown on the display of the AR device. The "load" command may be received by the AR device via the detection and interpretation (e.g., performed by a processor of the AR device) of a gesture performed by the user within the field of view of a camera of the AR device (e.g., based on video data captured by the camera). This gesture may include the user double tapping the air in a location corresponding to the selected virtual scene in the displayed list of available virtual scenes.

[0029] At step 308, the AR device automatically adds virtual objects of the selected virtual scene to the virtual scene of the current session in order to recreate the selected virtual scene. The locations, orientations, and states of the added virtual objects may be defined by metadata stored in memory. The metadata may further include a reference identifier for the virtual object to which the metadata corresponds. For example, for each virtual object of the selected virtual scene, a corresponding data file (e.g., JSON file) may be stored in memory, which identifies not only the virtual object, but also the location and orientation of the virtual object with respect to the virtual markers and additional object information. This object information, for example, may include scaling information for the virtual object, a timestamp indicating a particular frame if the virtual object is animated, color scheme information, a log of user interactions with the virtual object that occurred during one or more previous sessions, state information if the virtual object may exist in one of a multitude of states (e.g., displaying one of a multitude of animation sequences), annotations associated with the object, or any other applicable information about the virtual object.

[0030] As described above, upon starting an application on the AR device, a user may select a mode of operation that determines whether the AR device will join or host a network session or will begin a single-user session. FIG. 4 shows an illustrative method 400 by which an AR device (e.g., AR device 100 of FIG. 1 ) may start or join a session and may interact with a virtual scene of that session. For example, the method 400 may be performed, at least in part, by executing, with a processor (e.g., processor 102 shown in FIG. 1 ), instructions (e.g., part of the augmented reality component 110 shown in FIG. 1 ) stored in memory (e.g., the memory 104 shown in FIG. 1 ) of the AR device. While method 400 is described in connection with an AR device, it should be readily understood that method 400 may instead be performed using a VR device.

[0031] At step 402, an application may be started on the AR device. The application may be, for example, a software application stored in the memory of the AR device, which includes a database of 2D and/or 3D virtual objects designed for use with the application. In some embodiments, additional virtual objects may be downloaded to the memory of the AR device or otherwise accessed from one or more remote databases communicatively coupled to the AR device via an electronic data communications network. In some embodiments, upon starting the application, a login sequence may be performed in which a user is prompted to enter login credentials (e.g., a username and password) corresponding to an account of the user. This login sequence may be useful for verifying that the user is a registered user of the software application (e.g., that the user account is associated with a license of the software application).

[0032] At step 404, the AR device may generate and display a prompt to the user (e.g., by executing instructions on a processor of the AR device) in the AR scene shown on the display of the AR device. The prompt allows the user to select from a variety of displayed modes of operation for the AR device. These modes may include starting a network session with the AR device as a host, joining a session that is already in progress, and starting a single-user session. The prompt may be shown on a display of the AR device, and the mode may be selected by the user via the user's performance of a detectable gesture. For example, the user may perform an air tap gesture in a location of one of the modes displayed in the prompt in the AR scene shown on the display of the AR device. The gesture may be performed within the field of view of one or more cameras of the AR device so that the gesture may be detected and interpreted by the processor of the AR device based on video data captured by the one or more cameras.

[0033] At step 406, the method 400 progresses based on which mode was selected in step 404. If the mode of starting a network session was selected, the method 400 progresses to step 408. If the mode of joining a network session was selected, the method 400 progresses to step 410. If the mode of beginning a single-user session was selected, the method 400 progresses to step 416.

[0034] At step 408, in response to the selection of the mode of starting a network session, the AR device may communicate with a remote CMS database (e.g., CMS database 212 of FIG. 2) to add a new session to a list of available (e.g., open) network sessions that is maintained by the remote CMS database. An identifier corresponding to the AR device (e.g., a local or global internet protocol (IP) address, or a media access control (MAC) address of the AR device) may also be added to the CMS database, indicating that the AR device is the host of the new session.

[0035] At step 410, in response to the selection of the mode of joining a network session, the AR device may retrieve and display the list of available network sessions from the CMS database. In some embodiments, the CMS database may only show a subset of the total number of available network sessions managed by the CMS database. For example, the displayed list of available network sessions may only include network sessions corresponding to a particular school, LAN, or range of IP addresses. In this way, the CMS database may ensure that only relevant network sessions are presented to the user.

[0036] At step 412, the AR device may determine that the user has selected one of the displayed available network sessions, and may identify the selected network session. For example, the AR device may identify the host device of the selected network session.

[0037] At step 414, the AR device may join the selected network session. For example, the AR device may establish communication with a host device of the selected network session over an electronic data communications network. The host device may, for example, be another AR device, or may be a computer server. It should be noted that, if the CMS database determines that two AR devices are simultaneously operating within a network session using the same user credentials (e.g., corresponding to the same license of the software application), the CMS database may automatically remove one or both of these AR devices from the network session.

[0038] At step 416, the user of AR device adds first and second virtual markers to the virtual scene at first and second locations of the physical scene in order to calibrate the AR scene displayed by the AR device. The locations and orientations of virtual objects in the virtual scene will be defined in terms of these virtual markers. In some embodiments, the user may be prompted via the display of the AR device to place the virtual markers in opposing corners of a room.

[0039] At step 418, if the AR device has joined a network session being hosted by another AR device or a computer server, the AR device may load and display any existing virtual objects (e.g., shared virtual objects) from the virtual space of the network session. These virtual objects may be loaded having locations, orientations, and object information corresponding to metadata received (e.g., as data files such as JSON files) from the host device. In some embodiments, this metadata may instead be tracked and stored in the remote CMS database, and the metadata may be retrieved by the AR device from the remote CMS database. Step 418 may be skipped if the AR device is hosting its own network session or if the AR device is running a single-user session.

[0040] At step 420, the AR device may detect (e.g., based on video data captured by one or more cameras of the AR device) that a user has performed an action to modify a virtual object displayed in the AR scene or to add a virtual object to the virtual scene, and may update the metadata corresponding to the modified or added virtual object accordingly. For example, the user may move, reorient, or resize a virtual object displayed in the AR scene (e.g., by performing gestures that are detectable by the AR device), and the AR device may update the location, orientation, or scaling information of the metadata (whether stored in the memory of the AR device or stored in a remote memory) of the virtual object accordingly. As another example, the user may add a virtual object to the AR scene, and the AR device may add a data file (e.g., a JSON file) to memory containing metadata for the added virtual object.

[0041] At step 422, if the AR device is hosting a network session or has joined a network session, the action performed by the user may be replicated across all networked AR devices in the session. For example, if a virtual object is modified or added by the user of the AR device, the virtual scene of the session will be updated to reflect that modification or addition of that virtual object, and the modified or added object will be displayed in the AR scene of each of the networked AR devices connected to the network session. If the AR device is running a single-user session, then step 422 may not be performed, since no networked AR devices would be able to join the single-user session.

[0042] FIG. 5 shows an illustrative AR scene 500 in which virtual markers have been placed during the calibration of the AR scene displayed on an AR device (e.g., AR device 100 of FIG. 1 ) during a session. A first virtual marker 502 may be placed in a first location of the physical scene, and a second virtual marker 504 may be placed in a second location of the physical scene. The location of all virtual objects displayed in AR scenes throughout a given session will be defined by their distance from and orientation respective to the first and second virtual markers 502 and 504.

[0043] A given virtual object may have a location defined by two sets of coordinates stored as part of the metadata for the given virtual object. The first set of coordinates may define a distance in the x-direction, a distance in the y-direction, and a distance in the z-direction between a central point of the virtual object and the first virtual marker 502 for an (x,y,z) 3D Cartesian coordinate space. The second set of coordinates may define a distance in the x-direction, a distance in the y-direction, and a distance in the z-direction between the central point of the virtual object and the second virtual marker 504 for the x,y,z 3D Cartesian coordinate space. The use of Cartesian coordinates here is intended to be illustrative and not limiting. If desired, any other applicable coordinate system may be used. [0044] A given virtual object may have an orientation defined in the metadata for the given virtual object using the orientations of one or both of the first and second virtual markers as reference. The orientation of the given virtual object may be defined using an orientation vector, an orientation matrix, an orientation quaternion, or any other applicable system for expressing relative orientation.

[0045] As an example of a possible application of the virtual marker placement described above, a teacher could set up two real-world physical markers (e.g., shapes or objects drawn or placed on a flat object) at two different physical locations within a classroom to indicate to students where virtual markers should be placed. Then, when students using individual AR devices are calibrating their AR devices in a network session, the students would be instructed to place their first and second virtual markers at the respective locations of the two real-world physical markers, thereby providing controlled calibration for the students' AR devices. In this way, the teacher may ensure that the augmented scene displayed on all of the students' AR devices will be substantially the same due to this controlled calibration.

[0046] Once virtual markers have been placed to calibrate the AR scene, a user may add a virtual object to the AR scene at a location and orientation defined by the user. FIG. 6 shows a sequence of frames 604-608 depicting the placement of a virtual object (in this case, an animated hologram of a patient) into an AR scene using an AR device. Each of the frames 604-608 depicts a different AR scene that may be shown via the display of an AR device (e.g., the display 130 of AR device 100 of FIG. 1 ).

[0047] Frame 604 shows a virtual object 602 that has been added to the AR scene, but that has not yet been placed. For example, this frame may be what is shown on the display of the AR device immediately after the user performs a gesture to add the virtual object 602 to the AR scene. The virtual object 602 is shown here in frame 604 to have a partially transparent wireframe form that may help a user to accurately place the virtual object 602 in a desired location. This wireframe view of the virtual object 602 is meant to be illustrative and not limiting. If desired, other views of the virtual object 602 may be displayed during placement of the virtual object 602 (e.g., a view of the object as it will appear once placed). [0048] Frame 606 shows the virtual object 602 having been moved to the location desired by the user, but not yet having been placed by the user. A virtual plane 610 may be displayed in the AR scene when the virtual object 602 is moved within a predefined distance of a physical object on which the virtual object 602 may be placed (e.g., when the virtual object 602 is moved with the predefined distance of the chair shown in the AR scene). The virtual plane 610 may assist the user in identifying the distance, depth, and/or orientation of the virtual object 602 during placement of the virtual object 602. In some embodiments, the color of the virtual plane 610 may indicate whether the virtual object 602 is in a valid location for placement (e.g., with green corresponding to a valid location and red corresponding to an invalid location).

[0049] Frame 608 shows the virtual object 602 after placement in the physical chair in the AR scene. Upon placement by the user, the virtual object 602 may transition from a wireframe view to a solid view and, in some embodiments, animations that may be performed by the virtual object 602 may begin.

[0050] Once one or more virtual objects have been placed in a virtual scene of a session of the AR device, a user may desire to save the virtual scene to the memory of the AR device so that the virtual scene may be retrieved and loaded onto the AR device at a later time. FIG. 7 shows a method 700 by which a virtual scene may be established and subsequently saved by an AR device (e.g., AR device 100 of FIG. 1 ). For example, the method 700 may be performed, at least in part, by executing, with a processor (e.g., processor 102 shown in FIG. 1 ), instructions (e.g., part of the augmented reality component 110 shown in FIG. 1 ) stored in memory (e.g., the memory 104 shown in FIG. 1 ) of the AR device. While method 700 is described in connection with an AR device, it should be readily understood that method 700 may instead be performed using a VR device.

[0051] At step 702, the AR device may start a network session as a host, may start a single-user session, or may join a network session already in progress. For example, a prompt may be shown via a display of the AR device from which the user may select the mode in which to begin operation of the AR device, where the selectable modes include hosting a network session, joining a network session in progress, and beginning a single-user session.

[0052] At step 704, the user of AR device adds first and second virtual markers to the virtual scene at first and second locations of the physical scene in order to calibrate the AR scene displayed by the AR device. The locations and orientations of virtual objects in the virtual scene will be defined in terms of these virtual markers. In some embodiments, the user may be prompted via the display of the AR device to place the virtual markers in opposing corners of a room.

[0053] At step 706, the user adds virtual objects to the virtual scene of the session. For example, virtual objects may be added to the scene by the user one at a time (e.g., as described in connection with FIG. 6). As another example, multiple virtual objects may be loaded simultaneously in response to the AR device joining a network session that is already in progress (e.g., at step 418 of method 400 of FIG. 4) or in response to a user command being issued instructing the AR device to load a stored virtual scene from memory (e.g., as described in connection with method 300 of FIG. 3).

[0054] At step 708, metadata for each virtual object present in the virtual scene may be continuously updated as the session progresses. For example, metadata for a given virtual object in the virtual scene may be updated in response to manipulation of the given virtual object by the user of the AR device or by the user of a networked AR device in the session. As another example, for an animated virtual object, the metadata for the animated virtual object may be updated as the animated virtual object progresses through one or more animation sequences.

[0055] At step 710, the AR device receives a command from a user to save the current virtual scene of the session. For example, the user may select a "save" command from a list of available commands shown on the display of the AR device. The command may be received by the AR device via the detection and interpretation (e.g., performed by a processor of the AR device) of a gesture performed by the user within the field of view of a camera of the AR device (e.g., based on video data captured by the camera). This gesture may include the user double tapping the air in a location corresponding to the displayed "save" command in the list of available commands.

[0056] At step 712, in response to receiving the "save" command from the user, the AR device automatically stores the metadata for all virtual objects in the virtual scene in memory as part of a saved virtual scene. In this way, the saved virtual scene may be loaded back onto the AR device, or any other AR device with access to the saved virtual scene, during future sessions.

[0057] The methods of saving and loading scenes described in connection with FIGS. 3 and 7 may be useful for a variety of applications, including teaching. For example, a teacher may create a virtual scene containing virtual objects that are relevant to a lesson that the teacher plans to present to her students, well in advance of the lesson being taught. As a result of the calibration performed using the virtual markers, the teacher may even design the virtual scene in a room that is different from the room in which the teacher plans to give the lesson. Once the virtual scene has been created by the teacher, the teacher may save the virtual scene to the memory of her AR device. When giving the lesson to her students, the teacher may load the saved virtual scene to a session that networked AR devices of the teacher and the students have joined. In this way, the teacher may create a virtual scene that is relevant to a lesson being taught in advance of teaching the lesson so that the virtual scene does not have to be assembled during valuable class time.

[0058] It may be desirable in some instances to record the virtual scene of a session as it changes throughout the progression of the session so that the recorded virtual scene may be replayed at a later time. FIG. 8 shows a method 800 by which a virtual scene may be established and subsequently recorded by an AR device (e.g., AR device 100 of FIG. 1 ). For example, the method 800 may be performed, at least in part, by executing, with a processor (e.g., processor 102 shown in FIG. 1 ), instructions (e.g., part of the augmented reality component 110 shown in FIG. 1 ) stored in a memory device (e.g., the memory 104 shown in FIG. 1 ) of the AR device. While method 800 is described in connection with an AR device, it should be readily understood that method 800 may instead be performed using a VR device. [0059] At step 802, the AR device may start a network session as a host, may start a single-user session, or may join a network session already in progress. For example, a prompt may be shown via a display of the AR device from which the user may select the mode in which to begin operation of the AR device, where the selectable modes include hosting a network session, joining a network session in progress, and beginning a single-user session.

[0060] At step 804, the user of AR device adds first and second virtual markers to the virtual scene at first and second locations of the physical scene in order to calibrate the AR scene displayed by the AR device. The locations and orientations of virtual objects in the virtual scene will be defined in terms of these virtual markers. In some embodiments, the user may be prompted via the display of the AR device to place the virtual markers in opposing corners of a room.

[0061] At step 806, the AR device receives a command from a user to begin recording the current virtual scene of the session. For example, the user may select a "record" command from a list of available commands shown on the display of the AR device. The "record" command may be received by the AR device via the detection and interpretation (e.g., performed by a processor of the AR device) of a gesture performed by the user within the field of view of a camera of the AR device (e.g., based on video data captured by the camera). This gesture may include the user double tapping the air in a location corresponding to the displayed "record" command in the list of available commands.

[0062] Steps 808 and 810 may be performed in parallel while the virtual scene of the session is being recorded. At step 808, the AR device may store interaction data that is continuously updated to include all performed user gestures and interactions with virtual objects in the virtual scene as these gestures and interactions occur during the session, and that further includes the time of each interaction and the ID of the AR device through which each interaction was performed. At step 810, the AR device may store virtual scene data for the virtual scene by sampling (e.g., saving) the virtual scene at discrete points in time at a predetermined sampling rate. A given sample of virtual scene data may include metadata corresponding to each of the virtual objects in the virtual scene at a discrete point in time. Each sample may include a plurality of data files which may be, for example, JSON files, where each data file corresponds to a particular virtual object. The predetermined sampling rate may, for example, be one sample per second.

[0063] At step 812, the AR device receives a command from a user to stop recording the current virtual scene of the session. For example, the user may select a "stop recording" command from a list of available commands shown on the display of the AR device. The command may be received by the AR device via the detection and interpretation (e.g., performed by a processor of the AR device) of a gesture performed by the user within the field of view of a camera of the AR device (e.g., based on video data captured by the camera). This gesture may include the user double tapping the air in a location corresponding to the displayed "stop recording" command in the list of available commands.

[0064] As an example of one possible application for the method 800 of FIG. 8, a teacher may use an AR device to record a virtual scene of a network session in which multiple students using multiple networked AR devices watch the teacher manipulate a virtual object as part of a lesson, so that the teacher may assess how engaged students were with the lesson. As another example, a recorded virtual scene corresponding to a teacher's lesson may be played back for students who were unable to be present for the original lesson.

[0065] In an example embodiment, an augmented reality device may include a display, a memory device, a camera, and a processor. The camera may be configured to capture video data. The processor may be configured to execute computer-readable instructions for displaying an augmented reality scene on the display, the augmented reality scene including a physical scene that is overlaid with a first virtual scene, calibrating the augmented reality scene by adding first and second virtual markers to the first virtual scene of the augmented reality scene, the first and second virtual markers being placed in respective first and second locations corresponding to the physical scene based on first video data captured by the camera of the user selecting the first and second locations, receiving a load command, and loading a second virtual scene stored in the memory device to be shown via the display by modifying the first virtual scene to include virtual objects of the second virtual scene, the virtual objects having locations and orientations defined with respect to the locations and orientations of the first and second virtual markers, and the locations and orientations of the virtual objects being defined by a first plurality of data files stored in the memory device. The display may be periodically refreshed to show changes to the virtual scene.

[0066] In some embodiments, the processor may be further configured to execute computer-readable instructions for receiving a save command, and storing the modified first virtual scene in the memory device by storing a second plurality of data files in the memory device. A given data file of the plurality of data files may include metadata for a virtual object of the virtual objects. The metadata may define a location and an orientation of the virtual object with respect to the locations and orientations of the first and second virtual markers.

[0067] In some embodiments, the processor may be further configured to execute instructions for detecting a record command based on second video data captured by the camera, storing virtual scene data by periodically sampling the modified first virtual scene at a predetermined sampling rate, and storing interaction data that includes gestures and interactions detected by the augmented reality device, wherein the processor continuously updates the interaction data.

[0068] In some embodiments, the plurality of data files may include JavaScript Object Notation files.

[0069] In some embodiments, the augmented reality device may include a network interface connected to an electronic data communications network. The processor may be further configured to execute instructions for generating and presenting, on the display, a prompt to select one of a plurality of modes of operation for the augmented reality device, determining, based on second video data captured by the camera, that a mode of operation of the plurality of modes of operation has been selected, retrieving a list of available network sessions from a database over the electronic data communications network, determining, based on third video data captured by the camera, that a network session has been selected from the list of available network sessions, and joining the network session through communication over the electronic data communications network.

[0070] In some embodiments, the processor may be further configured to execute instructions for displaying a second augmented reality scene on the display, wherein the second augmented reality scene comprises the physical scene that is overlaid with a third virtual scene, calibrating the augmented reality scene by adding third and fourth virtual markers to the third virtual scene of the second augmented reality scene, the third and fourth virtual markers being placed in respective third and fourth locations corresponding to the physical scene based on fourth video data captured by the camera of the user selecting the third and fourth locations, receiving metadata from a host device of the network session defining shared virtual objects of a fourth virtual scene of the network session, and modifying the third virtual scene to match the fourth virtual scene of the network session by adding the shared virtual objects to the third virtual scene, the shared virtual objects having locations and orientations defined by the metadata received from the host device.

[0071] In some embodiments, the processor may be further configured to execute instructions for detecting, based on fifth video data captured by the camera, that the user has performed a modification of a shared virtual object of the shared virtual objects, and causing the modification of the shared virtual object to be replicated and displayed across multiple augmented reality devices that have joined the network session.

[0072] In an example embodiment, a method may include, with a processor of an augmented reality device, causing an augmented reality scene comprising a physical scene that is overlaid with a first virtual scene to be shown on a display of the augmented reality device, periodically refreshing the display to show changes to the virtual scene, with the processor, calibrating the augmented reality scene by adding first and second virtual markers to the first virtual scene at first and second locations within the physical scene, in response to selection of the first and second locations by a user detected based on first video data captured by a camera (120) of the augmented reality device, with the processor, receiving a load command, and, with the processor, loading a second virtual scene stored in a memory device of the augmented reality device to be shown via the display by modifying the first virtual scene to include virtual objects of the second virtual scene, the virtual objects having locations and orientations defined with respect to the locations and orientations of the first and second virtual markers, the locations and orientations of the virtual objects being defined by a first plurality of data files stored in the memory device.

[0073] In some embodiments, the method may further include, with the processor, receiving a save command, and, with the processor, storing the modified first virtual scene in the memory device by storing a second plurality of data files in the memory device, wherein a given data file of the plurality of data files includes metadata for a virtual object of the virtual objects, the metadata defining a location and an orientation of the virtual object with respect to the locations and orientations of the first and second virtual markers.

[0074] In some embodiments, the plurality of data files may include JavaScript Object Notation files.

[0075] In some embodiments, the method may further include, with the processor, generating and presenting, via the display, a prompt to select one of a plurality of modes of operation for the augmented reality device, with the processor, determining, based on second video data captured by the camera, that a mode of operation of the plurality of modes of operation has been selected, with the processor, retrieving a list of available network sessions from a database over an electronic data communications network, with the processor, determining, based on third video data captured by the camera, that a network session has been selected from the list of available network sessions, and, with the processor, joining the network session through communication with a host device of the network session over the electronic data communications network.

[0076] In some embodiments, the method may further include, with the processor, causing the display to show a second augmented reality scene that includes the physical scene overlaid with a third virtual scene, with the processor, calibrating the second augmented reality scene by adding third and fourth virtual markers to the third virtual scene of the second augmented reality scene, the third and fourth virtual markers being placed in respective third and fourth locations corresponding to the physical scene based on fourth video data captured by the camera of the user selecting the third and fourth locations, with the processor, receiving metadata from the host device of the network session defining shared virtual objects of a fourth virtual scene of the network session, and, with the processor, modifying the third virtual scene to match the fourth virtual scene of the network session by adding the shared virtual objects to the third virtual scene, the shared virtual objects having locations and orientations defined by the metadata received from the host device.

[0077] In some embodiments, the method may further include, with the processor, detecting, based on fifth video data captured by the camera, that the user has performed a modification of a shared virtual object of the shared virtual objects, and, with the processor, causing the modification of the shared virtual object to be replicated and displayed across multiple augmented reality devices that have joined the network session.

[0078] In some embodiments, the method may further include, with the processor, receiving a record command, with the processor, storing virtual scene data by periodically sampling the modified third virtual scene at a predetermined sampling rate, and, with the processor, storing interaction data detected by the augmented reality device.

[0079] In some embodiments, the method may further include, with the processor, constantly updating the stored interaction data to include gestures and interactions performed by users of augmented reality devices participating in the network session.

[0080] In an example embodiment, an augmented reality device may include a display, a memory device, a camera, and a processor. The camera may be configured to capture video data. The processor may be configured to execute computer-readable instructions for displaying an augmented reality scene on the display, the augmented reality scene including a physical scene that is overlaid with a first virtual scene, and the display being periodically refreshed to show changes to the virtual scene. The processor may be further configured to execute computer-readable instructions for displaying education content in the augmented reality scene. Other embodiments and uses of the above inventions will be apparent to those having ordinary skill in the art upon consideration of the specification and practice of the invention disclosed herein. The specification and examples given should be considered exemplary only, and it is contemplated that the appended claims will cover any other such embodiments or modifications as fall within the true scope of the invention.