PRESENCE

[0001 ] The present application is a non-provisional filing of, and claims benefit under 35 U.S.C. §119(e) from, U.S. Provisional Patent Application Serial No. 62/424,190, entitled "Systems and Methods for Matching Lighting Conditions for Shared Virtual Presence," filed November 18, 2016, the entirety of which is incorporated herein by reference.

BACKGROUND

[0002] Making social connections across multiple physical spaces enabled by augmented reality (AR) and virtual reality (VR) technologies is a powerful concept, as shown in Microsoft's Holoportation telepresence system. However, in use cases where content is captured and combined from multiple locations in real-time, visual quality of the resulting mixture of different realities affects how convincing the illusion of shared virtual presence is.

[0003] In capturing and combining several physical locations to be connected via virtual or augmented reality, discrepancies between environment conditions may cause serious usability problems. Different geometric setups of the spaces may cause conflict between physical and virtual elements, thereby breaking the illusion of virtual elements being present in the physical space or causing a user to collide with physical elements not present in the virtual reality.

[0004] However, just avoiding unwanted collisions and discrepancies between geometries of the shared spaces is not enough to create a believable illusion of a shared virtual presence. Humans are very good at observing detailed lighting conditions and discontinuities, and conflicts in lighting may cause virtual elements in a scene to appear incongruous with real objects in a scene, which may cause visual discomfort and detract from the augmented reality experience.

[0005] Matching of lighting conditions between different content elements makes up a large portion of a film's post production special effects. Similarly, the level with which a virtual experience seamlessly connects different elements together may determine how believable is the end result. Methods developed for film production, where most lighting conditions matching results from careful pre-planning of set lighting and rendering of virtual elements, are not suitable for real-time use. No satisfactory methods exist for real-time matching of content from several sources. SUMMARY

[0006] Lighting characteristics of remote locations used for AR experiences are different. Previous technologies focus on general lighting characteristics but do not address the enhancement of realism for joining various real spaces. With embodiments related to a shared virtual presence, a display viewer with a well-implemented content combination provides greater realism of virtual elements and allows for more natural immersion in an augmented reality environment. Exemplary systems and methods disclosed herein create a unified appearance by joining various real spaces into a single virtual space.

[0007] Methods and systems that match in real-time different content elements are described herein. Some exemplary techniques include one or more of the following processes: (i) generating a model of an environment's lighting; (ii) separating controllable lighting and determining range of impact for controllable lighting; (iii) globally optimizing a process for finding the best matching alignment and lighting conditions for all content elements combined together; and (iv) adjusting rendering of different content elements to minimize lighting differences where matching by aligning and controlling lighting settings is not possible. Systems and methods disclosed herein match in real-time different content systems and provide a solution to these sub- issues.

[0008] Some exemplary systems and methods described herein match lighting conditions between shared AR and VR experiences that combine content from multiple sources. By providing a globally optimal lighting solution which results from maximizing similarity of lighting conditions for all parts of the content, such methods and systems described herein improve quality of AR and VR experiences and enable stronger immersion and more realistic sensing of a shared virtual presence.

[0009] For one embodiment, the methods and systems described herein detect prevailing (or ambient) lighting conditions and controllable lighting options from all different physical environments and have a centralized manner of comparing and optimizing alignment, lighting control settings, and rendering settings for all users and parts of content combined in the same experience session.

[0010] For one embodiment, a session manager is used to match alignment and lighting conditions for all different spaces and virtual elements combined for a virtual or augmented experience. For one embodiment, all clients joining an experience session from different physical locations send information about their own physical environment and lighting conditions to the session manager, which finds optimal alignment and lighting settings for all clients. The session manager may also alter a rendering image characteristic to balance the overall characteristics of the structured lighting conditions.

[0011 ] For one embodiment of systems and methods disclosed herein, lighting information is received for a first and a second physical location and a first orientation with respect to a combined virtual geometry for the first physical location and a second orientation with respect to a combined virtual geometry for the second physical location are determined to maximize a correlation of the first location lighting information adjusted to the first orientation with the second location lighting information adjusted to the second orientation.

[0012] For one embodiment, a method may comprise connecting a first user client with local controllable lighting systems in an environment of the first user client; determining, by the first user client, ambient lighting conditions, controllable lighting structures, and range of available controls; sending, from the first user client to a session manager, a request to connect to a session; sending, from the first user client to the session manager, a description of lighting conditions and control options; optimizing globally alignment, lighting, and rendering settings based on a three-dimensional model of an environment for each of a plurality of user clients connected to the session; sending, from the session manager to the first user client, alignment, lighting, and rendering settings for each environment of the plurality of user clients; setting controllable lighting systems based on the alignment, lighting, and rendering settings sent from the session manager; and streaming, receiving, and rendering content for the session.

[0013] For one embodiment, systems disclosed herein may comprise a processor and a non-transitory computer-readable medium for storing instructions that are operative, when executed on the processor, to perform the functions of receiving lighting information for the first and second physical locations and determining a first orientation with respect to a combined virtual geometry for the first physical location and a second orientation with respect to a combined virtual geometry for the second physical location to maximize a correlation of the first location lighting information adjusted to the first orientation with the second location lighting information adjusted to the second orientation.

[0014] For one embodiment, a method or system disclosed herein may comprise receiving a first lighting model representing lighting conditions of a first site, receiving a second lighting model representing lighting conditions at of a second site, selecting a coordinate transformation between the first site and the second site based at least in part on an amount of correlation between the first lighting model and the second lighting model under the selected coordinate transformation; and establishing a telepresence session between the first site and the second site using the selected coordinate transformation, such that at least one object at the first site is displayed at the second site at a position determined by the selected coordinate transformation. For one embodiment, an apparatus may comprise a processor and a non-transitory computer-readable medium for storing instructions that are operative, when executed on the processor, to perform the functions listed in one of the previous embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more detailed understanding may be had from the following description, presented by way of example in conjunction with the accompanying drawings. [0016] FIG. 1 is a schematic diagram depicting the overall process of matching lighting conditions for shared virtual presence.

[0017] FIG. 2 is an exemplary sequence diagram of a session initialization with two participating clients.

[0018] FIGs. 3A and 3B are an exemplary flowchart of processes executed by a client.

[0019] FIG. 4 is an exemplary flowchart of processes executed by a session manager.

[0020] FIG. 5 is a schematic diagram showing a method of combining content without making lighting adjustments in two physical environments.

[0021 ] FIG. 6 is a schematic diagram showing a method employing calibration of color balance and brightness values alone as applied to the same two physical environments shown in FIG. 5.

[0022] FIG. 7 is a schematic diagram showing a method of combing content while adjusting lighting conditions in two physical environments.

[0023] FIG. 8 is a schematic diagram showing an example method of adjusting environment alignment and lighting source origin.

[0024] FIG. 9 depicts an example wireless transmit/receive unit (WTRU) that may be used within a communications system.

[0025] FIG. 10 depicts an exemplary network entity that may be used within a communications system.

[0026] The entities, connections, arrangements, and the like that are depicted in— and described in connection with— the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure "depicts," what a particular element or entity in a particular figure "is" or "has," and any and all similar statements— that may in isolation and out of context be read as absolute and therefore limiting— may only properly be read as being constructively preceded by a clause such as "In at least one embodiment, For brevity and clarity of presentation, this implied leading clause is not repeated throughout this specification.

DETAILED DESCRIPTION

[0027] Exemplary systems and methods described herein are directed to improvement of the quality of augmented reality (AR) and virtual reality (VR) experiences where content from two or more sources is combined. For one embodiment, quality improves as similarity of lighting conditions for all mixed elements are maximized, thereby providing better visual quality and enabling better immersion and more realistic feeling of a shared virtual presence.

[0028] FIG. 1 shows a schematic diagram 150 of the overall process of matching lighting conditions for shared virtual presence. If a user initiates a shared virtual or augmented experience session, initial settings concerning the experience session and associated constraints may be sent from a client to a session manager along with a new session initialization request. Initial settings concerning the experience may include any settings that are relevant to how the session is going to be configured. These settings include, for example, information about who can join in, what kind of content elements are attached together, possible constraints determining what kind of lighting solutions are acceptable, potential other constraints to be considered by the session manager when solving the globally optimal lighting settings, such as whether geometric constraints should be considered etc.

[0029] If a new experience session is initialized, other clients may request to join in. Prior to sending join requests, each client collects information about the physical environment in which that client is executed. This information about the physical environment may be sent by the client to a session manager if a join request has been sent. A session manager compares different characteristics of the physical environments provided by each client, and based on the comparison, determines optimal alignment and settings (such as light source levels for controllable light sources and rendering settings) for each client that together optimize the quality of the experience for all participating (or connected) clients.

[0030] FIG. 1 illustrates operational principles with an example of matching lighting between two separate physical locations 152 sharing virtual content captured from both environments. Two physical locations 152 prepare to virtually share content captured from both environments. In the example shown in FIG. 1 , the content is a 3D capture of the users' appearances. The local lighting environments are measured and models of the local lighting environment are determined 154. This process may include measuring (or capturing) ambient light levels for a user environment. The models may describe the lighting in full 3D space, even though the models are shown in 2D for clarity. Controllable light sources 156 are detected, along with detection of the range of available light adjustments. Data comprising local lighting environment models of the physical locations, detected controllable light sources, and detected range of available light adjustments may be sent 158 to a session manager. For one embodiment, controllable light sources are turned off 160 and ambient user environment light levels are measured and captured. A matrix of user environment light levels may be measured (or captured) for each controllable light source within a user environment. For one embodiment, the light level for each controllable light source may be adjusted in a step-by-step (or step like) method through the range of adjustable light levels with the other controllable light sources turned off. Lighting data from each environment may be sent to a session manager. Light data may include a matrix of client user environment light levels for each controllable light source in a user environment. In an exemplary embodiment, the matrix describes the light level, luminance and color, of each setting of the controllable light source. Specific values of luminance and color coordinates can be used to represent the values. Alternately a maximum luminance and possible percentage levels may be specified. A session manager may determine optimal alignment and controllable lighting adjustments that minimize discrepancies between lighting models of the two locations. For one embodiment, each lighting model of a location represents lighting conditions of the respective location. For one embodiment, each lighting model may include a vector indicating lighting direction, where a coordinate transformation may be selected to align the vectors for two lighting models. For one embodiment, each lighting model may include coordinates indicating position of a light source, where a coordinate transformation may be selected to align the location coordinates of the light sources. For one embodiment, each lighting model may include information indicating light intensity and ability to control light intensity of each associated light source. For one embodiment, a lighting model may include position coordinates, lighting directions, lighting intensity of one or more light sources, and indications of which light sources are primary and which are secondary.

[0031] For some embodiments, determining an optimized alignment of client user environments is based in part on the location of one or more physical objects in a user environment, with an alignment being selected such that physical objects in one environment are substantially aligned with physical objects in the other environment.

[0032] For the example of FIG. 1 , a model of a combined user environment may result in an optimal alignment between the two user environments. In the example shown in FIG. 1 , three out of four controllable light sources are turned off 160 as part of the optimization. A session manager may send 162 optimal alignment, light source level, and rendering settings data to clients. The clients may set controllable lighting accordingly and may stream real-time 3D captures directly to other clients. Streamed 3D content may be rendered by receiving clients using alignment data determined by a session manager. Real-time 3D captured content may be augmented as part of the physical environment. For one embodiment, due to alignment and lighting settings determined by a session manager, the illumination of augmented reality elements 164 is similar to the illumination that those elements would have if they were physically present in the environment.

[0033] For one embodiment, a client executes a lighting environment process. A client may connect with (or poll for) controllable lighting systems in an environment. The client may determine lighting environment conditions and the range of available controls. For one embodiment, lighting environment conditions may include ambient lighting conditions and controllable lighting structures. For one embodiment, a list of controllable and non-controllable lighting systems is determined. A lighting state for each controllable lighting system may be stored prior to adjusting lighting settings. For one embodiment, controllable light source settings may be received by a client and sent to an associated controllable lighting system. For an embodiment, the client connects to a session manager and requests to initialize a new session or to join an existing session. The client sends messages describing lighting conditions and control options to a session manager. For one embodiment, messages describing lighting conditions messages may include a range of light source levels for one or more controllable light sources in a user environment. For one embodiment, lighting condition description messages may include locations of one or more physical objects in a user environment. For one embodiment, a client waits for globally optimized alignment, lighting, and rendering settings from a session manager. For one embodiment, controllable lighting system settings may be stored (or sent to a client) for a user environment previously used by a client. Upon receiving settings, a client may set controllable lighting system settings. A client may stream, receive, and render content for an experience session. For rendering, a client, for one embodiment, uses rendering settings received from a session manager that match an experience session.

[0034] For one embodiment, a session manager executes a lighting environment process. A session manager waits to receive messages from a client. If a (new) client sends a new session request to a session manager, a new session may be created. For at least one embodiment, the session manager waits for other clients to join the session. If a (new) client connects to the session, the session manager compares lighting conditions and control options to determine optimal settings where lighting conditions between different clients are minimized. For at least one embodiment, similarity of lighting conditions is maximized. Alignment, light control, and rendering settings are sent to each client. For at least one embodiment, these settings are determined as part of a global lighting optimization process. Upon sending alignment, light control, and rendering settings to each client, the session manager returns to waiting for other clients to join the session.

[0035] A session manager may be executed on a user device selected to act as a master node communicating with other clients directly, or a session manager may be executed as a dedicated service running on separate server. For one embodiment, if a session manager is executed as a separate server, all user devices connect and communicate with the server as clients.

[0036] FIG. 2 is a message sequence diagram 200 of messages and actions performed by two clients 202, 204 and a session manager 206 in an exemplary use case of setting up a shared virtual presence session between two clients in different physical locations. For the example shown in FIG. 2, client #1 (202) determines 208 the local lighting environment and available lighting control options. Client #1 (202) sends 210 a lighting model and a new session request message to a session manager 206. For this example, the session manager 206 initializes 212 a new session and waits for other clients to join the session. Client #2 (204) determines 214 local lighting environment and available lighting control options. Likewise, client #2 (204) sends 216 a lighting model and a new session request message to the session manager. The session manager 206 determines 218 alignment, lighting, and rendering settings for all clients (clients #1 and #2 in this example). The session manager 206 sends 220, 222 alignment, controllable lighting, and rendering settings messages to clients #1 and #2 (202, 204). Clients #1 and #2 (202, 204) set 224, 226 controllable lighting settings according to received setting messages. Clients #1 and #2 (202, 204) also execute 228, 230 the AR/VR experience using alignment and rendering settings received. Clients #1 and #2 (202, 204) stream 232 content data between each other for this example. [0037] FIGs. 3A, 3B, and 4 illustrate with flowcharts 300, 350, 400 the processes executed by a client and a session manager. The processes shown in FIGs. 3A, 3B, and 4 serve as example processes of the interactions between a client and a session manager. FIG. 3A and 3B show an example process executed by a client. The top of FIG. 3A shows that a user or another process may initialize 302 a client's lighting condition matching process. A new experience session may be initiated or a client may join an existing session. An explicit user request may be made to start a client process (such as by executing a client program, which for some embodiments may be a dedicated client program) or another process may make a request to start a client process (such as, for example, a social media client responding to a shared session invitation). If a client process is initialized, a client polls 304 for controllable lighting systems available in the environment and sends communications to them. Controllable lighting devices may include, for example, smart lighting systems, software-controlled curtains, electrochromic window glasses, and projection lighting.

[0038] If controllable lighting systems are available 306, some software for controllable lighting systems store 308 the current state prior to adjustment so that the current settings may be restored after an experience session ends. For one embodiment, after saving the current state, all controllable lighting systems are set 310 to minimize their impact on the local lighting conditions (for example, lights are turned off and light sources are blocked).

[0039] With available controllable lighting systems turned off, prevailing (or ambient) lighting conditions are detected 312. Lighting samples are collected 312 and light source locations, sizes, and intensities may be determined in part from the collected lighting samples to create a model of existing lighting in the environment. Light probes may be used to determine 314 a lighting model of uncontrollable environment lighting. Also, a 3D model of the environment and high dynamic range (HDR) environment maps may be created. The collected data, along with 3D models and HDR environment maps, may be stored and used to determine light sources. Iterative methods, along with heuristic rules may be used to determine a lighting model from collected sample data. The lighting model for uncontrollable sources is stored 316 in memory.

[0040] If controllable lighting systems are available 318, a client goes through an iterative process to determine a model of controllable lighting sources. A controllable lighting source is set 320 to a lighting condition setting. Lighting data is collected 322 to measure lighting conditions. The process repeats and cycles through lighting settings and parametric values for each lighting entity. A model of each available controllable lighting entity is determined 324 and stored as the process cycles through available controllable lighting entities. For one embodiment, collected lighting data collected is used to determine each model. For some embodiments, the process for creating these controllable lighting source models is similar to the process for creating uncontrollable environment lighting models. The controllable and uncontrollable lighting models are combined 326 into an overall model of environment lighting, which is sent to a session manager. [0041] FIG. 3B continues with an example client process flowchart 350. If a client requests 358 a session manager to initialize a new session, session specific settings are sent 356 by the client to the session manager along with a new session initialization request. These session specific settings may include, for example, information about which clients may join a session, information about what kinds of content elements are attached together, geometric constraints for lighting systems, and other constraints to be used by a session manager for determining globally optimal lighting settings, such as if geometric constraints may be used.

[0042] If a client joins an existing session, the client specifies which session when the client sends 356 a join request message to a session manager. The client waits 360 for a session manager to provide content alignment, controllable lighting, and rendering settings that the client may use for the session. Once the session manager sends settings to each client, the clients set 360 the controllable lighting accordingly and use the content alignment and rendering settings received from the session manager for the AR/VR experience. The experience, for example, may be a shared virtual presence between two AR head mounted display (HMD) users in two different physical locations. Alignment settings received 362 from a session manager may determine how data received from one client is registered and visualized as part of a physical environment. The alignment settings may determine 364 orientation and location for rendering 366 captured data streams received from another client. For some embodiments, a client may wait 368 for other clients to join the session and repeat the process of setting controllable lighting according to settings received from a session manager. For some embodiments, clients may dynamically join and leave a session during experience execution. A client maintains a connection with the session manager to receive updated settings during a session depending on changes in the set of clients currently connected. For some embodiments, if a client leaves 370 a session, controllable lighting settings are reset to the state prior to session adjustment.

[0043] FIG. 4 shows a flowchart 400 for a session manager lighting matching process. A process executed by the session manager for managing an experience session may be initialized by a request for a new session 402 from a client. Along with a new session request 402, a client may send (or receive 404) session settings messages to be used for a newly created session.

[0044] These session specific settings may include, for example, information about which clients may join a session, information about what kinds of content elements are attached together, geometric constraints for lighting systems, and potential other constraints to be used by a session manager when determining globally optimal lighting settings, such as if geometric constraints may be used. A session manager may also receive 406 a local lighting model. For one embodiment, a local lighting model is received from a client when the client requests a new session initialization. A session manager waits 408 to receive a termination signal, a new client request, or a client exit message. [0045] If a new session request is received 412, the session manager will also receive associated model data from the client. If a client already in a session exits the session, the session manager removes 414 the lighting model of that client from a list of active clients. If a termination session request is received, the session manager may send 416 a session termination signal to all clients, and the session may be terminated 418. For one embodiment, a session manager determines 420 for each currently participating client, alignment and controllable lighting settings. For one embodiment, alignment and controllable lighting settings are determined 422 to globally optimize lighting conditions for all clients with similar lighting conditions for all clients and global alignment of content elements to minimize differences or discrepancies of lighting sources and levels of merged content elements. Alignment, controllable lighting, and rendering settings may be sent 424 to clients. For one embodiment, alignment of user environments may include receiving geometric constraint settings, geometric constraint priority levels, and lighting matching priority levels and determining optimized alignment of client user environments for clients based on geometric constraints and priority levels of geometric constraints and lighting matching.

[0046] For some embodiments, optimal global settings may be determined using a linear system or a heuristic method embodiment. For some embodiments, optimal settings that minimize errors between lighting models may be determined iteratively. One embodiment determines the primary light source in each model (the light with the highest achievable intensity) and maximizes similarity of primary lights while optimizing the remaining lights. Such an embodiment maintains the floor level and scale of the model the same from iteration to iteration, while alignment iterations translate the model and rotate the orientation around a vertical axis. For some embodiments, a translation or rotation may be added to a mapping between different content sources that causes matching of 3D lighting models of different content source environments to be maximized. For some embodiments, the primary light source may be an ambient light source. For some embodiments, relative strength of primary light sources for two user environments may be determined. For some embodiments, determining optimized light source levels for controllable light sources may be based on the relative strength of primary light sources for two user environments. For some embodiments, orientation of two user environments with respect to a combined virtual geometry for the two user environments may be determined by maximizing a correlation of lighting information for the first user environment adjusted for the first user's orientation with lighting information for the second user environment adjusted for the second user's orientation.

[0047] For some embodiments, determining optimized light source levels for controllable light sources may include minimizing lighting source differences between two or more client user environments. For some embodiments, minimizing lighting source differences may comprise determining a primary light source for each client user environment, merging each client user environment into a merged user environment, aligning in the merged user environment the location of the primary light source for each client, adjusting the light level of the primary light sources to maximize similarity of each client user environment, and adjusting the light level of the non-primary light sources to maximize similarity for each client user environment.

[0048] For one embodiment, a system may comprise a processor and a non-transitory computer- readable medium for storing instructions that are operative when executed on the processor, to perform the functions of determining light levels for a client in a user environment, comprising (i) turning off all client user environment controllable light sources, (ii) capturing ambient user environment light levels for the client, and (iii) capturing a matrix of client user environment light levels individually for the one or more client user environment controllable light sources turned on to one or more light source levels. The embodiment may further include performing the functions of communicating the matrix of client user environment light levels to a session manager, adjusting one or more controllable light sources based on received controllable light source settings, capturing content data for the client user environment, streaming the content data to one or more other clients; and rendering content data for the client based in part on received rendering settings and received client user environment alignment settings. For one embodiment, a processor may further perform the functions of receiving a matrix of client user environment light levels for a first and a second client, determining an optimized alignment of client user environments for the first and second clients, determining optimized light source levels for the controllable light sources in the first and second client user environments, determining optimized rendering settings for the first and second client user environments, and communicating, to the first and second clients, the optimized alignment of the first and second client user environments, the optimized light source levels for the controllable light sources in the first and second client user environments, and the optimized rendering settings for the first and second client user environments.

[0049] For one embodiment, 3D data may be captured for different locations using, e.g. known VR/AR telepresence techniques such as the Holoportation technique of Microsoft, and 3D models of the lighting for each physical environment are compared with one another. An error function may be calculated to determine the amount of deviation between light sources from one model to another model. For some embodiments, based on light source brightness, a weighting of the amount of error caused by the deviation may be used. For one embodiment, an iterative method may be used to determine an optimal transformation that minimizes the error function. In embodiments using a Holoportation telepresence system, alignment of an environment controls how content from one location is mapped to an environment where another user is viewing the content. This mapping may be used by both participating users, where the 3D captured appearance of the other user is rotated and translated. Such a mapping may be used to maximize continuity of lighting between the physical space and the augmentation. For one embodiment, a telepresence session comprises a virtual environment that enables users physically located at disparate locations to interact as if present in the same location. For one embodiment, a telepresence session may include virtual representations of people and objects, such as a table or a chair. For one embodiment, a telepresence session may use a plurality of light intensity settings for a plurality of light sources that are adjusted to minimize, between two or more locations, differences in light intensity of light sources. For one embodiment, adjustments are made to the settings and communicated to each location. For one embodiment, light intensity settings are adjusted at the physical light sources, light intensity is measured for each light source, and readjustments may be made if differences in light intensity still exist. For one embodiment, light intensity of two light sources for a telepresence session may be adjusted if each light source is controllable. For one embodiment, a third site may join a telepresence session that already includes two sites. A third lighting model representing lighting conditions of the third site may be received, and a coordinate transformation between the first and third sites may be determined. The third site may be added to the telepresence session using the determined coordinate transformation and one or more objects from the first site may be displayed at the third site at a position determined by the coordinate transformation. For one embodiment, light intensity settings of one more light sources at the third site may be adjusted to minimize differences in light intensity between the three sites, and the adjusted light intensity settings may be communicated to each site.

[0050] In one exemplary embodiment, an error function may be calculated based on light levels of a first light source Li in a first environment and a second light source I2 in a second environment. A base error function may be calculated using a square of differences method as follows:

BaseError(L ₁ , L ₂ ) = \L — L ₂ \ ²

[0051] In some embodiments, a full color error metric may be used based on, for example, luminance and chrominance differences. In some embodiments, an error function takes into consideration not only differences between light levels in different environments, but also differences from initial light levels in those environments. For example, it may be desirable to minimize lighting level changes from initial lighting values. In an exemplary embodiment, this is done by forming a total error summing the error between each modified level and the error between each original value. Let Oi denote the original light level at site i, and let M, denote the modified level at site i. The base error function may be used to define a total error function. The total error may be calculated as the sum over sites i and j of base error functions.

TotalError = BaseErroriO^ M^ + BaseError M M _j )

TotalError = - M _;

[0052] For example, consider a case in which modified levels are set to the same value Mi=M for all i. In that case, the last error term is zero. The value of M is then selected to minimize the total modification among all sources; that is, M is chosen to minimize sum of errors due to all modifications. The value of M is chosen to minimize the remaining total error i.e.∑;|0; - | ² .

[0053] In other embodiments, other base and total error functions may be used. For example, the basic error may be weighted. Similarly, the formulation of total error may be weighted.

[0054] Using the error function described above, optimal orientation and parameters for the controllable lights may be solved with several embodiments. One embodiment may use all different available light settings as individual light sources. With this embodiment, each light source with each parameter setting may be defined by a 3D vector of the light location and a 3D vector of intensity (luminance) and chrominance (hue) values, or alternatively, by a 3D vector of the light location and a 3D vector of RGB values.

Light Location LI loc = ( i, i,Zi)

Light L1 values depend on the control parameters (intensity, chrominance) with a step size 0.1 V1 resulting in a 10 by 10 matrix:

^" vOO · · · i;09 ^"

VI = ; ··. ; , where v ^ = (intensity, chrominance) or = (R, G, B).

-v90 ··· v991

From each entry in V1, a light source is created by combining light location L1 /oc_and light color metrics Vij . This combination of parameters results in a list of lights:

( i _< Yi _> ^z i _< intensity ₀₀ , chrominance ₀₀ ) ( i _< Yi _> ^z i _< intensity ₀₁ , chrominance ₀₁ ) ( i< Yi> ^z i< intensity ₀₂ < chrominance ₀₂ )

( i _< Yi _> ^z i _< intensity ₉₉ , chrominance^

[0055] For one embodiment, the same procedure may be done for each light source in both the first and second environment. Two sets of light sources with all available color metrics may be denoted M and S. For one embodiment, a point set registration may be used to calculate the best orientation and parameters. For a point set registration, best fit for two point sets may be solved by calculating a transformation T that maps two point sets to as close a match as possible, where T(M) = S. For one embodiment, a transformation T is limited to rigid transformations. For one embodiment, transformation T may be limited further to translations along a floor plane and rotations around a vertical axis to use an original floor level and to not have augmented elements floating in the air, for example. One embodiment may further limit a transformation to affect only a 3D location of a light. One embodiment may have solutions that are constrained so that each light location from both M and S may be used only once. For one embodiment, a transformation minimizes the TotalError parameter. An exemplary embodiment of a transformation for sets described above using error function TotalError, may be solved using a point set registration algorithm, such as iterative closest point or robust point matching, that is modified to account for specific constraints described above. For one embodiment, a coordinate transformation may include a translation of a coordinate system of one location (or site) to a coordinate system of a second location (or site) and a rotation of one or both coordinate systems for each site. For one embodiment, a coordinate transformation may include a transformation of coordinates at a first site to coordinates of a combined virtual geometry and a transformation of coordinates at a second site to coordinates of a combined virtual geometry.

[0056] In another embodiment, parameters and a transformation may be solved using a dedicated shader program executed on a graphics processing unit (GPU). For this embodiment, an error function does not use 3D locations of different light sources, but rather uses shading of a virtual test object resulting from lighting parameters and a specific transformation. For this embodiment, a spherical virtual test object is rendered using a captured environment lighting model. Resulting shading of the spherical object is projected into a texture, which maps a surface of the sphere into a 2D image plane. With parallelism of GPUs, a test object with different lighting parameters and different transformations may be rendered effectively using the lighting model of the first space and compared against similar renderings using an environment lighting model of the second space.

[0057] Shown below is a pseudo code listing for one embodiment for calculation of lighting parameters. A rendering of a perfectly Lambertian sphere is projected onto a 2D texture for each stochastic sampling of parameter values in a set of light sources M. A 3D array of locations within light source S is stored. Similarly, for each location created in a set of light sources S, a rendering of a perfectly Lambertian sphere is projected onto a 2D texture for each stochastic sampling of parameter values in a set of light sources S. For each texture previously calculated, pixel value errors are calculated between the sets of light sources M and S and all possible texture UV offset values. A texture pair of M and S textures with a UV offset that minimizes the error is selected. If the smallest error is less than a threshold, lighting parameter values used to render M and S textures are outputted. Otherwise, the process repeats to use a stochastic sampling closer to the current best match.

Stochastically sample set of lighting parameter values for each light source in the Mj

For each selection of parameter values in the previously selected set, do:

{Render a perfectly lambertian sphere using the lighting model of M and set of lighting parameters; Project rendering to a 2D texture and store texture}

Create a 3D array of location within the S. If available

transformations are constrained, that can be taken into an account here;

Stochastically sample set of lighting parameter values for each light source in the S;

For each location created for the Sj do:

{For each selection of parameter values in the previously selected set, do:

{Render a perfectly lambertian sphere at the current location selected from the location array using the lighting model of S and set of lighting parameters;

Project rendering to a 2D texture and store texture}}

For each texture resulting from the previous step, do:

{Compare texture with all the textures stored from the rendering of M with all possible texture UV offset values. Since we are using spherical objects, offsetting texture corresponds rotating the object. If rotations are constrained, it can be taken into an account here;

Calculate error between pixel values of textures}

Select texture pair of M and S textures with the UV offset that

results the smallest error;

If smallest error is below a threshold:

{End and return lighting parameter values that were used to render M and S textures with the minimal error, transformation used when rendering minimal error S texture and orientation which

corresponds with the minimal error texture UV offset}

Else:

{Go back to the start and use the best match transformation from this iteration as the initial values and set the stochastic sampling to focus on values closer to the current best match and transformation steps smaller closer to the current best match.}

[0058] After determining globally optimized lighting settings, a session manager may estimate each client's deviation from the globally optimized lighting settings. If a client involved with the optimization does not converge to globally optimal lighting settings, a session manager may fail to produce lighting conditions for all clients that are close to each other (where globally optimized settings have a big error function residue). This situation may occur if one or more clients have extremely different lighting conditions, such as outdoor versus indoor locations or environments without controllable lighting systems. In one embodiment for such situations, a session manager may determine client rendering configuration settings for each client individually. If rendering settings are determined for each client individually, a session manager may minimize remaining lighting condition differences between clients. This minimization may not completely match content alignment and controllable lighting settings. A session manager sends resulting content alignment, controllable lighting, and rendering settings to each client. For at least one embodiment, a session manager returns to wait for new clients to join a session or existing clients to exit a session or terminate a session. If a session manager receives a termination session message or signal, the session manager ends the current session.

[0059] Depending on the methods used for lighting capture and desired accuracy, lighting data may be described with several different formats. One method maximizes a lighting model's accuracy, describing total amount of light emitted from various light sources using units such as lumens to describe total radiated energy in full spectrum of visible light. However, the method measures total emitted energy of light sources in the environment. Some cameras capture only a limited band of visible light, though the band may be changed dynamically by increasing or decreasing the camera's exposure time. For example, in the case of camera- based lighting capture, to describe environment lighting with total emitted energy values (such as lumens), the measuring method controls the camera exposure settings to produce multiple exposure captures and capture a larger spectral range. The camera is calibrated to scale captured values accurately.

[0060] To avoid calibration of systems, another method uses less precise lighting values, where the overall model is more of an approximation than an exact description of light energy emissions. With this more relaxed method, a lighting model may use, for example, RGB values provided directly by a lighting capture system to describe approximate intensity and color of the lights in the lighting model. Depending on the color depth provided by the lighting capture system, such as 8- or 16-bit, dynamic range and fidelity of the lighting model may vary. If capturing systems are uncalibrated and vary from client to client, local lighting models provided by different clients to a session manager may vary in range and accuracy. Using data with such approximations may enable deployment across more systems because a wider range of light capture methods may be used without calibration. With this method, a lighting model may approximate light sources with RGB values that cover an estimated band of visible light. For at least one embodiment, measurements from different clients (and slightly different light spectrum bands) may be standardized to cover a uniform spectrum band.

[0061] Using either lighting description method, a three-dimensional (3D) lighting model is created. This 3D lighting model may use 3D vectors for light direction or may describe lights as entities in 3D space with a 3D location and area for each light source. For example, a light probe may produce a light model where lights are described as: general directions from which they emit light to a measuring point, areas of lights, intensities, and colors. For another embodiment, different methods may be used to collect samples from several points and to produce a lighting model that estimates full 3D locations, sizes, intensities, and colors of each light source.

[0062] For one or more embodiments, real-time 3D capture of content may be performed using RGB- D sensors. For one embodiment, a matrix of client user environment light levels may be communicated to a session manager using RGB-D values. For reconstructing lighting model RGB-D sensor data, several methods may be used. In some embodiments, lighting model capture with RGB-D sensor data is performed using techniques described in Lukas Gruber, Jonathan Ventura, & Dieter Schmalsteig, Image-Space Illumination for Augmented Reality in Dynamic Environments, 2015 IEEE VIRTUAL REALITY (VR), 127-134 (2015). Methods using RGB-D sensors may determine static 3D reconstruction of geometries outside a sensor's current field of view, dynamic geometries inside a sensor's field of view, and dynamic estimations of environment lighting from reflections observed in a scene inside a sensor's field of view. For RGB-D sensor methods, an environment lighting model may be approximated without probes, extensive user input, or preprocessing of data. Also, a lighting model may be continually updated, which enables interactive content lighting that matches scene run-time changes. Lighting model capture and rendering methods that may be used in some embodiments include those described in Kronander, Joel, et al., Photorealistic Rendering of Mixed Reality Scenes, COMPUTER GRAPHICS FORUM, 643-665 (2015). This journal article describes different probe-based methods, indoor and outdoor use methods, and data capturing methods.

[0063] Content capture may be handled separately from lighting capture, and for some embodiments, lighting capture may be performed with completely different sensors and methods than content capture. However, for at least one embodiment, if identical lighting conditions are not possible for all participating clients or for all content elements merged together, content rendering settings may be used to further improve matching of lighting for different content elements.

[0064] To achieve higher end quality, some embodiments address, at each stage of the pipeline, content capturing conditions, content rendering methods, and factors that impact how a viewer perceives rendered content, in addition to matching lighting conditions between different content elements. Similar to light model methods, content rendering adjustments performed by a session manger may be executed more accurately if content capture and display devices are calibrated. Likewise, with relaxed content rendering constraints, content values may be approximated.

[0065] For relaxed content rendering methods, rendering settings may improve lighting model matching. For example, such improvements may occur if lighting models of clients are close to each other (such as if the main light source is in a similar location) or if the same overall brightness level may not be achieved. For some embodiments, a session manager may set a client's settings to use a higher overall environment lighting brightness to increase the brightness of content elements received from a client with a lower overall environment brightness, and vice versa, a client with a lower environment brightness lowers the brightness of content received when rendering the content. Similar capture settings may be used for both clients. If capture conditions are uncontrolled (for example, a sensor automatically adjusts the dynamic range used for capturing the visual content, such as via camera shutter speed), an adjustment of the rendering brightness may be done differently. For one embodiment, if capturing conditions are uncontrolled, a session manager compares content captured by different clients and sets rendering settings for each client so that an average tonal range of content elements is close for each client.

[0066] Accuracy of visual reproduction of captured elements may vary from one client to another. For example, if one user's appearance is captured with RGB-D sensors in one location and augmented to another user wearing an optical see-through augmented reality (AR) HMD, there may be multiple factors that may hinder accurate content reproduction. For such an example, content reproduction may be limited by the method of content capture. Capturing systems and content formats may limit the spectral range band and capture more of an approximation of the true physical light reflected by physical elements. Lack of calibration of capturing devices may be another reason content data is more of an approximation of light energy intensities. Additionally, display systems may limit the ability to reproduce content. Environment conditions may further limit how displayed content is perceived. Effects of environmental lighting conditions and display brightness on user perception of images are described in Hisashi Kobiki and Masahiro Baba, Perceiving Perceived Brightness of Displayed Images Over Different Illumination Conditions, 2010 17TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2485-2488 (2010).

[0067] With displays having different color gamut output abilities, displays may lack the ability to reproduce content with the same tonal range measured in the content capture. In a perfect environment, content capture, lighting capture, and content rendering is performed in controlled and calibrated conditions similar to the conditions described in the article by Kobiki and Baba. These methods maximize similarity of perception for data captured in one location and rendered in another location. If identical lighting conditions are not possible for all clients, a model may be created to determine what changes may be made to improve content perception. A display that is limited with respect to color gamut output range may be limited in tonal range for content reproduction. Also, with a display that is limited with respect to color gamut, brightness range, and opacity range in comparison with some optical see-through AR HMDs, only a limited portion of content may be rendered so that the content is perceived as the content was captured or as the original physical element would be perceived by a naked eye before capture. While some capture and display technologies may have shortcomings, methods and systems described herein improve the quality of experience by matching lighting conditions for different content elements, even if lighting conditions are approximated.

[0068] In one embodiment, environment lighting conditions and geometric characteristics are determined. In one or more embodiments, geometric constraints set conditions on how different content elements may be aligned. Depending on geometric constraints, there may still be room to further adjust alignment to optimize global lighting and to maximize uniformity of lighting for all participants (or users) and elements without violating geometric constraints. In one embodiment, each client determines a 3D reconstruction model of the environment where the client is executed and sends the 3D reconstruction model to the session manager. For one embodiment, a client initializing a session sets geometric constraints and priorities (such as, more priority on geometric constraints or on lighting matching). The client may communicate the geometric constraints and priorities to a session manager. For another embodiment, 3D reconstruction may be performed by a session manager while a client streams RGB-D data collected with client-accessible sensors. For one embodiment, global optimization of content may be based in part on geometric constraints and per client geometric layout, resulting in global content element alignments and lighting settings which avoid geometric conflicts between clients and providing lighting conditions maximized with respect to similarity for each client.

[0069] In an exemplary embodiment, a client performs a full iteration of available controllable lighting systems and individual elements of a system. In one embodiment, available controllable lighting system configurations are stored if a system is detected and characteristics are determined. If a client is executed in an environment where the client was previously used, the characteristics of controllable lighting systems available may be retrieved from storage. This action avoids repeated full iteration and detection of system characteristics each time a client is executed in the same environment.

[0070] Exemplary embodiments employ known techniques disclosed in relation to, e.g. Holoportation, to collect real-time captures of several users and merged them with captures of physical environment elements of each user by visualizing other participants as 3D graphics attached to each user's physical environment as if they were physically present in the same space. Systems and methods described herein may be used to stay within geometric constraints and to determine alignment and controllable lighting settings that enable all participants to enjoy a maximized quality of experience.

[0071] FIGs. 5 to 7 illustrate the differences for: (i) not matching lighting between locations; (ii) only calibrating color balance and brightness levels; and (iii) using systems and methods described herein. FIG. 5 shows an example of a method 600 of combining content without making lighting adjustments in two physical environments. On the top left of FIG. 5, there is a picture of a physical environment 602 that will be used for capturing a user's appearance. For this example, the primary light source 604 is on the right side of the environment while a secondary source 606 is on the left side. Shadows 608 appear on the left side of the user while the right side 610 of the user is bright. The user is shown with two levels of shading. One level of shading 608 is indicated with diagonal lines, and a second, lighter level of shading 610 is indicated as a white shape on the right side of the user. A bright spot 612 is shown on the right side of the table.

[0072] On the top right of FIG. 5, there is a picture of a physical environment 614 that will be augmented with a user's captured appearance. For this example, the primary light source is on the left side of the environment with a bright spot 616 on the left side of the ball. Levels of shading 616, 618, 620, 622 are also shown on the ball with shading level 1 (616) corresponding to the bright spot on the ball and shading level 4 (622) being the darkest shadow on the ball. The table also has a bright spot 624 on the left side of the table.

[0073] The two physical environments 602, 614 are directly combined 626 without adjusting lighting in either physical environment. Visual artefacts are present in the resulting augmented environment 628. The bright spot on the ball 630 and the bright area on the user 632 are on conflicting sides due to the primary light source being on opposite sides of the respective physical environments. Also, the overall lighting level is completely different in the two physical environments, causing an overall lighting level conflict in the combined environment. Additionally, for this example, a bright spot 634 is shown on the left side of the table.

[0074] FIG. 6 shows an example of a method 700 that only calibrates color balance and brightness levels applied to the same two physical environments 702, 730 shown in FIG. 6. The top of FIG. 6 shows the same two physical environments 702, 730 shown in FIG. 6. For this method 700, light sensors 714, 728, 734 are placed in physical environments 702, 716, 730 to measure the overall intensity in each environment. Readings from these sensors 714, 728, 734 are used to determine an overall combined intensity of the lighting in both environments 702, 730. Lights 704, 706, 718, 720 in the environment 702, 716 with the user are dimmed to make lighting intensity in both locations 702, 730 match. Dimming the lights 718, 720 causes the shading 722, 724 of the user to become darker and the bright spot 712, 726 on the table to decrease in intensity. The physical space 730 that is augmented with the user has a bright spot 732 on the ball and a bright spot 736 on the left side of the table. The dimmed user environment 716 is combined with the other environment 730. The combined environment 738 contains visual artefacts regarding bright spots 740, 742 on the user and the ball. The right side of the user is bright, but the left side of the ball is bright. This conflict is not resolved by calibration of color balance and brightness values alone. The combined environment 738 also contains a bright spot 744 on the left side of the table.

[0075] FIG. 7 shows an example 800 of adjusting lighting conditions in the same two physical environments 802, 828 shown in FIG. 7. The top of FIG. 7 shows the same two physical environments 802, 828 shown in FIG. 7. For this method 800, a session manager receives lighting models and available controllable lighting systems, adjustment settings, and adjustment setting ranges for each physical environment. Based on the lighting model and controllable lighting optimization, the method 800 adjusts the overall lighting intensity level to match in each environment. The method 800 also controls lighting so that the whole lighting model maximizes similarity of the two environments 802, 828. In an exemplary embodiment, no alignment adjustment is made, only adjustment of the controllable lighting. In the example shown in FIG. 7, the initial secondary light source 806, 818 is increased in intensity for the adjusted physical space 814. The initial primary light source 804, 816 is turned off for the adjusted physical space 814. The shading of the user changes to have the darkest area 822 on the right side, a darker area 822 in the center, and a bright area 824 on the left side. Also, a bright spot 826 appears on the left side of the table due to the lighting changes. The bright spot on the table matches the bright spot 830 appearing in the physical space 828 for augmenting with the user. In the combined environment 832, no artefacts are seen, especially the artefact for the bright side of the user and the ball that occurs in the methods of FIGs. 5 and 6.

[0076] FIG. 8 shows an example 900 of combining (including adjusting) two environments into one environment. The first (top) environment 902 shows a window where the sun 904 is shining in the window onto a first person 906. The second (middle) environment 908 is a separate physical environment where two triangle-shaped lights 910 are shining on a second person 912. The combined (bottom) environment 914 is a combination of the first and second environments 902, 908. For this example, the second (primary) light source 910 for the second environment 908 is rotated to be a light source 920 originating from the right, similar to the sun 904 in the first environment 902. Also, the alignment of the first and second environments 902, 908 is adjusted in the z-plane (into the paper) and in the x-y plane. The alignment of the first and second environments 902, 908 results in an adjusted second light source 920 and an adjusted second person 922. The outline of the unadjusted second environment 916 is shown with dashed lines, while the outline of the adjusted second environment 918 is shown with thick, solid lines. Other examples may adjust the light and alignment in other ways.

[0077] In addition to augmenting other users captured in real-time as part of a physical environment, real-time captures may be embedded into a virtual environment. A virtual environment may be a completely synthetic 3D virtual environment or a capture of a real-world physical location where none of the participants are physically present. These examples use slightly different methods as described below.

[0078] Examples of a virtual environment where real-time captured visualizations of users are merged are 360° 3D video, a pre-captured light field, or a completely synthetic virtual world with pre-calculated lighting which may not be changed at run-time. For these examples, for virtual content that has static lighting, the environment may be handled similarly to a client connecting to a session with no controllable lighting available. A model of static lighting is created for this client. For one embodiment, a method for determining optimal alignment and lighting settings searches for the best alignment between a virtual environment with static lighting and controllable lighting settings for clients with control over at least a portion of local lighting systems. [0079] Real-time captured visualizations of participating users may be merged with a completely synthetic virtual world that may be fully re-lit at run-time. Optimization methods search for the best alignment match for each user where each user may have similar lighting conditions. Each user environment is re-lit based on this optimization method. The virtual environment may be handled similarly to a client with only fully controllable lighting models (no uncontrollable lighting models). For virtual elements that may be re-lit with any lighting model, the lighting model description may be able to determine lighting which also allows the number of light emitters to be chosen freely.

[0080] For one embodiment, pre-captured content playback elements may be handled similarly to clients without controllable lighting because a lighting setup used during content capture may not be changed after capture. Furthermore, depending on the capture method, a pre-captured part of the content may have alignment constraints. For example, if a sensor used for capture does not provide full, 360° visualization of a captured target, there may be alignment constraints, and a lighting model used by pre-captured content elements may be used (and stored) if a lighting model may not be retrieved from captured content.

Network Architecture.

[0081 ] A wireless transmit/receive unit (WTRU) may be used as a session manager or as a client device to capture environments in embodiments described herein.

[0082] FIG. 9 is a system diagram of an example WTRU 102. As shown in FIG. 9, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, a non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. The transceiver 120 may be implemented as a component of decoder logic 119. For example, the transceiver 120 and decoder logic 119 may be implemented on a single LTE or LTE-A chip. The decoder logic may include a processor operative to perform instructions stored in a non-transitory computer-readable medium. As an alternative, or in addition, the decoder logic may be implemented using custom and/or programmable digital logic circuitry.

[0083] It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment. Also, embodiments contemplate that the base stations 114a and 114b, and/or the nodes that base stations 114a and 114b may represent, such as but not limited to transceiver station (BTS), a Node-B, a site controller, an access point (AP), a home node-B, an evolved home node-B (eNodeB), a home evolved node-B (HeNB), a home evolved node-B gateway, and proxy nodes, among others, may include some or all of the elements depicted in FIG. 9 and described herein.

[0084] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 9 depicts the processor 118 and the transceiver 120 as separate components, the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0085] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, as examples. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. The transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0086] In addition, although the transmit/receive element 122 is depicted in FIG. 9 as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0087] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11 , as examples.

[0088] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0089] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. As examples, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium- ion (Li-ion), and the like), solar cells, fuel cells, and the like.

[0090] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. The WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0091] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0092] FIG. 10 depicts an example network entity 190 that may be used within a communication system. As depicted in FIG. 10, network entity 190 includes a communication interface 192, a processor 194, and non-transitory data storage 196, all of which are communicatively linked by a bus, network, or other communication path 198.

[0093] Communication interface 192 may include one or more wired communication interfaces and/or one or more wireless-communication interfaces. With respect to wired communication, communication interface 192 may include one or more interfaces such as Ethernet interfaces, as an example. With respect to wireless communication, communication interface 192 may include components such as one or more antennae, one or more transceivers/chipsets designed and configured for one or more types of wireless (e.g.,

LTE) communication, and/or any other components deemed suitable by those of skill in the relevant art. And further with respect to wireless communication, communication interface 192 may be equipped at a scale and with a configuration appropriate for acting on the network side— as opposed to the client side— of wireless communications (e.g., LTE communications, Wi-Fi communications, and the like). Thus, communication interface 192 may include the appropriate equipment and circuitry (which may include multiple transceivers) for serving multiple mobile stations, UEs, or other access terminals in a coverage area.

[0094] Processor 194 may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated DSP.

[0095] Data storage 196 may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random- access memory (RAM) to name but a few, as any one or more types of non-transitory data storage deemed suitable by those of skill in the relevant art may be used. As depicted in FIG. 10, data storage 196 contains program instructions 197 executable by processor 194 for carrying out various combinations of the various network-entity functions described herein.

[0096] In some embodiments, the network-entity functions described herein are carried out by a network entity having a structure similar to network entity 190 of FIG. 10. In some embodiments, one or more of such functions are carried out by a set of multiple network entities in combination, where each network entity has a structure similar to network entity 190 of FIG. 10. In various different embodiments, network entity 190 is— or at least includes— one or more of (one or more entities in) RAN 103, (one or more entities in) RAN 104, (one or more entities in) RAN 105, (one or more entities in) core network 106, (one or more entities in) core network 107, (one or more entities in) core network 109, base station 114a, base station 114b, Node-B 140a, Node-B 140b, Node-B 140c, RNC 142a, RNC 142b, MGW 144, MSC 146, SGSN 148, GGSN 150, eNode B 160a, eNode B 160b, eNode B 160c, MME 162, serving gateway 164, PDN gateway 166, base station 180a, base station 180b, base station 180c, ASN gateway 182, MIP-HA 184, AAA 186, and gateway 188. Other network entities and/or combinations of network entities may be used in various embodiments for carrying out the network-entity functions described herein, as the foregoing list is provided by way of example and not by way of limitation.

[0097] Note that various hardware elements of one or more of the described embodiments are referred to as "modules" that carry out (perform or execute) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by those of skill in the relevant art for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and those instructions may take the form of or include hardware (or hardwired) instructions, firmware instructions, software instructions, and/or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM or ROM. [0098] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element may be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.