NEAR-EYE DISPLAY

BACKGROUND

[0001] Traditional computer user input methods involve physical control devices such as a smartphone touchscreen, a gamepad, or a remote control for a gaming system. These devices are familiar and comfortable to users, and do provide a precise input reading. However, physical input devices can also be inconvenient. Users may need to retrieve a device from their pocket or bag before it is usable. In addition, physical input devices may not be usable when out-of-view, resulting in a user occasionally having to interrupt their activity to look at the device.

[0002] A near-eye display (NED) device, such as a head mounted display (HMD) device, may be worn by a user for an augmented reality (AR) experience or a virtual reality (VR) experience. Natural user input such as gaze action, speech and hand gestures allow users to interact with virtual objects in a three dimensional space around the user. However natural user input presents challenges in verifying selection of an item, adjusting for physical characteristics of a user, and having real world locations as a display space as opposed to predetermined electronic display spaces of known dimensions.

SUMMARY

[0003] The technology provides one or more embodiments of a web-like hierarchical menu interface for a near-eye display (NED) system. The interface comprises stored data representing one or more web-like hierarchical menu display configurations. When executing on one or more processors, an embodiment of the web-like hierarchical menu interface selects and adapts a display configuration for display in the user's three dimensional environment and for the user's head comfort based on monitoring sensor data from one or more sensors of the NED system when worn by a user. The sensors may also include image capture devices. The interface also identifies user interactions with a displayed menu based on the sensor data. In many embodiments, the web-like hierarchical menu interface provides a hands-free user interface.

[0004] The technology provides one or more embodiments of a method determining a web-like hierarchical display configuration for displaying a menu by a near-eye display (NED). An embodiment of the method comprises identifying by one or more processors data indicating a menu is to be displayed by the near-eye display (NED). An available menu space about a user head and a user head view direction are determined by the one or more processors based on a stored three dimensional (3D) mapping of an environment of the user. One or more processors select a position for the menu in a display field of view of the NED based on the user head view direction of a menu activation location.

[0005] A web-like hierarchical menu display configuration is selected based on the available menu space, the user head view direction and user head comfort rules. The one or more processors adapt menu items to the selected web-like hierarchical display configuration based on the available menu space, the user head view direction and user head comfort rules. The one or more processors cause the near-eye display to display the menu with its menu items adapted in accordance with the selected web-like hierarchical menu display configuration.

[0006] The technology provides one or more embodiments of a method for adapting activation of a web-like hierarchical menu interface being executed by one or more processors of a near-eye display (NED) system based on user selection metrics. An embodiment of the method comprises adjusting by one or more processors activation parameters in menu item selection criteria for the web-like hierarchical menu interface based on stored user head motion data. Head view direction of the user is monitored for determining whether a change in the head view direction activates a change in a menu display layout. Additionally, it is determined whether a change in the available menu space activates a change in the menu display layout. User selection of a menu item is identified, and feedback confirming user selection of the menu item is displayed by the NED under control of the one or more processors.

[0007] The technology provides one or more embodiments of a system for a near-eye display (NED) system for displaying a menu in a web-like hierarchical menu display configuration. An embodiment of the system comprises a near-eye support structure, and a near-eye display (NED) supported by the near-eye support structure and having a field of view. An image generation unit supported by the near-eye support structure outputs image data, like a menu in a web-like hierarchical menu display configuration, for display by the near-eye display (NED).

[0008] One or more processors have access to a memory storing web-like hierarchical menu display configurations and web-like hierarchical menu display configurations rules. Being communicatively coupled to the image generation unit, the one or more processors control display of a menu in accordance with one of the stored web-like hierarchical menu display configurations and user head comfort rules. [0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram depicting example components of one embodiment of a near-eye display (NED) device system.

[0011] Figure 2A is a side view of an eyeglass temple of a frame in an embodiment of the NED device having an optical see-through AR display, and the NED device being embodied as eyeglasses providing support for hardware and software components.

[0012] Figure 2B is a top view of an embodiment of a display optical system of the embodiment of the NED device.

[0013] Figure 3A illustrates an example of a web-like hierarchical display configuration with flexible links connecting levels of menu selections included in an embodiment of web-like hierarchical menu user interface.

[0014] Figure 3B illustrates another version of the example web-like hierarchical menu display configuration of Figure 3A with an example of another visual aid for illustrating progress in making a menu selection.

[0015] Figure 4 A illustrates another example of a web-like hierarchical menu display configuration with shape flexible links connecting levels of menu selections. Figure 4B illustrates another example of a web-like hierarchical menu display configuration with zig zag shaped flexible links connecting menu items.

[0016] Figure 5A is a block diagram of an embodiment of a system from a software perspective including a web-like hierarchical menu interface for operation within a near- eye display device.

[0017] Figure 5B illustrates an example of a web-like hierarchical display configuration data set illustrating examples of data items which may be used in embodiments of the technology.

[0018] Figure 5C illustrates an example of a user selection metrics data set illustrating examples of data items therein which may be used in embodiments of the technology.

[0019] Figure 6 is a flowchart of an embodiment of a method for determining a web-like hierarchical menu display configuration for displaying a menu by a near-eye display (NED).

[0020] Figure 7 is a flowchart of an embodiment of a method for adapting activation of a web-like hierarchical menu interface being executed by one or more processors of a near-eye display (NED) system based on user selection metrics.

[0021] Figure 8 is a flowchart of an embodiment of a process for adapting activation parameters for selecting a menu selection from a web-like hierarchical menu based on user head motion data.

[0022] Figure 9 is a flowchart of an embodiment of a method for monitoring head view direction and determining whether a change in the head view direction activates a change in a menu layout.

[0023] Figure 10 is a flowchart of an embodiment of method for identifying user selection of a menu item.

[0024] Figure 11 is a flowchart of an embodiment of a method for adjusting a menu display configuration based upon a change in available menu space.

[0025] Figure 12 is a block diagram of one embodiment of a computing system that can be used to implement a network accessible computing system, a companion processing module or control circuitry of a near-eye display device.

DETAILED DESCRIPTION

[0026] In an augmented reality or virtual reality environment, menus may be associated with menu activation locations which are positions in a user 3D environment, tethered to real or virtual objects in the user environment, body locked to a user, and even head locked to a user. An example of an event triggering display of a menu is detection of a user perspective or head view direction including one of these menu activation locations. In other words, detection of the menu activation location in a field of view of the near-eye display. For example, when a user looks at the body locked position associated with the menu or a real or virtual object associated with the menu, a menu may be displayed or pop up for the user. In some instances, a cursor appears near the menu activation location in the field of view.

[0027] One or more embodiments of a web-like hierarchical menu display configuration links menu levels and menu items within a menu level with flexible spatial dimensions for menu elements. An embodiment of a web-like hierarchical menu interface selects and adapts the configuration to display a menu hierarchy with a goal of maintaining user head comfort for a wearer of a near-eye display (NED) device. In fact, besides distances between menu items, many aspects of a web-like hierarchical menu display configuration can be defined to be flexible, as governed by rules associated with the configuration, such as size and shape of menu item, number of levels displayable simultaneously, shape variation of flexible links, angle separation between menu items and the like.

[0028] A configuration is web-like in the sense of a spider's web. The configurations are analogous in that they can be of varying overall size, and include portions which are more compact than others depending on the space available for their display like natural spider webs are based on the spaces in which they are woven.

[0029] In many examples, natural user input in terms of user head view direction and change in user head position can be used to select a menu item. For example, user focus on a menu item may be identified by head view direction, which is from which angle a user is viewing a scene. The menu item may be highlighted or otherwise visually indicated to be a selection candidate, and the user wearing the NED system can perform a confirmatory action with his or her head like a nod as a change in head position, or a side to side change in head position traversing the menu item, sometimes referred to as a head swipe while the user head view direction stays within a range with respect to the menu item. (One or more sensors on the NED will detect the head swipe.) Head based movements can be determined from an inertial sensing unit including an accelerometer which is a less costly sensor package than an eye tracking system in terms of power.

[0030] Some embodiments of the web-like hierarchical menu interface provide hands- free interaction with a menu. This allows the user to keep performing everyday activities with their hands and use their head to navigate the virtual menu. Additionally, gesture recognition, if available, can be computationally expensive. An application may already be using gesture recognition resources for identifying hand and finger gestures as user input related to an activity being provided by the application. Other embodiments may also recognize gaze based actions and speech as user actions for selecting an item. For example, gaze and speech may be fallback user selection actions if selection resolution issues arise based on head position or head movement. As mentioned above, gaze detection uses additional hardware and processor time for tracking eye movements and tracking gaze changes. Sound or voice input may be distracting, if not to the user, to others in the environment and may also reduce privacy.

[0031] Figure 1 is a block diagram depicting example components of one embodiment of a near-eye display (NED) device system. In the illustrated embodiment, the system includes a near-eye display (NED) device as a head mounted display (HMD) device 2 and communicatively coupled via a wire 6 to another computer system identified as a companion processing module 4. In other embodiments, wireless communication between the NED device 2 and a companion processing module 4 may be used. In other embodiments, the functionality of the companion processing module 4 may be integrated in software and hardware components of the display device 2.

[0032] In these embodiments, NED device 2 is in the shape of eyeglasses in a frame 115, with a respective display optical system 14 positioned at the front of the NED device to be seen through by each eye when the NED is worn by a user. In this embodiment, each display optical system 14 uses a projection display in which image data is projected into a user's eye to generate a display of the image data so that the image data appears to the user at a location in a three dimensional field of view in front of the user. Each display optical system 14 is also referred to as a display, and the two display optical systems 14 together may also be referred to as a display.

[0033] In these embodiments, frame 115 provides a convenient eyeglass frame as a near-eye support structure for holding elements of the NED device 2 in place as well as a conduit for electrical connections. The frame 115 includes a nose bridge 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136. A temple or side arm 102 of the frame rests on each of a user's ears, and in this example, the right side arm 102r is illustrated as including control circuitry 136 for the NED device 2.

[0034] In some embodiments, companion processing module 4 is a portable computer system like a mobile device (e.g. smartphone, tablet, laptop). One or both of the NED device and the companion processing module 4 may communicate over one or more communication networks 50 to one or more network accessible computer systems 12, whether located nearby or at a remote location.

[0035] An application may be executing on a computer system 12 which interacts with or performs processing for an application executing on one or more processors in the near- eye display device system 8. For example, a 3D mapping application may be executing on the one or more computers systems 12, and the user's near-eye display device system 8 communicates image and depth data from image sensors as other sensor data like inertial sensor data which the one or more computer systems 12 use for building a detailed 3D mapping of the user's environment. The NED system 8 can then download the detailed 3D mapping for identifying what a user is looking at and where to display image data representing virtual objects the field of view of the display. The term "display field of view" refers to a field of view of a display of the NED device system. In other words, the display field of view approximates a user field of view as seen from a user perspective also referred to as a user head view direction. [0036] Figure 2A is a side view of an eyeglass temple 102r of a frame in an embodiment of the NED device having an optical see-through display. At the front of frame 115 id depicted one of at least two depth image capture devices 113, e.g. depth cameras, that can capture image data like video and still images, typically in color, of the real world as well as depth sensor data. The depth sensor data may be captured by depth sensitive pixels which correspond to pixels on an image sensor in the depth camera 113. The capture devices 113 are also referred to as outward facing capture devices meaning facing outward from the user's head.

[0037] The illustrated capture device is a front facing capture device which is calibrated with respect to a reference point of its respective display optical system 14. One example of such a reference point is an optical axis (see 142 in Figure 2B) of its respective display optical system 14. The calibration allows the display field of view of the display optical systems 14 to be determined from the depth image data captured by the capture devices 113. The depth data and image data form a depth map of the captured field of view of the depth image capture devices 113 which are calibrated to include the display field of view. A three dimensional (3D) mapping of a display field of view can be generated based on the depth map, and the 3D mapping is used to identify a user head view direction and where to display image data representing virtual objects for the user wearing the NED device 2 to see.

[0038] Control circuitry 136 provides various electronics that support the other components of head mounted display device 2. In this example, the right side arm 102r illustrates exemplary components of control circuitry 136 for the display device 2. The exemplary components include a display driver 246 for an image generation unit 120. For example, the display driver 246 may provide control signals to microdisplay circuitry and drive current to an illumination source for the microdisplay. Other exemplary components includes a processing unit 210 which controls the display driver 246, a memory 244 accessible to the processing unit 210 for storing processor readable instructions and data, a communication module 137 communicatively coupled to the processing unit 210, sensor/speaker interface circuitry 231, and a power supply 239 providing power for the components of the control circuitry 136 and the other components of the display device 2 like the capture devices 113, the microphone 110 and the sensor units discussed below. The interface circuitry 231 performs analog to digital conversion, if necessary, for sensor readings such as from the inertial sensing unit 132 and the location sensing unit 144, and performs digital to analog conversion for audio output to the earphones 130. The interface circuitry 231 may also buffer sensor readings directly into memory 244 or transfer the sensor readings to processing unit 210.

[0039] The processing unit 210 may comprise one or more processors including a central processing unit (CPU) and a graphics processing unit (GPU), particularly in embodiments without a separate companion processing module 4 which contains at least one graphics processing unit (GPU).

[0040] Inside, or mounted to a side arm 102, are an earphone of a set of earphones 130 as an example of an audio output device, an inertial sensing unit 132 including one or more inertial sensors, and a location sensing unit 144 including one or more location or proximity sensors, some examples of which are a GPS transceiver, an infrared (IR) transceiver, or a radio frequency transceiver for processing RFID data. In one embodiment, inertial sensing unit 132 includes a three axis magnetometer, a three axis gyroscope, and a three axis accelerometer as inertial sensors. The inertial sensing unit 132 senses position, orientation, and sudden accelerations of NED device 2. From these sensed movements, head position, (also referred to as head orientation) and thus orientation of the display device, may also be determined which indicate changes in the user perspective, which is the user head view direction and the display field of view for which image data is updated to track with the user perspective.

[0041] In this embodiment, the image generation unit 120 can display a virtual object to appear at a designated depth location in a display field of view to provide a realistic, in- focus three dimensional display of a virtual object which can interact with one or more real objects. In some examples, rapid display of multiple images or a composite image of the in-focus portions of the images of virtual features may be used for causing the displayed virtual data to appear in different focal regions. In other examples, Z buffering may be used.

[0042] In the illustrated embodiment of Figure 2 A, the image generation unit 120 includes a microdisplay and coupling optics, like a lens system. In this example, image data output by the microdisplay is directed to a reflecting surface or element 124. The reflecting surface or element 124 optically couples the light from the image generation unit 120 into a display unit 1 12 (see Figure 2B), which directs the light representing the image into a user's eye when the device 2 is worn by a user.

[0043] Figure 2B is a top view of an embodiment of a display optical system of a NED device. In order to show the components of the display optical system 14, in this case 14r for the right eye, a portion of the frame 115 surrounding the display optical system is not depicted. In this embodiment, the displays 141 and 14r are optical see-through displays, and each display includes a display unit 112 illustrated between two optional see-through lenses 116 and 118 and including a representative reflecting element 134E representing the one or more optical elements like a half mirror, grating, and other optical elements which may be used for directing light from the image generation unit 120 towards a user eye area 140. Arrow 142 represents an optical axis of the display optical system 14r. An example of a display unit 112 for an optical see-through NED includes a light guide optical element. An example of a light guide optical element is a planar waveguide.

[0044] In this embodiment, the NED display 14r is an optical see-through, augmented reality display so that it may allow light from in front of the near-eye display (NED) device 2 to be received by a user eye, thereby allowing the user to have an actual direct view of the space in front of NED device 2 in addition to seeing an image of a virtual object from the image generation unit 120. However, the technology works with video-see displays as well.

[0045] In this embodiment, display unit 112 implements an integrated eye tracking and display system. Again, one or more applications may use eye tracking for the activities they provide for a user, even if the web-like hierarchical menu interface does not, or not as a primary interaction mechanism. For example, an infrared (IR) illumination source may be optically coupled into each display unit 112. The one or more optical elements which direct visible light towards the eye may also direct the IR illumination towards the eye and be bidirectional in the sense of being able to direct IR reflections from the eye to an IR sensor such as an IR camera. A pupil position may be identified for each eye from the respective IR sensor data captured, and based on a model of the eye, e.g. the Gullstrand eye model, and the pupil position, a gaze line for each eye may be determined by software extending from an approximated fovea position. A point of gaze in the display field of view can be identified. An object at the point of gaze may be identified as an object of focus.

[0046] In this embodiment, display unit 112 includes a planar waveguide which acts as part of the display and also integrates eye tracking. The representative reflecting element 134E represents the one or more optical elements like mirrors, gratings, and other optical elements which direct visible light representing an image from the planar waveguide towards the user eye 140. In this embodiment, representative reflecting element 134E also performs bidirectional reflection of infrared light as part of the eye tracking system. Infrared illumination and reflections, also traverse the planar waveguide for an eye tracking system 134 for tracking the position and movement of the user's eye, typically the user's pupil. Eye movements may also include blinks. The eye tracking system 134 comprises an eye tracking IR illumination source 134A (an infrared light emitting diode (LED) or a laser (e.g. VCSEL)) and an eye tracking IR sensor 134B (e.g. IR camera, arrangement of IR photodetectors). Wavelength selective filters 134C and 134D with representative reflecting element 134E implement bidirectional infrared (IR) filtering which directs IR illumination towards the eye 140, preferably centered about the optical axis 142 and receives IR reflections from the user eye 140, preferably including reflections captured about the optical axis 142, which are directed from the waveguide to an IR sensor 134B.

[0047] Again, figures 2 A and 2B show half of the head mounted display device 2. For the illustrated embodiment, a full head mounted display device 2 may include another display optical system 14 as well as another image generation unit 120, another of outward facing capture devices 113, eye tracking system 134, and another of the earphones 130.

[0048] Figure 3 A illustrates an example of a web-like hierarchical display configuration 250 with flexible links connecting levels of menu selections included in an embodiment of web-like hierarchical menu user interface. The menu displays two menu levels below a root 251 which is labeled "Social Networking Main Menu". In this example, display menu elements of flexible links 242, 241, and 243 link children nodes represented by visual elements of hexagons "Social Networking Profile" 252, "Social Networking Contacts" 254 and "Social Networking Posts" 256 respectively to the root 251. At a second menu level, display menu elements of flexible links 245, 247, and 249 link children nodes represented by visual elements of hexagons "Friends" 258, "Associates" 260 and "Work Colleagues" 262 respectively to their parent node "Social Networking Contacts 254". An application provides the content ("Friends", "Associates", Social Networking Profile") for menu items and the tree hierarchy. A display configuration defines how they are represented by display menu elements on a display.

[0049] In this example configuration, the length and shape of the flexible links are adjustable. For example, the arc of the link may be adjusted within a range as well as the lengths. Adjustable links also indicate the positions of the menu elements may be repositioned as per constraints defined for the configuration as discussed below. Menu items may be holographic in their appearance, appearing as 3D objects. As discussed below, menu items may maneuver within a depth range as well even if the display menu element representing the menu item appears two dimensional. [0050] Figure 3 A also illustrates a cursor 300 which may used as a visual aid identifying a user natural user input position within the menu or a navigation path within the menu configuration. For example, the cursor may move relational to a user head movement or gaze movement. To avoid a jumpy cursor, the cursor position changes may be smoothed and associated with a guide like a flexible link to show the user the progress they are making in a selection. In the illustrated example, the near-eye display under control of one or more processors of the NED system 8 may display cursor 300 movement along one of the flexible links 241 which is highlighted for indicating progress in selecting one of the menu items, for example with a user action like a head movement or a gaze action.

[0051] Figure 3B illustrates another version of the example web-like hierarchical menu display configuration of Figure 3A with an example of another visual aid for illustrating progress in making a menu selection. The elliptical shapes 253, 255, and 257 are icons which grow or get bigger in relation to a user action indicating navigation or selection of the item. In this example, as the cursor 300 nears menu item 255, its icon 255 grows proportionally. An icon may change color or reach a maximum size (which may be scaled in relation to menu dimensions) indicating successful selection of a menu item, or be replaced by the cursor 300.

[0052] Based on a 3D mapping of a user environment or at least a field of view of the display, an amount of space for menu display is identified, referred to hereafter as available menu space, as well as a user head view direction or user perspective of the menu. As discussed further below, user head comfort rules or the amount of available menu space may trigger the menu to be reformatted to another display configuration, or menu elements in the current display to be reformatted. Some examples of reformatting or repositioning menu elements include adapting a size of menu items in at least one menu level displayed simultaneously, adapting a number of menu levels displayed simultaneously and changing a depth position of menu items based on their selection status. For example, user head view direction of a menu item may indicate a current user head position does not satisfy user head comfort rules. As permitted by configuration rules, currently unselected menu items are moved in a depth direction away from the NED in a display field of view of the NED and display locations of menu items of a currently user selected menu level are moved to a position in a 3D mapping of the user environment at which the user view direction can be from a center head position. A center head position is looking straight ahead without rotation of the head. Minimizing player rotation through careful UI placement is a benefit that reduces strain on a user's neck during interaction with the menu.

[0053] Figure 4A illustrates another example of a web-like hierarchical menu display configuration with shape flexible links connecting levels of menu selections. This example illustrates a new display configuration 269. The hexagonal menu elements are again used to represent menu items, but due to available menu space, the menu items are closer together in an overall more horizontally compact display configuration. Only one representative flexible link 270 is labeled to avoid overcrowding the drawing.

[0054] As indicated by cursor 300, the currently selected item is Social Networking Contacts 274 in a second menu level and its children nodes Friends, 271, Associates 273 and Work Colleagues 275 are positioned around it. Over time, the selection of menu items in lower levels may cause the user to look down too long which is detected as a violation of user head comfort rules. However, the user may have a preference for leaving traversed menu levels open or a visual cue to them being visible. This configuration may reposition its elements by pushing the parent menu level for the node "Main Menu" 270 to a depth level further into the field of view so its menu elements of hexagonal icons 272, 270 and 276 and its links appear smaller. Even the content displayed for a menu item may be altered in reformatting as here where Social Networking has been removed prior to "Profile" "Main Menu" and "Posts".

[0055] Figure 4B illustrates another example of a web-like hierarchical menu display configuration 280 with zig zag shaped flexible links 291, 292 and 293 connecting menu items Friends 285, Associates 286 and Work Colleagues 287 of a same menu level to their parent Social Networking Contacts 284 currently selected as indicated by the cursor 300. The available menu space may have been identified as having more space in a vertical direction than a horizontal direction with respect to the user head view direction, so weblike hierarchical menu interface display software may select this configuration to fit the space. The menu items may be adapted in the configuration 280 to show one level at a time to avoid exceeding up and down head rotation thresholds.

[0056] Figure 5A is a block diagram of an embodiment of a system from a software perspective including a web-like hierarchical menu interface for operation within a near- eye display device. Figure 5A illustrates an embodiment of a computing environment 54 from a software perspective which may be implemented by a system like NED system 8, one or more remote computer systems 12 in communication with one or more NED systems or a combination of these. Additionally, a NED system can communicate with other NED systems for sharing data and processing resources.

[0057] As noted above, an executing application determines which hierarchical menu content is to be displayed, and in this embodiment, the menu content is adapted to a web- like hierarchical display configuration by a web-like hierarchical menu interface display engine 304. In this embodiment, an application 162 may be executing on one or more processors of the NED system 8 and communicating with an operating system 190 and an image and audio processing engine 191. In the illustrated embodiment, a remote computer system 12 may also be executing a version 162N of the application as well as other NED systems 8 with which it is in communication for enhancing the experience.

[0058] Application data 329 for one or more applications may also be stored in one or more network accessible locations. Some examples of application data 329 may be one or more rule datastores for rules linking action responses to user input data, rules for determining which image data to display responsive to user input data, reference data for natural user input like for one or more gestures associated with the application which may be registered with a gesture recognition engine 193, execution criteria for the one or more gestures, voice user input commands which may be registered with a sound recognition engine 194, physics models for virtual objects associated with the application which may be registered with an optional physics engine (not shown) of the image and audio processing engine 191, and object properties like color, shape, facial features, clothing, etc. of the virtual objects and virtual imagery in a scene.

[0059] As shown in the embodiment of Figure 5A, the software components of a computing environment 54 comprise the image and audio processing engine 191 in communication with an operating system 190. The illustrated embodiment of an image and audio processing engine 191 includes an object recognition engine 192, gesture recognition engine 193, display data engine 195, a web-like hierarchical menu interface display engine 304, a sound recognition engine 194, gaze determination software 196 and a scene mapping engine 306. Additional functionality may be added as indicated by ... The individual engines and data stores provide a supporting platform of data and tasks which an application 162 can leverage for implementing its one or more functions by sending requests identifying data for processing and receiving notification of data updates. The operating system 190 facilitates communication between the various engines and applications. The operating system 190 makes available to applications which objects have been identified by the object recognition engine 192, gestures the gesture recognition engine 193 has identified, which words or sounds the sound recognition engine 194 has identified, and the positions of objects, real and virtual from the scene mapping engine 306.

[0060] The operating system 190 also makes available points of gaze and objects of focus in a display field of view identified by a processor executing the gaze determination software 196 based on eye tracking data like the IR image data received from the eye tracking system 134. Based on the eye tracking data and image and depth data captured by front facing capture devices 113, a point of gaze is determined by one of a variety of processes. In one example of determining gaze, the executing gaze determination software 196 identifies a pupil position within each eye and models a gaze line for each eye extending from an approximated location of a respective fovea. The one or more processors determine a position in the display field of view where the gaze lines meet. Based on a 3D mapping of objects in the display field of view (see discussion of scene mapping engine 306), an object at which the gaze lines meet is an object of focus.

[0061] The computing environment 54 also stores data in image, audio and sensor data buffer(s) 199 which provide memory for image data, and audio data which may be captured or received from various sources as well as memory space for image data to be displayed and audio data to be output. The buffers also provide memory for buffering sensor readings such as from sensors like a 3 axis accelerometer, a 3 axis gyroscope and a 3 axis magnetometer as may be embodied in inertial sensing unit 132. The buffers may exist on both the NED, e.g. as part of the overall memory 244, and may also exist on the companion processing module 4.

[0062] In many applications, virtual data is to be displayed in relation to a real object in the real environment. The object recognition engine 192 of the image and audio processing engine 191 detects and identifies real objects, their orientation, and their position in a display field of view based on captured image data and captured depth data from outward facing image capture devices 113 if available or determined depth positions from stereopsis based on the image data of the real environment captured by the capture devices 113. The object recognition engine 192 distinguishes real objects from each other by marking object boundaries, for example using edge detection, and comparing the object boundaries with structure data 200. Besides identifying the type of object, an orientation of an identified object may be detected based on the comparison with stored structure data 200. Accessible over one or more communication networks 50, structure data 200 may store structural information such as structural patterns for comparison and image data as references for pattern recognition. Reference image data and structural patterns may also be available in user profile data 197 stored locally or accessible in cloud based storage 322.

[0063] The scene mapping engine 306 tracks the three dimensional (3D) position, orientation, and movement of real and virtual objects in a 3D mapping of the display field of view where image data is to be displayed or in a 3D mapping of a volumetric space about the user based on communications with the object recognition engine 192 and one or more executing applications causing image data to be displayed like an application 162 or the web-like hierarchical menu interface display engine 304.

[0064] A depth map representing captured image data and depth data from outward facing capture devices 113 can be used as a 3D mapping of a display field of view of a near-eye display. The depth map has a view dependent coordinate system for the display field of view approximating a user perspective. The captured data may be time tracked based on capture time for tracking motion of real objects. Virtual object positions can be registered in the depth map.

[0065] Mapping what is around the user in the user's environment can be aided with sensor data. Data from an inertial sensing unit 132, e.g. a three axis accelerometer and a three axis magnetometer, determines position changes of the user's head and correlation of those head position changes with changes in the image and depth data from the outward facing capture devices 113 can identify positions of objects relative to one another and at what subset of an environment or location a user is looking. This subset of the environment is what is the scene in the display field of view from which a user perspective or user head view direction can be identified.

[0066] The scene mapping engine 306 can also use a view independent coordinate system for 3D mapping, and a copy of a scene mapping engine 306 may be in communication with other scene mapping engines 306 executing in other systems (e.g. 12, 20 and 8) so the mapping processing can be shared or controlled centrally by one computer system which shares the updated map with the other systems. Overlapping subject matter in the depth images taken from multiple perspectives may be correlated based on a view independent coordinate system and time, and the image content combined for creating the volumetric or 3D mapping (e.g. an x, y, z representation or a 3D mesh surface reconstruction model) of a location or environment (e.g. a room, a store space, or a geofenced area). Thus, changes in light, shadow and object positions can be tracked. The view independent map can be stored in a network accessible location accessible via a remote computer system 12.

[0067] A web-like hierarchical menu interface display software engine 304, based on data and rules, stored in a network (or local) accessible web-like hierarchical menu interface display library 305, determines a web-like hierarchical display configuration for a menu to be displayed for an application, adapts menu items within the configuration based on changes in available menu space about a user's head, and for head comfort in navigating the menu. Some factors affecting the available menu space about a user's head are other objects surrounding a user. For example, a user seated on a train may have the seat in front of him or her limiting his or her view. A user in a living room may have six feet or more of unobstructed distance between her position on the couch and a television. Menu placement criteria or rules may also indicate a preference of not obscuring an object with which a user may be interacting. For example, a user may be moving virtual or real objects in an activity related to the menu, so the menu does not cover the objects or only to an allowed percentage or has its transparency adjusted. In another example, a user's friend walks into the room and overlaps a same space as one of the menu items. In this scenario, the web-like hierarchical menu interface display software engine 304 adjusts the display configuration to avoid overlapping the user's friend, for example by moving the menu item.

[0068] The web-like hierarchical menu interface display engine 304 also identifies user selection of menu items in accordance with user-specific menu selection criteria and provides visual aids to give a user feedback on selection success. The web-like hierarchical menu interface display software engine 304 executing on one or more processors of the NED system 8, the network accessible computer system 12, or both may perform the method embodiments discussed below based on data and rules stored in its accessible library 305.

[0069] An application 162 or the web-like hierarchical menu interface display engine 304 identifies a target 3D space position in the 3D mapping of the display field of view for an object represented by image data and controlled by the application. For example, the web-like hierarchical menu interface display engine 304 identifies changes in the position and object properties of the display menu elements based on the user actions, or other changes triggering menu layout changes. The display data engine 195 performs translation, rotation, and scaling operations for display of the image data at the correct size and perspective. The display data engine 195 relates the target 3D space position in the display field of view to display coordinates of the display unit 112. For example, the display data engine may store image data for each separately addressable display location or area, e.g. a pixel, in a Z-buffer and a separate color buffer. The display driver 246 translates the image data for each display area to digital control data instructions for controlling the image generation unit 120.

[0070] Figures 5B and 5C illustrate some examples of data sets and rules which may be included in an embodiment of a web-like hierarchical menu interface display library 305.

[0071] Figure 5B illustrates an example of a web-like hierarchical display configuration data set illustrating examples of data items which may be used in embodiments of the technology. Web-like hierarchical display configuration rules 422 are also illustrated as part of web-like hierarchical menu interface display library 305 and when executed by a processor enforce constraints defined in a web-like hierarchical display configuration data set 402 store for each display configuration.

[0072] Data stored for the web-like hierarchical display configurations comprise data identifying size, shape and distance ranges for different display menu elements. The data item examples illustrated for the configuration data set 402 are representative and are not meant to be all inclusive as indicated by the The data item examples illustrated are default and minimum spatial dimensions in height, width, and depth per number of levels displayed at one time 404. This may be for example identifying a size range of an overall display configuration for each number of levels displayed. Another example is a range of number of levels which can be displayed simultaneously 406. The next set of examples may be used when adapting display menu elements representing menu items in a menu. Default configuration data as well as minimum constraints are identified: default and minimum 3D positional separation between menu items in height, width and depth (for each menu level) 408, shape definitions of flexible links (for each menu level) 410, shape definitions for menu item icons (for each menu level) 412, default and minimum length of flexible links in each direction (for each menu level) 414, and default and minimum sizes for menu item icons (for each menu level) 416. Additionally, configuration data for aesthetic aspects of display menu elements may be provided such as the examples of color of menu item icons at each level 418 and color of flexible links at each level 420.

[0073] Figure 5C illustrates an example of a user selection metrics data set illustrating examples of data items which may be embodied in the web-like hierarchical menu interface display library 305. User head comfort rules 526 and user specific menu item selection criteria 528 are applied based on data like that illustrated defined in the exemplary user selection metrics data set 500. Again, the data items illustrated are exemplary and not exhaustive, and all the data items identified may not be used by an embodiment of the technology.

[0074] The user selection metrics data set 500 includes an exemplary user physical characteristic data set 502, current user height 520, current user body position 522 and a current user motion state 524. A current user body position 522 may be determined based on data from the inertial sensing unit 132 alone or in combination with identifying user surroundings based on depth image data and 3D mapping of the user environment. Some examples of a body position are sitting, standing, leaning over and lying on one's back or side. There may be different ranges of head motion defined for different body positions. A current user height may be determined based on a user's stored height, e.g. standing height, and current body position. (User stored height may be determined based on the inertial sensing unit 132, alone or in combination with identifying a height of the NED based on its view of a 3D mapped user environment. For example, sitting height is much less than standing height. Current user motion state 524 may be used for interpreting user action when a user is moving toward a virtual menu tethered to a fixed point in a 3D mapping. Additionally, motion state may be used for safety reasons. For example, adjusting a position of the menu or closing the menu based on the user motion state so as not to interfere with a user's view of the real world.

[0075] User head motion data set 505 includes examples of data items tracked specifically for a user over time based on head orientation data generated by for example the accelerometer, gyroscope or both of the inertial sensing unit 132. In many examples, the user head comfort rules 526 apply there rules specifically for the user based on these data values in enforcing comfort criteria. The user head motion data set 505 includes head rotation distance range in up and down direction 504, head rotation distance range in side to side direction 506, speed of head rotation in up and down direction 508 and speed of head rotation in side to side direction 510.

[0076] Other physical characteristic data which may be stored includes optional gaze range right eye data 512, optional gaze range left eye data 514, and user height 518 data which is the user's standing height.

[0077] The user head comfort rules 526 may apply to a number of head based features. The rules 526 may have an ordered list of head positions by comfort, for example predetermined based on human factors or ergonomics data. For example, a center head position indicating the user head is level meaning unrotated and facing straight ahead may be identified as the most comfortable head position, at least for a sitting or standing body position. Additionally, one or more proportions of head rotation range and head rotation speed may be used for one or more thresholds indicating comfort levels. The rotation range and speed would preferably be those determined from measurements for the user. A time factor for holding a particular head position may also be used for determining a user comfort level.

[0078] A head based feature may also include eyes and a mouth for making sound. Comfort rules for gaze may apply to gaze angles used to activate an item and length of gaze. With respect to sound, level of volume, length of sound to be made could be comfort factors, and slowness or quickness of speech required.

[0079] The technology may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of modules, routines, applications, features, attributes, methodologies and other aspects are not mandatory, and the mechanisms that implement the technology or its features may have different names, divisions and/or formats.

[0080] For illustrative purposes, the method embodiments below are described in the context of the system and apparatus embodiments described above. Examples of the one or more processors referenced below are one or more processors of the NED system 8 (with or without the companion processing module 4) or one or more remote computer systems 12 or both. However, the method embodiments are not limited to operating in the system embodiments described above and may be implemented in other system embodiments. Furthermore, the method embodiments may be continuously performed while the NED system is in operation and an applicable application is executing.

[0081] Figure 6 is a flowchart of an embodiment of a method for determining a web-like hierarchical display configuration for displaying a menu by a near-eye display (NED). The embodiment of the method comprises in step 602 identifying by one or more processors data indicating a menu is to be displayed by the near-eye display (NED). As mentioned above, the data may be that the scene mapping engine 306 has identified the menu activation location is located in the current display field of view. In step 604, an available menu space about the user head is determined by the one or more processors based on a stored three dimensional (3D) mapping of an environment of the user. For example, the one or more processors identify available menu space for displaying the menu in the weblike hierarchical display configuration based on at least one of the 3D mapping of the display field of view or a 3D mapping of the user environment. [0082] In step 606, the one or more processors select a position for the menu in a display field of view of the NED based on a user head view direction of a menu activation location. Additionally, the position for the menu in the display field of view may be selected based on at least one of a user height or a current user height so the user sees the menu at a comfortable eye level.

[0083] In step 608, a web-like hierarchical menu display configuration is selected based on the available menu space, the user head view direction and user head comfort rules, and in step 610, menu items are adapted to the selected web-like hierarchical display configuration based on the available menu space, the user head view direction and user head comfort rules. As mentioned in the discussion for Figures 4A and 4B, some examples of adapting may include adapting a size of menu items in at least one menu level displayed simultaneously and adapting a number of menu levels displayed simultaneously based on the available menu space. In step 612, the one or more processors cause the image generation unit 120 to display the menu with its menu items adapted in accordance with the selected web-like hierarchical display configuration.

[0084] As discussed further below, the head view direction and available menu space are continuously monitored during display of the menu for determining whether to change the menu display layout. For example, monitoring of the head view direction and head position may indicate a user head position is rotating further away from a center head position. To avoid discomfort for the user, the displayed distance between menu levels may be decreased.

[0085] Figure 7 is a flowchart of an embodiment of a method for adapting activation of a web-like hierarchical menu interface being executed by one or more processors of a near-eye display (NED) system based on user selection metrics. The embodiment of the method comprises in step 616 adjusting by one or more processors activation parameters in menu item selection criteria for the web-like hierarchical menu interface based on stored user head motion data. The one or more processors in step 618 monitor head view direction and determine whether a change in the head view direction activates a change in a menu display layout. In step 620, the one or more processors determine whether a change in the available menu space activates a change in the menu display layout. In step 622, user selection of a menu item is identified, for example by identifying natural user input based on data from sensors on the NED system like an eye tracking system 134, the inertial sensing unit 132 and the depth cameras 113. The NED system displays feedback confirming user selection of the menu item in step 624. [0086] Figure 8 is a flowchart of an embodiment of a process for adapting activation parameters for selecting a menu selection from a web-like hierarchical menu based on user head motion data. In step 626, the one or more processors track user head motion data over time from orientation sensor data during use of a NED system and store the data values. In 628, a weighting scheme is applied to the tracked data values giving preference to later tracked data values.

[0087] In steps 630, 634, 636 and 638, each of the weighted head rotation distance range values for the up and down direction, average weighted head rotation distance range values in a side to side direction, average weighted speed of head rotation for the up and down direction, and average weighted speed of head rotation in the side to side direction are averaged and stored in a user head motion data set.

[0088] In step 640, head movement speed and range activation parameters are adjusted in menu item selection criteria based on the weighted averages tracked over time for the user head motion data set to generate user-specific menu item selection criteria.

[0089] Figure 9 is a flowchart of an embodiment of a method for monitoring head view direction and determining whether a change in the head view direction activates a change in a menu layout. The embodiment of the method comprises in step 642, checking view direction of user with respect to menu, for example checking menu position in the NED field of view based on the depth images captured by the front depth cameras 113. In step 644, it is determined whether the view direction is within a range for menu selection. The user's view direction is likely to move around a bit just due to normal user head motion. Additionally, some confirmation of selection actions may cause the user to change his view angle or direction to the menu. In some examples, range of menu selection criteria may be based on a percentage of the displayed menu being in the display field of view within a time period. This can be an indicator that the user is still engaging with the menu.

[0090] Responsive to the view direction not being within the range for menu selection, in step 648, the menu is closed. This is to prevent unintended menu selections. Optionally, the current menu selection state can be saved in step 646 in case the user comes back within a predetermined time. Then, the user can traverse the hierarchical menu where she left off. In step 650, responsive to the view direction being within the range for menu selection, determining whether the view direction is within a menu boundary range. A menu boundary may be an edge of a menu display configuration. Responsive to the view direction being within the menu boundary, in step 652, adjusting the visual appearance of a cursor to indicate user head view direction is one of moving away or towards the menu. For example, as a user view direction gets too near an edge, the cursor fades, and as he or she returns user head view direction towards the menu, the cursor brightens.

[0091] Sensor data from the inertial sensing unit 132 provides orientation data and acceleration data from which head position data and head movement data can be determined. Additionally, the one or more processors in step 654 determine whether user head position data for the current head view direction satisfies user head comfort rules, and responsive to the user head comfort rules not being satisfied, in step 656 repositions the menu items for satisfying the user head comfort rules. If the rules are satisfied, the one or more processors in step 658 return to other processing until the next head view direction check.

[0092] The repositioning of menu items in step 656 may comprise altering a current web-like display configuration or selecting a new web-like display configuration as in the examples of Figures 4 A and 4B. For example in Figure 4 A, if a current user head position does not satisfy user head comfort rules, currently unselected menu items may be moved in a depth direction away from the NED in a display field of view of the NED and display locations of menu items of a currently user selected menu level moved to a position in a 3D mapping of the user environment at which the user view direction can be from a center head position.

[0093] Figure 10 is a flowchart of an embodiment of method for identifying user selection of a menu item. The embodiment of the method comprises in step 662 identifying one or more menu items in the head view direction based on a 3D mapping of the display field of view of the NED and in step 664, detecting one or more user actions of a head based feature while the head view direction stays within a menu selection range. An example of head based features are the head or the eyes.

[0094] In step 667, optionally, one or more visual aids can be displayed relating the one or more user actions to a user navigation path within the menu. In step 668, the one or more processors determine whether the one or more user actions satisfy user specific menu item selection criteria for a menu item. In step 668, the one or more processors determine whether the one or more user actions satisfy user specific menu item selection criteria.

[0095] The following example illustrates detecting one or more user actions of a head based feature while the head view direction stays within a menu selection range and determining whether the one or more user actions satisfy user specific menu item selection criteria for natural user input in terms of head view direction and head position. [0096] The one or more processors track changes to the user head view direction based on updates to the 3D mapping of the display field of view and correlating tracked user head view directions to the currently displayed menu items by identifying positions of the menu items in the NED field of view. The one or more processors determine whether a user head view direction has been maintained with respect to one of the currently displayed menu items satisfying a selection candidate time period stored in user-specific menu item selection criteria. Responsive to the selection candidate time period being satisfied, the one or more processors display a visual aid to the user indicating that the one of the currently displayed menu items is a selection candidate. Based on head orientation data determined from sensor data from an inertial sensing unit of the NED system, the one or more processors determine whether a selection confirmation head movement indicated by a head position change has occurred within a confirmation time period which may coincide with the selection candidate time period stored in user-specific menu item selection criteria or start after completion of the selection candidate time period.

[0097] In step 670, responsive to the one or more user actions satisfying user specific menu item selection criteria, the NED system outputs feedback by the near-eye display system confirming selection of the menu item. The feedback may be audio through audio output devices 130, visual data displayed by the NED or audiovisual data. In step 672, the one or more processors cause the NED system to perform a pre-defined action associated with user selection of the menu item based on rules for the menu. The pre-defined action and rules for the menu may be defined by the application providing the menu content.

[0098] If step 674, responsive to the one or more user actions not satisfying user specific menu item selection criteria, feedback may be output to the user indicating a menu item selection could not be resolved, and optionally in step 676, another head based feature movement for selecting a menu item such as gaze may be suggested to the user. Another option, if the available menu space permits is to expand the displayed configuration by increasing size of menu items or separation of menu items, either by distance or angle or choose another configuration which has increased separation between menu items or bigger items.

[0099] Figure 11 is a flowchart of an embodiment of a method for determining whether a change in the available menu space activates a change in the menu display layout. The embodiment of the method comprises in step 682, determining whether available menu space has changed for the user. Responsive to the available menu space not having changed for the user, in step 684, the one or more processors return to other processing until a next available menu space check. Responsive to the available menu space having changed for the user, in step 686 determining whether the minimum display configuration spatial dimensions are met for a current web-like hierarchical display configuration.

[00100] In step 688, responsive to the minimum display configuration spatial dimensions not being met for the current web-like hierarchical display configuration, the one or more processors determine another web-like hierarchical display configuration for the menu and update the menu with the other web-like hierarchical display configuration in step 690.

[00101] Responsive to the minimum display configuration spatial dimensions being met, in step 692, the one or more processors determine whether the current web-like hierarchical display configuration spatial dimensions are being met. If so, the one or more processors return to other processing until a next available menu space check in step 696. Responsive to the current web-like hierarchical display configuration spatial dimensions not being met, in step 694, the one or more processors modify a layout of menu items for the current display configuration based on the web-like hierarchical display configuration rules 422 for the current display configuration.

[00102] Figure 12 is a block diagram of one embodiment of a computer system that can be used to implement a network accessible computing system 12, a companion processing module 4, or another embodiment of control circuitry 136 of a near-eye display (NED) device which may host at least some of the software components of computing environment 54. The complexity and number of components may vary considerably for different embodiments of the computer system 12, the control circuitry 136 and the companion processing module 4. Figure 12 illustrates an exemplary computer system 900. In its most basic configuration, computing system 900 typically includes one or more processing units 902 including one or more central processing units (CPU) and one or more graphics processing units (GPU). Computer system 900 also includes memory 904. Depending on the exact configuration and type of computer system, memory 904 may include volatile memory 905 (such as RAM), non-volatile memory 907 (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in Figure 12 by dashed line 906. Additionally, computer system 900 may also have additional features/functionality. For example, computer system 900 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in Figure 12 by removable storage 908 and non-removable storage 910. [00103] Computer system 900 may also contain communication module(s) 912 including one or more network interfaces and transceivers that allow the device to communicate with other computer systems. Computer system 900 may also have input device(s) 914 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 916 such as a display, speakers, printer, etc. may also be included.

[00104] The example computer systems illustrated in the figures include examples of computer readable storage devices. A computer readable storage device is also a processor readable storage device. Such devices may include volatile and nonvolatile, removable and non-removable memory devices implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Some examples of processor or computer readable storage devices are RAM, ROM, EEPROM, cache, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, memory sticks or cards, magnetic cassettes, magnetic tape, a media drive, a hard disk, magnetic disk storage or other magnetic storage devices, or any other device which can be used to store the information and which can be accessed by a computer.

[00105] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.