USER

BACKGROUND

The present invention relates to extended reality technology, and more particularly to technology that presents multi-layer extended reality images to a user.

It is expected that extended reality (XR) systems will soon become a common way for people to receive and interact with all sorts of digital information. Extended reality (XR) systems (such as e.g., augmented reality (AR) and mixed reality (MR) systems) can be used to bridge the gap between physical and digital worlds, augmenting and projecting digital information into the physical world.

A head mounted display (HMD) is one possible device implementation in an XR system. Conventional XR glasses rely on computing power and light sources (e.g., LED, Laser, or other projecting technologies) that are integrated into the XR glasses, and this makes them heavy and bulky. There is therefore a need to find solutions that allow XR glasses to be smaller, less visible, and smarter, and that provide for an improved user experience.

SUMMARY

It should be emphasized that the terms “comprises” and “comprising”, when used in this specification, are taken to specify the presence of stated features, integers, steps or components; but the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

Moreover, reference letters may be provided in some instances (e.g., in the claims and summary) to facilitate identification of various steps and/or elements. However, the use of reference letters is not intended to impute or suggest that the so-referenced steps and/or elements are to be performed or operated in any particular order. In accordance with one aspect of the present invention, the foregoing and other objects are achieved in technology (e.g., methods, apparatuses, nontransitory computer readable storage media, program means) that supplies multi-plane images for viewing by a user. The technology includes an image generator, an image director, and a first output port. The image generator generates a first image to be seen by the user as being a first distance from a user point of view, and a second image to be seen by the user as being a second distance from the user point of view. The first image is comprised of a number of optical wavelength components, and the second image is comprised of the number of optical wavelength components. The image director directs the first image to traverse a first optical path to the first output port of the apparatus, and directs the second image to traverse a second optical path to the first output port of the apparatus, wherein the first optical path corresponds to the first distance and the second optical path corresponds to the second distance, and wherein the first optical path and the second optical path have different lengths. The first output port connects to a first optical waveguide that is configured to guide the number of optical wavelength components to a user display device.

In another aspect of some embodiments, the technology further includes the first optical waveguide coupled at a first end to the first output port; and a see- through optical waveguide coupled to a second end of the first optical waveguide.

In another aspect of some embodiments, the image director comprises a first electro-mechanical mirror array located in the path of the first and second images supplied by the image generator; and a controller. The controller causes the first electro-mechanical mirror array to direct the first image to the first optical path, and causes the first electro-mechanical mirror array to direct the second image to the second optical path.

In another aspect of some embodiments, the image generator time multiplexes generation of the first and second images. When the image generator is generating the first image, the controller causes the first electro-mechanical mirror array to direct the first image to the first optical path; and when the image generator is generating the second image, the controller causes the first electro mechanical mirror array to direct the second image to the second optical path.

In another aspect of some embodiments, the image generator simultaneously generates the first image and the second image as component parts of an overall image, wherein the first image is located within a first part of the overall image and the second image is located within a second part of the overall image; the first electro-mechanical mirror array has a first area part and a second area part; the first electro-mechanical mirror array is disposed within the apparatus such that the first area part of the first electro-mechanical mirror array is in the path of the first image output by the image generator and the second area part of the first electro-mechanical mirror array is in the path of the second image output by the image generator. The controller causes the first area part of the first electro-mechanical mirror array to direct the first image to the first optical path; and causes the second area part of the first electro-mechanical mirror array to direct the second image to the second optical path.

In another aspect of some embodiments, the first and second images are configured for viewing by a first eye of the user; the image generator generates a third image to be seen by the user as being the first distance from the user point of view, and a fourth image to be seen by the user as being the second distance from the user point of view, wherein the third image is comprised of the number of optical wavelength components, and the second image is comprised of the number of optical wavelength components, and wherein the third and fourth images are configured for viewing by a second eye of the user. The image generator simultaneously generates the first image, the second image, the third image, and the fourth image as component parts of the overall image, wherein the third image is located within a third part of the overall image and the fourth image is located within a fourth part of the overall image. Also in such embodiments, the first electro-mechanical mirror array has a third area part and a fourth area part; the first electro-mechanical mirror array is disposed within the apparatus such that the third area part of the first electro-mechanical mirror array is in the path of the third image output by the image generator, and the fourth area part of the first electro-mechanical mirror array is in the path of the fourth image output by the image generator. In such embodiments, the controller causes the third area part of the first electro-mechanical mirror array to direct the third image to a third optical path that corresponds to the first distance; and causes the fourth area part of the first electro-mechanical mirror array to direct the fourth image to a fourth optical path that corresponds to the second distance, wherein: the first electro-mechanical mirror array is configured to direct the first image to the first output port via the first optical path, and to direct the third image to a second output port via the third optical path, wherein the second output port is configured to connect to a second optical waveguide that is configured to guide the number of optical wavelength components to the user display device; and the technology further comprises a second electro-mechanical mirror array configured to direct the second image to the first output port via the second optical path, and to direct the fourth image to the second output port via the fourth optical path.

In another aspect of some embodiments, the first and second images are configured for viewing by a first eye of the user; the image generator generates a third image to be seen by the user as being the first distance from the user point of view, and a fourth image to be seen by the user as being the second distance from the user point of view, wherein the third image is comprised of the number of optical wavelength components, and the fourth image is comprised of the number of optical wavelength components, and wherein the third and fourth images are configured for viewing by a second eye of the user; the image director receives the first image and the second image from the image generator during a first period of time; the image director receives the third image and the fourth image from the image generator during a second period of time; the image director is configured to direct the first image to the first output port via the first optical path during the first period of time; the image director is configured to direct the second image to the first output port via the second optical path during the first period of time; the image director is configured to direct the third image to a second output port via a third optical path during the second period of time, wherein the third optical path corresponds to the first distance; and the image director is configured to direct the fourth image to the second output port via a fourth optical path during the second period of time, wherein the fourth optical path corresponds to the second distance.

In another aspect of some embodiments, the first period of time comprises a third period of time and a fourth period of time; the second period of time comprises a fifth period of time and a sixth period of time. When it is the third period of time, the image director causes the first image to be directed to the first output port via the first optical path; when it is the fourth period of time, the image director causes the second image to be directed to the first output port via the second optical path; when it is the fifth period of time, the image director causes the third image to be directed to the second output port via the third optical path; and when it is the sixth period of time, the image director causes the fourth image to be directed to the second output port via the fourth optical path.

In another aspect of some embodiments, the image director comprises a second electro-mechanical mirror array, disposed in the second optical path and the fourth optical path, and configured to direct the second image to the first output port during the first period of time; and to direct the fourth image to the second output port during the second period of time.

In another aspect of some embodiments, the first and second images are configured for viewing by a first eye of the user; the image generator generates a third image to be seen by the user as being the first distance from the user point of view, and a fourth image to be seen by the user as being the second distance from the user point of view, wherein the third image is comprised of the number of optical wavelength components, and the fourth image is comprised of the number of optical wavelength components, and wherein the third and fourth images are configured for viewing by a second eye of the user; the image director receives the first image and the third image from the image generator during a first period of time; the image director receives the second image and the fourth image from the image generator during a second period of time; the image director is configured to direct the first image to the first output port via the first optical path during the first period of time; the image director is configured to direct the third image to a second output port via a third optical path during the first period of time, wherein the third optical path corresponds to the first distance; the image director is configured to direct the second image to the first output port via the second optical path during the second period of time, and to direct the fourth image to the second output port via a fourth optical path during the second period of time, wherein the fourth optical path corresponds to the second distance.

In another aspect of some embodiments, the first period of time comprises a third period of time and a fourth period of time and the second period of time comprises a fifth period of time and a sixth period of time. When it is the third period of time, the image director causes the first image to be directed to the first output port via the first optical path; when it is the fourth period of time, the image director causes the third image to be directed to the second output port via the third optical path; when it is the fifth period of time, the image director causes the second image to be directed to the first output port via the second optical path; and when it is the sixth period of time, the image director causes the fourth image to be directed to the second output port via the fourth optical path.

In another aspect of some embodiments, the image director comprises a second electro-mechanical mirror array, disposed in the second optical path and the fourth optical path. The image director is configured to direct the second image to the first output port during the first period of time; and to direct the fourth image to the second output port during the second period of time.

In another aspect of some embodiments, the technology further includes an autofocus lens arrangement that is dynamically controlled by the controller to cause the first distance to have a first optical length when a first control signal is generated, and to have a second optical length when a second control signal is generated, wherein the first and second optical lengths are different. BRIEF DESCRIPTION OF THE DRAWINGS

The objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the drawings in which:

Figure 1 depicts a conventional headset display.

Figure 2 depicts an arrangement in which the image generator is not integrated with the headset and is therefore separated from the see-through optical waveguides.

Figures 3, 4, and 5 depict arrangements of a remote device having an image generator that supplies images to see-through waveguides by means of suitable optical waveguides, and in which time multiplexing is used to display images situated on multiple virtual planes.

Figures 6, 7, and 8 illustrate aspects of pixel resolution division in accordance with some aspects of the technology.

Figures 9 and 10 illustrate aspects of the disclosed technology in which pixel resolution division is used to display images situated on multiple virtual planes.

Figures 11 and 12 illustrate further aspects of pixel resolution division in accordance with some aspects of the technology relating to stereoscopic images having multiple virtual planes.

Figures 13, 14, and 15 illustrate exemplary embodiments in which pixel resolution division is used in conjunction with time division multiplexing to display stereoscopic images situated on multiple virtual planes.

Figures 16 and 17 illustrate alternative exemplary embodiments in which pixel resolution division is used in conjunction with time division multiplexing to display stereoscopic images situated on multiple virtual planes.

Figure 18 illustrates another alternative exemplary embodiment in which pixel resolution division is used to display stereoscopic images situated on multiple virtual planes.

Figures 19A, 19B, and 19C illustrate an exemplary arrangement for dynamically altering the location of a virtual plane in accordance with some embodiments consistent with the invention. Figure 20 is, in some respects, a flowchart of actions performed in some exemplary embodiments that utilize time multiplexing to display multiple virtual plane images.

Figure 21 is, in some respects, a flowchart of actions performed in some exemplary embodiments that utilize pixel resolution division to display multiple virtual plane images.

Figure 22 is a block diagram of an arrangement in accordance with some but not necessarily all embodiments consistent with the invention.

Figure 23 illustrates an exemplary controller of a remote device in accordance with some but not necessarily all exemplary embodiments consistent with the invention.

DETAILED DESCRIPTION

The various features of the invention will now be described with reference to the figures, in which like parts are identified with the same reference characters.

The various aspects of the invention will now be described in greater detail in connection with a number of exemplary embodiments. To facilitate an understanding of the invention, many aspects of the invention are described in terms of sequences of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be recognized that in each of the embodiments, the various actions could be performed by specialized circuits (e.g., analog and/or discrete logic gates interconnected to perform a specialized function), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term “circuitry configured to” perform one or more described actions is used herein to refer to any such embodiment (i.e., one or more specialized circuits alone, one or more programmed processors, or any combination of these). Moreover, the invention can additionally be considered to be embodied entirely within any form of nontransitory computer readable carrier, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Thus, the various aspects of the invention may be embodied in many different forms, and all such forms are contemplated to be within the scope of the invention. For each of the various aspects of the invention, any such form of embodiments as described above may be referred to herein as “logic configured to” perform a described action, or alternatively as “logic that” performs a described action.

An aspect of the herein-described technology involves moving all scene rendering computing power and related hardware (such as LED light sources, LCOS, and lens packages) from the XR glasses to a separate XR computing device that, for example, could be placed in a pouch or worn on a belt. This separation can be achieved by inserting one or more fiber optics cables between the lens packages and the waveguides in the XR glasses.

To illustrate this aspect, reference is made to Figure 1, which depicts a conventional headset display 100. The headset display 100 can be, for example, implemented as a pair of glasses or a helmet.

It is desired to display a first virtual image 105 and a second virtual image 107 in a manner such that the first image 105 is seen by the user as being a first distance dl from a user point of view 109, and the second image 107 is seen by the user as being a second distance d2 from the user point of view 109. The first distance dl and the second distance d2 may also be referred to herein as first and second virtual planes (VP). The two virtual planes are displaced from one another by a distance d3.

The first image 105 is comprised of a number (n) of optical wavelength components (e.g., in some but not necessarily all embodiments, three optical wavelengths corresponding to Red, Green, and Blue - RGB), and the second image 107 is comprised of the same number (n) of optical wavelength components. To achieve this, an image generator 101 in the headset 100 projects all of the optical wavelength components onto a corresponding set of see-through waveguides 103. In this example, since there are two virtual planes, and three optical wavelength components per plane, the image generator 101 projects a total of six optical wavelength components (Rdl, Gdl, Bdl, Rd2, Gd2, Bd2) onto the see-through waveguides 103, and this is done such that each of the optical wavelength components will be guided through a respective one of six see- through waveguides 103, three see-through waveguides for each virtual plane as shown in the figure.

The see-through waveguides 103 can be implemented as, for example, holographic waveguides, which are a type of display technology very often used in optical see-through AR to enable a small form factor design.

Figure 2 shows an alternative arrangement still involving an image generator 201 and see-through waveguides 203, but in which the image generator 201 is no longer integrated with the headset and is therefore separated from the see-through optical waveguides 203. Because of this separation, the image generator 201 can no longer project the images directly onto the see-through waveguides 203. To solve this problem a bundle of optical fibers 205 (which are also a type of optical waveguide) is introduced.

Optical fiber provides a direct one to one image transfer mechanism. Typically there is a one-to-one correspondence between optical fibers and pixels in a complete image. The image can be enlarged with an optical lens system at some point before it enters the holographic waveguide if the image needs to be larger. However, enlarging an image in this way has the drawback of some pixilation (i.e., a kind of degradation in image quality due to an insufficient amount of pixel information for the enlarged image).

Optical fiber can be made very thin. There are optical fibers that have been made as small as 0.8um. To be able to transfer a Full High Definition (FHD) image with, for example, lum thick optical fiber per pixel, the dimension of the fiber cable would be around ~6mm x ~3.3mm ((1920 pixel resolution x 3 sub pixels per pixel x lum) x (1080 pixel resolution x 3 sub pixels per pixel x lum)), in other words, an area of ~19mm ² . In an XR device with one display for each eye, two optical fiber cables with these dimensions would be needed.

It has been shown that it is possible to propagate multiple wavelengths of light through a single optical fiber. Therefore in embodiments utilizing such technology, the number of optical fibers used may be fewer than the number of optical wavelength components associated with a virtual plane since a single fiber can carry several wavelength components. But still, conventional technology requires that the number of optical fibers be replicated for each different virtual plane.

As shown in the figure, each optical fiber 205 receives a respective one of the optical wavelength components projected by the image generator 201, and guides this to a respective one of the see-through waveguides 203. It should be noted that the depiction of a one-to-one correspondence between fibers and wavelength components is made to ease the discussion. In practical embodiments, a same fiber can be used to carry some or all wavelength components associated with an image pixel (e.g., all three of the red, green, and blue wavelength components), with the projectors in such systems operating in a field sequential fashion such that, for example, a red part of the image is projected first, followed by a green part and then a blue part, and so on.

In the arrangement of Figure 2, the display system of the image generator 201 can be the same set of hardware parts as in a waveguide AR headset. So, for example, it can comprise a Liquid Crystal On Silicon (LCOS) display with a light source or a micro Organic Light Emitting Diode (OLED) that will supply the image to the holographic waveguides. In addition to the image producing part, the optics to produce the distance to the virtual plane are also positioned here (but not shown in order to avoid cluttering the figure). A benefit of adding the optics in this part of the system (i.e., separate from the display device that the user looks through) is that one is no longer as constrained in space as when the optics are placed in, for example, the temples of the AR glasses. This means that the optical components can be made comparatively larger, which enables them to be able to manipulate light in a better/simpler way. Optical properties that can easily be changed include virtual planes, image size, and image positions.

In particular, and as one example, having more room within which to install optical and other components means that the size and resolution of images can be increased compared to headset-bound technology. In one improvement, having a larger image size to work with enables different display surfaces to be used for different purposes. For example, in one aspect of the technology, a central part of the perceived image can be used for displaying images and a peripheral part (e.g., a bottom part) can be used to display a status field.

By separating the host system (including image generator 201) from the surface that displays the image (e.g., see-through waveguides 203), the processing of all images that are to be shown on the see-through waveguides 203 can also be moved out of the head-mounted device. As a result, the head-mounted device does not need a battery or any other electrically driven parts to show an image to the user. This makes it possible to manufacture the head-mounted device in a very slim fashion compared to older technology.

The arrangements described so far can be further improved, and these improvements are now described in greater detail.

Referring now to Figure 3, a remote device 300 having an image generator 301 supplies images to see-through waveguides 303 by means of suitable optical waveguides such as the illustrated bundle of optical fibers 305. Each image pixel comprises a number of optical wavelength components. For example, and without limitation each image pixel can comprise three optical wavelength components corresponding to read, green, and blue pixel information.

In an aspect of some but not necessarily all embodiments consistent with the invention, additional components are present in the remote device 300 that enable it to alter the optical path in a dynamic way. By doing this, the virtual distance of objects can be switched by dynamically changing the optical path inside the compute device. One benefit of doing this is that it eliminates the need to have a dedicated set of waveguide layers for each virtual plane to be depicted. For example, where Figure 2 shows an embodiment requiring six optical fibers 205 and a corresponding set of six see-through waveguides 203 (i.e., three waveguide layers for the virtual image at distance dl, and three additional waveguide layers for the virtual image at distance d2), embodiments consistent with the invention require only enough waveguides to convey and depict the number of optical wavelength components associated with one image pixel, and these same waveguides are used by all of the virtual planes. A benefit of reducing the number of waveguide layers is that it enables the XR device to be made more cheaply and lighter with only one optical interface to the waveguide.

Accordingly, the remote device 300 further includes an image director 307 configured to direct different images to traverse respectively different optical paths within the remote device 300 before arriving at a first output port 309 of the remote device 300. For example, if a first image is to be made to appear at a first distance dl from a user point of view and a second image is to be made to appear at a second distance d2 from the user point of view, then the image director 307 directs the first image to traverse a first optical path to the first output port 309 of the remote device 300, and directs the second image to traverse a second optical path to the first output port 309 of the remote device, wherein the first optical path corresponds to the first distance dl and the second optical path corresponds to the second distance d2, wherein the first optical path and the second optical path have different lengths corresponding to the difference d3 between the distances d2 and dl.

This dynamic changing of optical paths inside the remote device 300 will be described further with reference to Figure 3, Figure 4, and Figure 5. In this example, first and second images are to be depicted at respective first and second virtual planes. However, the depiction of only two virtual planes is for purposes of illustration only, and should not be construed as a limitation. To the contrary the principles upon which this technology is based are not limited to only two virtual planes, and more than two optical paths can be constructed in the remote device 300 which will enable more than two virtual planes.

An aspect of some embodiments consistent with the invention is reliance on time multiplexing, as shown in Figure 5. For the sake of simplicity, this example deals only with monoscopic renderings. Expansion of the principles to cover stereoscopic renderings of images is discussed later in this document.

As shown in the figure, the image director 307 receives (step 501), during a first period of time, a first image Mb to be perceived as being at a first distance dl from a user point of view. The image director 307 therefore directs (step 503) the first image Mh to a first optical image path corresponding to the distance dl. The first image Mh emerges at the end of the first optical image path and is supplied to the first output port 309 of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user (step 505).

Next, after a time At, the image director 307 receives (step 507), during a second period of time, a second image M2i to be perceived as being at a second distance d2 from the user point of view. The image director 307 therefore directs (step 509) the second image M2i to a second optical image path corresponding to the distance d2. The second image M2i emerges at the end of the second optical image path and is supplied to the first output port 309 of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user (step 511).

It will be understood that the display of the first image Ml i during the first period of time followed by the display of the second image M2i during the second period of time occur rapidly enough that the user does not perceive the switching, but instead perceives the two images as being displayed at the same time. This corresponds to one frame of a multi-plane image.

This process is repeated for subsequent frames as well. To illustrate this point, Figure 5 also shows that the image director 307 receives (step 513), during a next first period of time, a next first image Mh to be perceived as being at the first distance dl from the user point of view. The image director 307 therefore directs (step 515) the next first image Mhto the first optical image path corresponding to the distance dl. The next first image Mh emerges at the end of the first optical image path and is supplied to the first output port 309 of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user (step 517).

Next, after another time At, the image director 307 receives (step 519), during a next second period of time, a next second image M2 ₂ to be perceived as being at the second distance d2 from the user point of view. The image director 307 therefore directs (step 521) the next second image M2 ₂ to the second optical image path corresponding to the distance d2. The next second image M2 ₂ emerges at the end of the second optical image path and is supplied to the first output port 309 of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user (step 523).

In some but not necessarily all embodiments consistent with the invention, the image director 307 uses a controller to dynamically alter the positions of one or more electro-mechanical mirror arrays such that an image projected by an image generator will be caused to traverse one of a number of possible optical paths within the remote device 300. The electro-mechanical mirror arrays can for example be micro electro-mechanical mirrors (MEMS) as are known in the art.

To illustrate this point further, reference again is made to Figure 3 which shows an image director 307 comprising a first mirror 311 and a second mirror 313, which are electro-mechanical mirrors as just mentioned. Figure 3 depicts the arrangement at a first period of time tl. The controller 315 causes the first and second mirrors 311, 313 to position themselves such that an image generated by the image generator 301 will flow through a lens package 317 (for image focusing) directly to the first output port 309. The distance from the image generator 301 to the first output port 309 corresponds to the first optical path dl.

Figure 4 depicts the state of the remote device 300 at a later period of time, t2. Note: to avoid cluttering the figure, the depiction of the box representing the image director 307 has been omitted. However, it will be understood that in this and subsequent similar figures, the depicted controllers, mirrors, and lens packages are constituent parts of a corresponding image director.

Accordingly, the controller 315 causes the first and second mirrors 311,

313 to reposition themselves such that an image generated by the image generator 301 will flow through a second lens package 319 (for image focusing) to a third mirror 321 which deflects the image to a fourth mirror 323 which deflects the image to the second mirror 313 which has been repositioned to deflect the image to the first output port 309. The distance from the image generator 301 to the first mirror 311, then through the second lens package 319 to the third mirror 321, the fourth mirror 323, the second mirror 313 and then to the first output port 309 corresponds to the second optical path d2.

It will be appreciated from these figures and the discussion that the same number of optical fibers (and in fact the same actual optical fibers 305) is used to convey images for each of the two virtual planes VP1, VP2.

A class of alternative embodiments in which multi-plane images are generated without the use of time-multiplexing will now be described with reference to Figures 6 through 16. These embodiments rely in part on the use of pixel resolution division. Referring first to Figure 6, a pixel array 600 formatted as an NxN arrangement of pixels is available for representing one or more images. It is noted that this arrangement is for purposes of example only, and that in other embodiments the array need not take on the shape of a square.

Two alternatives will be described, one for monoscopic image generation, and the other for stereoscopic image generation. Starting first with the monoscopic example, as shown in Figure 7, the pixel array 600 is conceptually divided up into two partitions, herein denoted partitions A and B respectively. Where the pixel array 600 is considered to represent an overall image, each of the partitions A and B will be considered to represent a respective one of two component parts of the overall image. Next, as shown in Figure 8, different images are assigned to each of the component parts of the overall image (i.e., partitions of the overall image) in a manner that enables one of a user’s eyes to receive sets of plural images intended for display on different virtual planes. For example, Figure 8 depicts a non-limiting arrangement in which:

• an image intended to be seen by one of the user’s eyes at a distance of d2 from a user point of view will be projected within a first part (partition A) of the overall image

• an image intended to be seen by the same one of the user’s eyes at a distance of dl from the user point of view will be projected within a second part (partition B) of the overall image It is noted that for purposes of illustration, Figure 8 depicts a horizontal division of images. In alternative embodiments, other types of division (e.g., vertical division) can be used, and in such cases mirror control (to be described in the following) is adapted accordingly.

Referring now to Figure 9, in order to display these images to the user, the image director 307 receives (step 901), during a first period of time, a first image Mh to be perceived as being at a first distance dl from a user point of view and also during the first period of time, a second image M2i (step 903) to be perceived as being at a second distance d2 from a user point of view. These two images are configured as component parts of an overall image as previously shown in Figure 8. Accordingly, the image director 307 directs (step 905) the first image Ml i to a first optical image path corresponding to the distance dl and simultaneously directs (step 907) the second image M2i to a second optical image path corresponding to the distance d2. The first image Mh emerges at the end of the first optical image path and is supplied (step 909) to a first output port of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user. Concurrently, the second image M2i emerges at the end of the second optical image path and is also supplied (step 909) to the first output port of the remote device 300. The two images are then guided to the see- through waveguides 303 for viewing by the user.

This process is repeated for subsequent frames as well. To illustrate this point, Figure 9 also shows that the image director 307 receives (step 911), during a second period of time, another first image Mh to be perceived as being at the first distance dl from the user point of view and also during the second period of time, a second image M22 (step 913) to be perceived as being at the second distance d2 from the user point of view. These two images are configured as component parts of an overall image as previously shown in Figure 8. Accordingly, the image director 307 directs (step 915) the “another” first image Mhto the first optical image path corresponding to the distance dl and simultaneously directs (step 917) the “another” second image M22 to the second optical image path corresponding to the distance d2. The “another” first image Mh emerges at the end of the first optical image path and is supplied (step 919) to the first output port of the remote device 300, from which it is then conveyed to the see-through waveguides 303 for viewing by the user. Concurrently, the “another” second image M22 emerges at the end of the second optical image path and is also supplied (step 919) to the first output port of the remote device 300. The two images are then guided to the see-through waveguides 303 for viewing by the user during the second period of time.

In some but not necessarily all embodiments consistent with the invention, the image director 307 uses a controller to dynamically alter the positions of a first area part of an electro-mechanical mirror array such that a first image that is a component part of an overall image projected by an image generator will be caused to traverse one of a number of possible optical paths within the remote device 300, and concurrently alter the positions of a second area part of the electro-mechanical mirror array such that a second image that is a component part of the overall image projected by the image generator will be caused to traverse a different one of the number of possible optical paths within the remote device 300.

To illustrate this point further, reference is made to Figure 10 which depicts elements of an image director (not shown in order to avoid overly cluttering the figure) whose elements comprise, among others, a controller 1015 and a first mirror 1011, which is an electro-mechanical mirror as just mentioned. The controller 1015 causes the first mirror 1011 to effectively operate as two mirrors by positioning first and second area parts of the mirror differently. The first area part of the first mirror 1011 is in the path of a first component part of an overall image generated by the image generator 1001, and a second area part of the first mirror 1011 is in the path of a second component part of the overall image generated by the image generator 1001. Mirror control is done in a way such that the first component part of the overall image generated by the image generator 1001 will be reflected by the first area part of the mirror 1011 towards a direction that causes it to flow through a first lens package 1017 (for image focusing) directly to a first output port 1009. The distance from the image generator 1001 to the first output port 1009 corresponds to the first optical path dl.

At the same time, the controller 1015 causes the second area part of the first mirror 1011 to be angled in a way such that the second component part of the overall image generated by the image generator 1001 will be reflected by the second area part of the mirror 1011 in a direction that causes it to be reflected by second, third, and fourth mirrors 1021, 1023, and 1025 in succession and then through a second lens package 1027 (for image focusing) to the first output port 1009. The distance from the image generator 1001 to the first, second, third, and fourth mirrors 1011, 1021, 1023, and 1025 and then through the second lens package 1027 to the first output port 1009 corresponds to the second optical path d2.

It is worth noting that embodiments relying on pixel resolution division, such as those just described with respect to Figures 7, 8, and 9 bring up the issue of how to handle occlusion. For example, it can be seen that in an embodiment as illustrated in Figure 10, an image, associated with a first virtual plane, travels to the first output port 1009 via the first optical path dl and another image, associated with a second virtual plane, travels to the same first output port 1009 via the second optical path d2. This means that light from two different sources will both converge at the first output port 1009. This is essentially an issue that can be handled by the image generator 1001 in any of a number of different ways. For example, if the image depicted at the first virtual plane VP1 is considered to be in front of the image depicted at the second virtual plane VP2, the image generator can arrange for the pixel associated with the second virtual plane VP2 to be turned off, so that the user will see only the image depicted at the first virtual plane VP1. Alternatively, it may be desired to allow some degree of transparency when one image is in front of another. Again, this can be handled by the image generator 1001 by, for example, allowing the light from the two sources to mix at the first output port 1009. Or, in another alternative, the image generator VP1 can adjust the image associated with the first virtual plane VP1 to create the effect of transparency, while concurrently turning off the pixel associated with the second virtual plane VP2. Those skilled in the art will recognize that other solutions also exist.

In another class of alternative embodiments, the principal of image resolution division is extended to provide multiplane, stereoscopic images.

Several of these embodiments will now be described with reference to Figures 11 through 18.

In one aspect, as shown in Figure 11, the pixel array 600 of Figure 6 is conceptually divided up into four quadrants, hereinafter denoted quadrants A, B, C, and D respectively. Where the pixel array 600 is considered to represent an overall image, each of the quadrants A, B, C, and D will be considered to represent a respective one of four component parts of the overall image. Next, as shown in Figure 12, different images are assigned to each of the component parts of the overall image (i.e., quadrants of the overall image) in a manner that enables each of a user’ s eyes to receive corresponding left and right sets of images, with each one of the left and right sets itself comprising sets of plural images intended for display on different virtual planes. For example, Figure 12 depicts a non limiting arrangement in which:

• an image intended to be seen by a left eye at a distance of d2 from a user point of view will be projected within a first part (quadrant A) of the overall image

• an image intended to be seen by the left eye at a distance of dl from the user point of view will be projected within a second part (quadrant C) of the overall image

• an image intended to be seen by a right eye at a distance of d2 from the user point of view will be projected within a third part (quadrant B) of the overall image

• an image intended to be seen by the right eye at a distance of dl from the user point of view will be projected within a fourth part (quadrant D) of the overall image. Referring now to Figure 13, in order to display these images to the user, in one class of embodiments, the image director 307 receives (step 1301), during a first period of time, a first image LI to be perceived by a left eye of the user as being at a first distance dl from a user point of view and also during the first period of time, receives (step 1303) a second image R1 to be perceived by a right eye of the user as being at the first distance dl from the user point of view. These two images are configured as component parts C and D of the overall image as previously shown in Figure 12. Accordingly, the image director 307 directs (step 1305) the first image LI to a first optical image path 1307 corresponding to the distance dl and leading to a Left output port of the device and simultaneously directs (step 1305) the second image R1 to a second optical image path 1309 also corresponding to the distance dl but leading to a Right output port of the device. The Left and Right output ports of the device are then guided to respective Left and Right see-through waveguides (not shown) for viewing by the user.

Next, during a second period of time (e.g., occurring At after the first period of time), the image director 307 receives (step 1311) a third image L2 to be perceived by the left eye of the user as being at a second distance d2 from the user point of view and also during the second period of time, receives (step 1313) a fourth image R2 to be perceived by the right eye of the user as being at the second distance d2 from the user point of view. These two images are configured as component parts A and B of the overall image as previously shown in Figure 12. Accordingly, the image director 307 directs (step 1315) the third image L2 to a second optical image path corresponding to the distance d2 and leading to the Left output port 1307 of the device and simultaneously directs (step 1315) the fourth image R2 to the second optical image path also corresponding to the distance d2 but leading to the Right output port 1309 of the device. As before, the Left and Right output ports 1307, 1309 of the device are then guided to respective Left and Right see-through waveguides (not shown) for viewing by the user.

It will be understood that the display of the first and second images LI and R1 during the first period of time followed by the display of the third and fourth images L2 and R2 during the second period of time occur rapidly enough that the user does not perceive the switching, but instead perceives the four images as being displayed at the same time, two at a first virtual plane VP1 and the other two at a second virtual plane VP2. This corresponds to one frame of a stereoscopic three-dimensional image.

An exemplary embodiment corresponding to aspects discussed with reference to Figure 13 will now be described with reference to Figures 14 and 15. Referring first to Figure 14, a configuration of a remote device 1400 is shown at a first time period tl. An image generator 1401 projects a first image LI onto a first area part of a first electro-mechanical mirror 1403, and projects a second image R1 onto a second area part of the first electro-mechanical mirror 1403. A controller 1405 causes the first electro-mechanical mirror 1403 to act in a split fashion, such that the first area part of the electro-mechanical mirror 1403 redirects the first image LI through a first lens package 1407 to a first output port 1409 from which it progresses through optical waveguides to see-through waveguides for viewing by a user’s left eye; and concurrently causes a second area part of the electro-mechanical mirror 1403 to redirect the second image R1 through the first lens package 1407 to a second output port 1411 from which it progresses through optical waveguides to see-through waveguides for viewing by a user’s right eye. The distances that the first and second images LI and R1 cover from the image generator 1401 to the respective first and second output ports 1409, 1411 corresponds to the first distance dl. These two images are configured as component parts C and D of the overall image as previously shown in Figure 12

Figure 15 depicts a configuration of the remote device 1400 at a second time period t2. The image generator 1401 projects a third image L2 onto a third area part of the first electro-mechanical mirror 1403, and projects a fourth image R2 onto a fourth area part of the first electro-mechanical mirror 1403. The controller 1405 causes the third and fourth area parts of the first electro mechanical mirror 1403 to act in a split fashion, such that the third and fourth area parts of the first electro-mechanical mirror 1403 redirect the third image L2 and the fourth image R2 to a second mirror 1507 which redirects the images to a third mirror 1509, which then redirects the images to a fourth mirror 1511 that is also an electro-mechanical mirror. The controller 1405 causes the fourth mirror 1511 to act in a split fashion, such that a first area part of the fourth mirror 1511 redirects the third image L2 through a second lens package 1513 to the first output port 1409 from which it progresses through optical waveguides to see- through waveguides for viewing by the user’s left eye; and concurrently causes a second area part of the fourth mirror 1511 to redirect the fourth image R2 through the second lens package 1513 to the second output port 1411 from which it progresses through optical waveguides to see-through waveguides for viewing by the user’s right eye. The distances that the third and fourth images L2 and R2 cover from the image generator 1401, past the first, second, third, and fourth mirrors 1403, 1507, 1509, 1511, then through the second lens package 1513 to the respective first and second output ports 1409, 1411 corresponds to the second distance d2. These two images are configured as component parts A and B of the overall image as previously shown in Figure 12.

It will be appreciated that the just-described embodiments combine pixel resolution division with time multiplexing to present stereoscopic multiplane images to a user. In these embodiments, images are presented concurrently to the left and right eyes of the user, with time multiplexing being used to alternately display images associated with a first virtual plane VP1 and a second virtual plane VP2.

In other alternative embodiments, pixel resolution division and time multiplexing are again used, but this time to concurrently present images associated with the first virtual plane VP1 and the second virtual plane VP2 for one of the user’s eyes during a first period of time tl, and then during a second period of time t2 to concurrently present images associated with the first virtual plane VP1 and the second virtual plane VP2 for another one of the user’s eyes. These embodiments will now be illustrated by exemplary embodiments shown in Figures 16 and 17.

Figure 16 depicts elements of an image director (not shown in order to avoid overly cluttering the figure) whose elements comprise, among others, a controller 1615 and a first mirror 1603, which is an electro-mechanical mirror as mentioned earlier. During a first period of time tl, the controller 1615 causes the first mirror 1603 to effectively operate as two mirrors by positioning first and second area parts of the mirror differently. The first area part of the first mirror 1603 is in the path of a first component part of an overall image generated by the image generator 1601, and a second area part of the first mirror 1603 is in the path of a second component part of the overall image generated by the image generator 1601. Mirror control is done in a way such that the first component part of the overall image generated by the image generator 1601 will be reflected by the first area part of the mirror 1603 towards a direction that causes it to flow through a first lens package 1605 (for image focusing) directly to a first output port 1607. The distance from the image generator 1601 to the first output port 1607 corresponds to the first optical path dl when reflected by the first mirror 1603 in this manner.

At the same time, still during the first period of time tl, the controller 1615 causes the second area part of the first mirror 1603 to be angled in a way such that the second component part of the overall image generated by the image generator 1601 will be reflected by the second area part of the mirror 1603 in a direction that causes it to be reflected by second, third, and fourth mirrors 1609, 1611, and 1613 in succession and then through a second lens package 1617 (for image focusing) to the first output port 1607. The distance from the image generator 1601 to the first, second, third, and fourth mirrors 1603, 1609, 1611, and 1613 and then through the second lens package 1617 to the first output port 1607 corresponds to the second optical path d2. The first output port 1607 in this example attaches to an optical fiber package 1619 that guides the two virtual plane images to see-through waveguides for viewing by a user’s right eye.

Virtual plane images for viewing by the user’s left eye are generated next. In particular, Figure 17 depicts the configuration during a next period of time t2 when the controller 1615 causes the first mirror 1603 to again effectively operate as two mirrors by positioning first and second area parts of the mirror differently. The first area part of the first mirror 1603 is in the path of a first component part of an overall image generated by the image generator 1601, and a second area part of the first mirror 1603 is in the path of a second component part of the overall image generated by the image generator 1601. Mirror control is different from before (i.e., different from what is shown in Figure 16) because now it is done in a way such that the first component part of the overall image generated by the image generator 1601 will be reflected by the first area part of the mirror 1603 towards a direction that causes it to flow through the first lens package 1605 (for image focusing) directly to a second output port 1621. The distance from the image generator 1601 to the second output port 1621 corresponds to the first optical path dl when reflected by the first mirror 1603 in this manner.

At the same time, still during the second period of time t2, the controller 1615 causes the second area part of the first mirror 1603 to be angled in a way such that the second component part of the overall image generated by the image generator 1601 will be reflected by the second area part of the mirror 1603 in a direction that causes it to be reflected by second, third, and fourth mirrors 1609, 1611, and 1613 in succession and then through the second lens package 1617 (for image focusing) to the second output port 1621. The distance from the image generator 1601 to the first, second, third, and fourth mirrors 1603, 1609, 1611, and 1613 and then through the second lens package 1617 to the second output port 1621 corresponds to the second optical path d2. The second output port 1621 in this example attaches to a second optical fiber package 1623 that guides the two virtual plane images to see-through waveguides for viewing by a user’s left eye.

In yet another alternative embodiment, pixel resolution division is used without time multiplexing to produce stereoscopic multiplane images for viewing by a user. Here as before in some of the other embodiments, different images are assigned to each of the component parts of the overall image (i.e., quadrants of the overall image) in a manner that enables each of a user’s eyes to receive corresponding left and right sets of images, with each one of the left and right sets itself comprising sets of plural images intended for display on different virtual planes. Reference is again made to Figure 12 to illustrate this aspect. Referring now to Figure 18, in this class of embodiments, the image generator produces an overall image comprising four component parts as illustrated in Figure 12 and projects these (step 1801) to the image director. The image director (e.g., by controlling parts of several electro-mechanical mirror arrays) causes first virtual plane images LI and R1 to travel a first distance dl to one of two output ports (step 1803) and concurrently causes second virtual plane images L2 and R2 to travel a second distance d2 to the other one of the two output ports (step 1805). In addition, the image director sorts the four component parts of the overall image so that the two component parts to be viewed by one of the user’s eyes is directed to a first output port, and the other two component parts (which are to be viewed by the other one of the user’s eyes) is directed to a second output port (step 1807). The two output ports supply the image pairs to respective optical fibers which guide them to the see-through waveguides for viewing by the user’s left and right eyes (step 1809).

Figures 19A, 19B, and 19C depict aspects of the technology relating to dynamic adjustment of distances associated with virtual planes. Up to this point, the embodiments have illustrated only fixed virtual plane distances, for example, the distances dl and d2. However, other aspects of the technology permit these to be dynamically adjustable. Referring first to Figure 19A, a configuration for displaying a first virtual plane is shown. An image generator 1901 projects an image onto a first autofocus lens 1903. The image emerges from the first autofocus lens 1903, goes through a lens package 1905, and then goes to a second autofocus lens 1907. The image emerging from the second autofocus lens 1907 is focused onto an optical fiber package 1909, which guides the image to a user headset as described in earlier embodiments.

The size of the image can be altered to create the impression of the depicted image being a different distance away from the user. To achieve this, as shown in Figure 19B, the first autofocus lens 1903 is moved some amount (dx) to a different position (step 1911). (It is noted that the arrangement may include additional lenses inside the lens package that need to move as well in order to make the virtual distance appeared to be the wanted position - these have not been shown in the figure in order to avoid obscuring the discussion with unnecessary details that would be known to a person of ordinary skill in the art.)

In addition to changing the size of the image, moving the first autofocus lens 1903 also has the effect of blurring the image at the point that it is projected onto the optical fiber package 1909. This is shown in Figure 19B as the focal point 1913, which is an undesired amount (dz) inside the optical fiber package 1909.

To compensate for this unwanted blurriness, the second autofocus lens 1907 is also moved by an amount (dy) (step 1915) to a new position that reestablishes a sharp image at the entrance 1917 to the optical fiber package 1909.

To further facilitate an understanding of various aspects of the technology, Figure 20 is, in some respects, a flowchart 2000 of actions associated with a remote device in accordance with a number of the above-described embodiments that utilize time division multiplexing. In other respects, the blocks depicted in Figure 20 can also represent means (e.g., processing means) for carrying out the described actions.

Initially, an image generator projects virtual plane images, in time sequence, onto an electro-mechanical mirror array (step 2001). The mirror is then controlled to sequentially move into different positions (step 2003) that reflect each sequentially presented image onto a different path. Thus, for some arbitrary number (x) of virtual planes to be shown to a user, a first image is reflected onto a first optical path associated with a first virtual plane (step 2005), a second image is reflected onto a second optical path associated with a second virtual plane (step 2007), and so on until an x ^th image is reflected onto an x ^th optical path associated with an x ^th virtual plane (step 2009).

At the end of each optical path, the image is projected onto an optical fiber package (step 2011). The optical fiber then guides the image onto a set of see- through waveguides that are configured to display the number of optical wavelength components (e.g., three in the case of RGB images) (step 2013). The user is able to see the image by looking through the see-through waveguides. To further facilitate an understanding of other aspects of the technology, Figure 21 is, in some respects, a flowchart 2100 of actions associated with a remote device in accordance with a number of the above-described embodiments that utilize pixel resolution division. In other respects, the blocks depicted in Figure 21 can also be considered to represent means (e.g., processing means) for carrying out the described actions.

Initially, a display generator projects (step 2101) an overall image onto an electro-mechanical mirror (e.g., MEMS), wherein the overall image comprises respective component parts for each virtual plane to be displayed.

Different area parts of the electro-mechanical mirror are concurrently controlled such that, for some number of virtual planes (x): a first area part of the electro-mechanical mirror reflects a first component part onto a respective first path for a first virtual plane (step 2103), a second area part of the electro mechanical mirror reflects a second component part onto a respective second path for a second virtual plane (step 2105), and so on until an x ^th area part of the electro-mechanical mirror reflects an x ^th component part onto a respective x ^th path for an x ^th virtual plane (step 2107).

At the end of each optical path, the images are projected onto an optical fiber package (step 2109). The optical fiber then guides the image onto a set of see-through waveguides that are configured to display the number of optical wavelength components (e.g., three in the case of RGB images) (step 2111). The user is able to see the image by looking through the see-through waveguides.

Figure 22 is a block diagram of an arrangement 2200 in accordance with some but not necessarily all embodiments consistent with the invention. The arrangement 2200 comprises a remote device 2201 coupled to respective left and right holographic waveguides 2203, 2205 by respective bundles of optical fibers 2207, 2209.

Images are generated by a host system 2211 in the remote device 2201. The host system 2211 comprises a memory 2213 coupled to a video processing unit 2215. The video processing unit 2215 utilizes data and instructions from the memory 2213 to generate images which are supplied to interfaces 2217 that transfer the generated images to a display system 2219.

The display system 2219 can be configured in accordance with principles disclosed with respect to any of the image directors described earlier in this document. Images generated by the display system 2219 for a user’s left and right eyes are, as mentioned above, transferred to the holographic waveguides 2203, 2205 by the sets of optical fibers 2207, 2209.

Aspects of an exemplary controller as discussed in various embodiments are shown in Figure 23, which illustrates an exemplary controller 2301 of a remote device in accordance with some but not necessarily all exemplary embodiments consistent with the invention. In particular, the controller 2301 includes circuitry configured to carry out any one or any combination of the various functions described above. Such circuitry could, for example, be entirely hard-wired circuitry (e.g., one or more Application Specific Integrated Circuits - “ASICs”). Depicted in the exemplary embodiment of Figure 23, however, is programmable circuitry, comprising a processor 2303 coupled to one or more memory devices 2305 (e.g., Random Access Memory, Magnetic Disc Drives, Optical Disk Drives, Read Only Memory, etc.) and to an interface 2307 that enables bidirectional communication with other elements of the remote device (e.g., control interfaces of the one or more electro-mechanical mirrors). The memory device(s) 2305 store program means 2309 (e.g., a set of processor instructions) configured to cause the processor 2303 to control other remote device elements so as to carry out any of the aspects described above. The memory device(s) 2305 may also store data (not shown) representing various constant and variable parameters as may be needed by the processor 2303 and/or as may be generated when carrying out its functions such as those specified by the program means 2309.

The various embodiments provide a number of advantages over earlier technology. For example, aspects of exemplary embodiments enable XR glasses to be produced that are lighter weight than previously known in the art. Further, by separating heat generating elements from the XR glasses, the glasses become more comfortable to a user, and have a smaller form factor. Further, since fewer constraints need to be imposed on the wearable XR computing device that is placed in a pouch or worn on a belt, more power and computational resources can be added to the overall system without burdening the user. The invention has been described with reference to particular embodiments. However, it will be readily apparent to those skilled in the art that it is possible to embody the invention in specific forms other than those of the embodiment described above. Thus, the described embodiments are merely illustrative and should not be considered restrictive in any way. The scope of the invention is further illustrated by the appended claims, rather than only by the preceding description, and all variations and equivalents which fall within the range of the claims are intended to be embraced therein.