MODULATION

RELATED APPLICATIONS

[0001] The present invention claims priority to U.S. Patent Applications No. 15/335,298, filed on 10/26/2016, (Our Ref. 14100P0031 ) which claims priority to U.S. Patent Application No. 15/236,101 , filed on 8/12/2016 (14100P0030). The present invention also claims priority to U.S. Patent Application No. 15/358,040 filed on 1 1/21/2016 (14100P0036). All of the above applications are incorporated herein by reference.

FIELD

[0002] The present invention relates to a near-eye display system, and more particularly to a binocular near-eye display system including a modulation stack.

BACKGROUND

[0003] Near-eye display systems are becoming more common. Such near-eye display systems attempt to provide a three-dimensional display to the user. In the prior art, displays rendering multiple focal planes utilized mechanical movement such as gears or liquid lenses. Such mechanisms are expensive, slow, and relatively fragile. Another prior art method of displaying multiple focal lengths uses multiple mirrors and lenses. BRIEF DESCRIPTION OF THE FIGURES

[0004] Figure 1 is an illustration of one embodiment of a near eye display system, in which the present invention may be used

[0005] Figure 2A is a block diagram of one embodiment of a near eye display system.

[0006] Figure 2B is a block diagram of one embodiment of the adjustment elements, for customized image adjustment.

[0007] Figure 2C is a block diagram of one embodiment of a projection assembly.

[0008] Figures 3A and 3B are diagrams of one embodiment of a digital light path length modulator in a near eye display (NED) system.

[0009] Figures 3C-3F are diagrams of embodiments of a digital light path length modulator in a binocular near eye display system.

[0010] Figure 4 is a block diagram of one embodiment of a modulation stack including a plurality of digital light path length modulators.

[0011] Figure 5A illustrates one embodiment of the effect of using a transverse OPLE with non-reflected light.

[0012] Figure 5B illustrates one embodiment of the effect of using the OPLE of Figure 5A, with reflected light.

[0013] Figure 5C illustrates one embodiment of the effect of using a self- aligned OPLE, with reflected light.

[0014] Figure 5D illustrates one embodiment of using a longitudinal OPLE.

[0015] Figure 5E illustrates an example of light path extensions and the effect on a display. [0016] Figure 6 is a flowchart of one embodiment of using the binocular display system including multiple focal planes.

[0017] Figure 7 is a flowchart of one embodiment of adjusting the display to correct for defects of the eyes.

[0018] Figure 8 is a flowchart of one embodiment of using the binocular display to create the correct disparity.

[0019] Figure 9 is a flowchart of one embodiment of using the binocular display to match vergence, focus, and disparity.

[0020] Figure 10 illustrates some of the exemplary corrections that may be made using the binocular system.

DETAILED DESCRIPTION

[0021] A binocular near-eye display system utilizing a modulation stack is described. A modulation stack includes one or more digital light path length modulators, to adjust the path length of light. A digital light path length modulator can be used to create two focal planes. In one embodiment, using a modulation stack with a plurality of digital light path length modulators, the number of focal planes can be increased. Creating a display in which the 3D indicia of parallax, focus, and vergence match provides the capacity to build a system that can meet the physiological requirements of human vision. This produces a better quality 3D display than is currently possible and can prevent the discomfort associated with 3D displays.

[0022] The following detailed description of embodiments of the invention makes reference to the accompanying drawings. The drawings show various embodiments of practicing the invention. Description of these embodiments is in sufficient detail to enable those skilled in the art to practice the invention. One skilled in the art understands that other embodiments may be utilized and that logical, mechanical, electrical, functional and other changes may be made without departing from the scope of the present invention. The following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

[0023] Figure 1 is an illustration of one embodiment of a near-eye display system, in which the present invention may be used. The binocular near-eye display system, in one embodiment includes a head-mounted display, which includes a display for both eyes of a user. In one embodiment, the near-eye display system is a display mounted in another device, such as a camera, microscope display, focal assist in a microscope, binoculars, digital scope, medical or surgical display system, endoscope, binocular range finder, etc. In one embodiment, the near-eye display system is coupled with a speaker system to enable the playing of audio-visual output such as movies. The near-eye display system may provide an opaque display, partially transparent display, and/or transparent display. The near-eye display system may provide augmented reality, mixed reality, and/or virtual reality display.

[0024] Figure 2A is a block diagram of one embodiment of a near-eye display system 200. The near-eye display system 200 includes, in one embodiment, a display subsystem 205, including a right eye 205A and left eye 205B display subsystem, an audio subsystem 254, a processor 250, a memory 252, and optionally an auxiliary data system 260. The display subsystem 205 generates image data. Memory 252 may be a buffer-memory, enabling the near-eye display system 200 to stream content. The memory 252 may also store image and video data for display. I/O system 256 makes image, audio, video, VR, or other content available from other sources (e.g. enables downloading or streaming of content from various sources.)

[0025] The display subsystem 205 includes, in one embodiment, a right eye display subsystem 205A. The right eye display subsystem 205A includes an image source 21 OA, a projection assembly 220A, and an imaging assembly 240A.

[0026] The image source 21 OA in one embodiment includes a light source 217A, which in one embodiment is a spatial light modulator (SLM). The image source 210 in one embodiment also includes a digital correction system 215A, to correct the output of the light source 217A, to account for distortion in the projection assembly 220A. In one embodiment, the light source 217A may be a real image, in which case the light source 217A is external to the system, and there is no digital correction. In one embodiment, the NED 200 may be used for one or more of virtual reality (digital image source), augmented or mixed reality (a combination of real and digital image sources), and reality (real image source.)

[0027] The projection assembly 220A includes a polarizer 225A in one embodiment. The polarizer 225A passes through light with a particular polarization. In one embodiment the polarizer may provide pixel-based polarization. Utilizing pixel-based polarization, the system can provide multiple depths to both eyes.

[0028] The projection assembly 220A includes a modulation stack 230A. The modulation stack 230A includes one or more digital light path length modulators 235A, 237A. The digital light path length modulators 235A, 237A alter the light path length based on the polarization of the light. In one embodiment, polarizer 225A may be positioned after modulation stack 230A.

[0029] Imaging assembly 240A is used to display the image to the user. In one embodiment, the display subsystem 205A may include additional mechanical and optical elements which can provide correction or alteration of the image.

[0030] The display sub-system 205 includes a left eye subsystem 205B to provide binocular display. In one embodiment, the left eye subsystem 205B may include only an imaging assembly 240B, while the image source 21 OA, polarizer 225A, and modulation stack 230A may be shared between the right-eye display subsystem 205A and the left eye display subsystem 205B. In another embodiment, the binocular elements display subsystem 205B may include more of the elements, including one or more of a light source 210B, polarizer 225B, and modulation stack 230B, in addition to the separate imaging assembly 240B. In one embodiment, if the same modulation stack 230A is shared, then the elements appear at the same distance. In one embodiment, by having separate elements, the system can more easily adjust for the user's prescription. In one embodiment, the system provides monocular focal cues for each eye, including adjusting for motion parallax, depth, sizing, occlusion, etc. By adjusting two eyes, utilizing the binocular system, the system further provides binocular focal cues including binocular parallax and convergence.

[0031] In one embodiment, the system may include an eye tracking mechanism 265, for one or both eyes 265A, 265B. The eye tracking mechanism 265A/B tracks the gaze vector of the user's eyes. In one embodiment, the system may place image elements in one or more selected locations based on where the user's eyes are looking, using the eye tracking mechanism 265A/B. In one embodiment, the system may select one or more focal planes, based on where the user is looking, as determined based on data from the eye tracking mechanism 265A/B. In one embodiment, the eye tracking mechanism 265A/B is an infrared optical sensor or camera to sense light reflected from the eye. Other techniques may be used for eye tracking. Eye tracking mechanism 265A/B may track one or both eyes.

[0032] In one embodiment, the system may receive data from auxiliary data system 260. The auxiliary data system may provide information for selecting the focal lengths and may provide controls. As noted above, the modulation stack 230 can create a perception of an image element at various virtual object distances. The auxiliary data system 260 may be used to select a virtual object distance, based on various factors. The auxiliary data system 260 may also be used by the user/wearer to provide feedback or commands.

[0033] In one embodiment, biometric systems 270 may be used to detect the user's state, including the user's identity, emotional state, etc. In one embodiment, the biometric systems 270 may be used to customized and/or control the system.

[0034] In one embodiment, the biometric system 270 may be used to detect the user's vision correction, and provide adjustment based on the vision correction. In one embodiment, this may be done by scanning the eye. In one embodiment, the user may be requested to input his or her prescription. In one embodiment, the position of the display subsystem (one or both) may be adjusted based on the user's inter-pupillary distance (IPD). In one embodiment, there may be user interface to receive medical data 261. In one embodiment, the prescription and IPD may be entered through manual adjustment 262. In one embodiment, the system may optionally store these settings, so that if multiple users share the near- eye display system 200, the system may be able to auto-adjust to the prior settings. In one embodiment, the adjustment may be based on the data from biometric systems 270 and/or eye tracking mechanism 265A/B.

[0035] Environmental feedback system 272 utilizes sensors to obtain data from the external environment. For example, the environmental feedback system 272 may identify the position of a wall, or window, or other targeted location or object, so data displayed by display subsystem 205 can have a virtual object distance appropriate for that target location. The environmental feedback system 272 may be a range sensor, camera, or other system.

[0036] Content data-based focal point selection 274 enables the system to selectively choose a virtual object distance, based on what is being displayed. For example, the system may selectively choose a portion of the image for focus.

[0037] In one embodiment, user input systems 276 enable focus selection based on head tracking, gestures, voice control, and other types of feedback or input systems. Such user input systems 276 may include video game controllers, microphones, cameras, inertial measurement sensors, and other sensors for detecting user input. In one embodiment, user input systems 276 may provide manual inputs, including one or more of sliders, dials, computer-based inputs, etc. In one embodiment, the user inputs systems 276 may be provided by a linked mobile device or other system.

[0038] Other control data 278 may also be provided to the system. Any of this data from auxiliary data system 260 may be used to adjust the virtual object distance of one or more image elements. In one embodiment, in addition to auxiliary data system 260, the system may additionally accept manual adjustment 262. In one embodiment, the manual adjustment may be used to correct for the user's optical issues, which sets a baseline for the user. In one embodiment, manual adjustment 262 may provide an initial IPD and diopter setting. In one embodiment, the manual adjustment is stored so that a user may have a customized setting, which may be beneficial if the near-eye display system is shared.

[0039] In one embodiment, the near-eye display 200 may provide depth blending. In one embodiment, the system 200 enables depth blending between the focal lengths created using the modulation stack 230. In one embodiment, depth blending uses weighting of pixel values between adjacent planes and sets opacity. This creates an appearance of continuous depth. In one embodiment, the weighting may be linear weighting. In one embodiment, nonlinear optimization techniques may be used. In one embodiment, the image source 210 adjusts the pixel values output, to create such depth blending.

[0040] Figure 2B is a block diagram of one embodiment of the adjustment elements, for customized image adjustment. [0041] In one embodiment, the customized image adjustment 280 may use one or more of the following elements: vergence correction 281 , focal plane correction 282, brightness/luminosity adjustment 283, magnification 284, color correction 285, artificial color addition for emphasis 286, contrast ratio correction 287, rotation and transposition-based displacement 288, and periodic adjustor for brain training 289.

[0042] Figure 2C is a block diagram of one embodiment of a projection assembly 290. The projection assembly 290, in one embodiment, includes a plurality of digital light path length modulators (293, 295) as well as a plurality of intermediate optics elements (292, 294, 296, 297) together forming a modulation stack 291 . In one embodiment, the projection assembly in a real system may include 6-30 elements which include lenses, mirrors, apertures, and the like, referred to as intermediate optics. In one embodiment, the intermediate optics may be interspersed with the digital light path length modulators. In one embodiment, the intermediate optics may be positioned before and/or after the set of digital light path length modulators. In one embodiment, polarization filter 299 may be positioned before 299A or after 299B in the modulation stack 291 .

[0043] In one embodiment, the projection assembly 290 may correct for chromatic aberration and other irregularities of optical systems.

[0044] Figures 3A and 3B are diagrams of one embodiment of a near-eye display (NED) system including a modulation stack. The light modulator 310 outputs polarized light, both state 1 and state 2 polarized light, in one embodiment.

Polarizing filter 320 removes the state 1 polarized light, and passes through state 2 polarized light only. [0045] The modulation stack 355 includes one or more digital light path length modulators 350. For simplicity the illustration here includes a single digital light path length modulator 350. The digital light path modulator 350 includes a polarization modulator 330, which can rotate the polarization of light, and an optical light path extender (OPLE) 340 which selectively extends the light path length, based on the polarization of the light. In one embodiment, the OPLE 340 may be a transverse OPLE or a longitudinal OPLE. These OPLEs are described in co-pending U.S. Patent Application No. 15/236,101 , filed on 8/12/2016 (14100P0030) and U.S. Patent Application No. 15/358,040 filed on 1 1/21/2016 (14100P0036). incorporated herein by reference.

[0046] The polarization modulator 330 in Figure 3A is "off," leaving the state 2 polarized light to pass through. In this context, the term "on" refers to a setting in which the polarization modulator 330 alters the polarization of light, while the term "off' refers to the setting in which the polarization modulator 330 does not alter the polarization of light.

[0047] The OPLE 340 in one embodiment is a transverse OPLE with a plurality of polarization sensitive reflective elements, which reflect state 1 polarized light, while passing through state 2 polarized light. Here, state 2 polarized light is transmitted straight through. The output in one embodiment is transmitted to near- eye display (NED) projection optics 360. Though it is not shown, additional optical elements may be included in this system, including lenses, correction systems, etc. In another embodiment, the OPLE 340 may be a longitudinal OPLE in which the state 1 polarized light is reflected back by a polarization sensitive reflective element.

[0048] Figure 3B is a diagram of the near-eye display system of Figure 3A with the polarization modulator "on." Here, again, the polarizing filter passes only state 2 polarized light. However, here, the polarization modulator 330 modulates the light, and outputs state 1 polarized light. The state 1 polarized light is reflected by the polarization sensitive reflective elements of transverse OPLE 340, in this illustration. In another embodiment, the polarization sensitive reflective element on the top of the OPLE 340 reflects the state 1 polarized light back through the OPLE 340. Thus, this light goes through a longer light path than the light with state 2 polarization, which is passed through without reflection.

[0049] A comparison of Figures 3A and 3B shows that the state 1 polarized light has a longer light path through the OPLE than the state 2 polarized light. In this way, a digital light path length modulator 350 can change the light path length. While only a single digital light path length modulator 350 is shown here, a plurality of digital light path length modulators 350 may be stacked to provide a larger number of light path lengths.

[0050] Figures 3A and 3B show a time sequential embodiment, in which all of the light entering the digital light path length modulator 350 has one polarization, and is either modulated or not modulated by polarization modulator 330. In this example, the system switches between the states shown in Figures 3A and 3B, in time. The polarization modulator 330 may use pixel-based polarization to selectively modulate the polarization of a subset of the light impacting at a time, in one embodiment. In one embodiment, modulation may be based on location, time, color, wavelength, and optionally other differentiable factors.

[0051] Figures 3C-3E illustrate embodiments various configurations for a binocular near-eye display system. Figure 3C illustrates one embodiment in which the system has fully separate display subsystems, with each eye having a light modulator 394, 395, a modulation stack 396, 397, and projection optics 398, 399. [0052] Figure 3D illustrates an embodiment in which the left and right eye share a light modulator 370, and each eye has a separate modulation stack 372, 374 and separate projection optics 376, 378. As noted above, the system may share one or more of the light modulator and modulation stack, but generally provides separate projection optics for each eye. In this configuration, the separate modulation stacks 372, 374 each include one or more OPLEs, and polarization modulators. Thus, the system can set path length for each eye independently.

[0053] Figure 3E illustrates a situation in which the system shares a single light modulator 380, and a single modulation stack 382, and each eye has projection optics 384, 386. Because of the positioning of the eyes, and the functioning of the modulation stack, described below in more detail, the light projected for the right eye does not interfere with the light projected for the left eye. In one embodiment, the projection optics 384, 386 may be adjusted manually as discussed above.

Subsequently, while displaying a series of images such as a video, the system may adjust the data for both eyes uniformly to maintain continuity of content. The adjustments maintain the matched changes. In one embodiment, when using one shared modulation stack 382 as shown here, the adjustments are made manually or through a separate optical element. In one embodiment, the manual changes may be made by using a knob or other setting tool to set options, and then using the system including the modulation stack 382 to maintain the fixed difference, while adjusting for different image elements and display categories. In another

embodiment, the system may maintain the matched changes using the system only, without a separate disparity adjustment mechanism

[0054] Figure 3F illustrates one embodiment in which the system has a shared modulation stack 390 but separate light modulators 388, 389 and separate projection optics 392, 393. The binocular system may operate with any of the configurations shown in Figures 3C-3F.

[0055] In one embodiment, this binocular display system may be used in near-eye displays (NED), as well as a binocular microscope to enable the display to show depth perception and for digital microscopes, recreating a similar eye feel to high powered microscopes. Other uses of the binocular display system may include other types of displays, such as those associated with cameras, binoculars, digital scopes, medical or surgical display systems, endoscope, binocular range finders, etc.

[0056] Figure 4 illustrates one embodiment of a modulation stack 400 including four digital light path length modulators.

[0057] The modulation stack includes four digital light path length modulators. Each of the digital light path length modulators 450, 455, 460, 465 includes a polarization modulator and an OPLE. In this example, the first OPLE 410 is a longitudinal OPLE, while the other OPLEs are transverse OPLEs. One of the transverse OPLEs 420 is a self-aligning OPLE.

[0058] In various embodiments, one or more of the following variations may be made: the effective thickness of the OPLEs may vary, as may the angles of the polarization sensitive reflective elements, and the OPLE may include one, two, or more plates. The effective thickness of the OPLE is defined as the cumulative thickness of the plates which are parts of the OPLE. Thus the effective thickness of OPLE 420 is different than the thickness of OPLE 440, even though the individual plates in the two OPLEs 420, 440 are identical.

[0059] With the shown set of four different OPLEs, the system can create up to sixteen, 2 ⁴ focal lengths by selectively modulating the polarization, as follows: OPLE 1 OPLE 2 OPLE 3 OPLE 4

State 1 State 1 State 1 State 1

State 1 State 1 State 1 State 2

State 1 State 1 State 2 State 1

State 1 State 1 State 2 State 2

State 1 State 2 State 1 State 1

State 1 State 2 State 1 State 2

State 1 State 2 State 2 State 1

State 1 State 2 State 2 State 2

State 2 State 1 State 1 State 1

State 2 State 1 State 1 State 2

State 2 State 1 State 2 State 1

State 2 State 1 State 2 State 2

State 2 State 2 State 1 State 1

State 2 State 2 State 1 State 2

State 2 State 2 State 2 State 1

State 2 State 2 State 2 State 2

[0060] Although the modulation stack may create a plurality of focal planes, in one embodiment the system provides for focal plane blending. Focal plane blending enables placement of elements between focal planes defined by the OPLEs. Focal plane blending creates a perception of intermediate focal planes. This enables the creation of a blended focal plane at a location where the desired position of the virtual object is, corresponding to appropriate parallax.

[0061] In one embodiment, because the light exits from both sides of a longitudinal OPLE, the longitudinal OPLE 410 is preferentially a first OPLE in a modulation stack 400 that includes longitudinal OPLEs. In one embodiment, the number of longitudinal OPLEs 410 is limited by the level of light loss for each longitudinal OPLE.

[0062] The figures of 5A-5C illustrate the use of transverse OPLEs for light path extension. Figure 5D illustrates the use of a longitudinal OPLE. One of skill in the art would understand that many of the illustrations and discussion below, with respect to transverse OPLEs apply to longitudinal OPLEs as well. [0063] Figure 5A illustrates one embodiment of the effect of using a transverse OPLE 510, with a light source for non-reflected light. The light source is real source 530. As can be seen, in this example, for state 2 polarized light which is not reflected by the polarization sensitive reflective elements, the real light source 530 and the "virtual" or perceived light source are in the same position. This figure additionally shows that for a real light source 530, light travels in a cone, rather than a straight light as is usually illustrated for simplicity.

[0064] Figure 5B illustrates one embodiment of the effect of using the OPLE 510 of Figure 5A, with a light source for reflected light. In this illustration, state 1 polarized light is reflected. Thus, the user's perceived "virtual light source" 540 is spatially shifted from the real light source 530. The lengthening of the light path shifts the virtual source vertically, while the movement of the light caused by the bounce shifts the apparent light source horizontally. In the example shown, the virtual light source is shifted to the right and back. The virtual rays show that the user's perception tracks back the light, to perceive the virtual source.

[0065] Figure 5C illustrates one embodiment of the effect of using a self- aligned transverse OPLE 550, with a light source, for reflected light. As can be seen, by using the self-aligned OPLE 550, the virtual light source 540 appears to be further away (e.g. lengthening the virtual object distance) from the real source 580, but not shifted in position. Although the self-aligned OPLE 550 shown in Figure 5C has no gap between the plates, a gap, including a gap with additional optical elements, could continue to provide this self-alignment feature. Furthermore, while the two plates in the shown self-aligned OPLE 550 are matched, they need not have an identical thickness or angle for the polarization sensitive reflective elements, as discussed above. [0066] Figure 5D illustrates one embodiment of a longitudinal OPLE 590, with a light source for reflected light. As can be seen, by using the longitudinal OPLE, the virtual light source 592 appears to be further away (e.g. lengthening the virtual object distance) from the real source 592, but it is not shifted in position. The center line, illustrating light traveling in a straight line is reflected from the top surface, then the bottom surface, before exiting the OPLE 590. This can be more clearly seen in the light shown impacting at an angle.

[0067] Figure 5E shows the effect of light extension on a perceived image. The illustration shows three degrees of light extension side by side, for comparison. The first one passes the light without any lengthening, so the image is perceived at the focal plane of the display. The second one lengthens the light path, which causes the user to perceive the image at a different focal plane. The third one lengthens the light path further, which causes the user to perceive a virtual image at a third focal plane. Thus, by controlling the length of the light extension, a system can create virtual images and image elements at various focal points. Using the digital light path length modulator, the system can adjust the light path digitally, and selectively position virtual images at various virtual object distances.

[0068] Figure 6 is a flowchart of one embodiment of using the binocular display system including multiple focal planes. The process starts at block 610. In one embodiment, this process is continuously active when images, whether still images of frames or subframes of video images, are being displayed.

[0069] At block 615, a plurality of image elements are identified. The image elements may be in the same image frame, or may be in separate subframes.

[0070] At block 620, the focal length and conjugate ratio is determined for each of the plurality of image elements, for both eyes. This is based on the intended virtual location for each image element, based on one or more of the user's intention, the creator's intention, and the user's physical circumstances.

[0071] At block 630, appropriate light lengthening is selected, to place the virtual object distance at the selected distance for image element. As noted above, this may be done on a per subframe or per pixel basis.

[0072] At block 640, the input is adjusted to correct for any artifacts created by the light path lengthening, and other issues. Other issues may include vergence-based disparity correction, luminosity correction, amblyopia correction, different perceived focal planes, perceived focus adjustments, etc. In one

embodiment, the system may adjust color to assist with color blindness, or decreased color perception. The adjustment may include adjusting color

differentials, removing noise in the color overlap region, or artificial color fringing. In one embodiment, this adjustment is done by the digital correction system 215A/B, for each eye. The digital correction system 215A/B in one embodiment adjusts the output of light source 217A/B to account for these issues, as well as for artifacts.

[0073] At block 650, the process uses the binocular display system to display image elements for each eye at the plurality of perceived focal distances. The process then ends at block 660. Note that while the process is shown as ending, as long as there are further image elements to display, the process continues.

[0074] Figure 10 illustrates some of the exemplary adjustments. An adjustment or correction may be referred to as a "display differential," representing a difference in the image element(s) presented to one or both of the eyes of the user from the no correction state. For example, for someone with perfect vision there may be no display differential (10A). [0075] If there is a vergence disparity, the vergence-based disparity correction may place the image elements further apart (10B) or closer together (10C) so that the perceived placement is correct. There may be brightness or luminosity correction (10D) if the two eyes do not perceive light the same way. In one embodiment, there may be inter-scene contrast ratio correction, by adjusting the brightness of elements relatively to each other for the whole, or portions of the image.

[0076] For amblyopia, the image element may be moved, and made less bright for the dominant eye, to force the other eye to work (10E). For different prescriptions (diopters), the perceived focal planes may be shifted (10F), so that despite having different prescriptions for each eye, the two eyes perceive the image elements on the same plane (or on different planes, as designed by the data.) For some vision problems, the system may provide differential magnification, to fix the effect of some vision problems. This may be done for the whole scene, so that the user's eyes perceive images at the same size. In one embodiment for monovision correction or strong eye dominance, which causes 3D fusion problems, the system may provide correction as well. In one embodiment, the correction may include altering one of the images, as shown. In one embodiment, the correction may include alternating the image displayed between the eyes. In one embodiment, the correction may include other changes. In one embodiment, the system may try various modifications to provide an experimental correction, based on the user's reaction. Because people with vision issues have different reactions to such corrections, in one embodiment, the processor may run through an experimental correction protocol, which utilizes various correction methods, and evaluates the user's response using eye tracking mechanisms and biometric systems, and user feedback to identify an optimal correction. In one embodiment, such experimental correction protocols are made available under supervision by a medical or other trained professional.

[0077] In one embodiment, there may be different color levels, for contrast. (10H). In one embodiment, the system can adjust color overall for generic color blindness. The color adjustment may remove noise, and overlap between the colors. In one embodiment, the system may also adjust the focal planes by color. The system may also selectively adjust color for one eye for color blindness or reduced color sensitivity in one eye due to macular degeneration or other causes. In one embodiment, the system may add artificial color fringing to provide focus clues. In one embodiment, the system may add a blur filter, to the entire image, to a portion of the image, or by color, to create a perceived focus adjustment. In one embodiment, a whole scene may be positioned in one focal plane, based on where the user is looking, and the elements that the user is not focused on may be artificially blurred.

[0078] In one embodiment, the system also provides rotational

displacement or transposition, in addition to the changes shown in Figure 10. In one embodiment, this is done with respect to the bounding reality box (e.g. within the field of view). In one embodiment, the transposition may be different for the two eyes, to address the limits of binocular fusion. In one embodiment, by taking auxiliary data, the system can change the orientation and focal plane, and make disparity changes as needed, as the user moves around, tilts the head, etc.

Disparity correction is ensuring that the parallax of the eyes is positioned so that the user's focus is at the correct location. A disparity is the angular misalignment of the images perceived by the two eyes that makes it difficult for the user's brain to fuse the images and perceive the scene as a single coherent scene. Because of the differences in perceptions of the right and left eye, the system applies disparity correction. All of these corrections are in the software, in adjusting the data output by the digital correction system and adjusting the modulation stack to position the image with the right color and brightness at the right focal plane.

[0079] In one embodiment, the system dynamically matches vergence and focus, and adjusts these elements in real time. For an augmented or mixed reality system, the display may additionally be adjusted based on external clues. For example, an image may be positioned on a wall, even as the user moves closer to or further away from that wall.

[0080] In one embodiment, the correction is to account for the parallax and focal shift, based on the real perception of the user, rather than the idealized perception assumed by a default system.

[0081] In one embodiment, the system, because it is fully adjustable on a binocular basis, may be used to train the brain, by changing parameters from the initial values to the more optimized values. The brain adjusts to the images being shown.

[0082] For example, if a user has a strongly dominant eye, generally the dominant eye provides resolution and the non-dominant eye only provides depth cues. However, the system can adjust for this, by adjusting luminosity to shift dominance. In one embodiment, the system may alternately or additionally adjust the distance of the object shown to the dominant eye. This can force the brain to learn and adjust. Thus, in one embodiment, the system may be used to train the user's brain, to compensate for existing eye issues. For example, for developmental amblyopia, the system may force the lazy eye to focus by placing the image shown to the dominant eye out of focus, or in soft focus. In one embodiment, the system may be used to train the eyes for cross-dominant shooters (e.g. to enable someone who needs to be able to adjust eye dominance to learn to do so.) In one

embodiment, the system may slowly adjust these factors, focus, magnification, luminance, focal plane, and vergence digitally. By combining such changes with testing, the system may be to improve vision. Additionally, by adjusting the display and forcing the user's vision to compensate, the system may be used for lengthening some muscles differentially for therapeutic reasons.

[0083] Figure 7 is a flowchart of one embodiment of adjusting the display to correct for defects of the eyes. The process starts at block 710. At block 720 light is received for display. At block 730, the condition of one or both of the eyes of the user are identified. The conditions may include myopia and/or hyperopia (near sightedness and/or far sightedness, astigmatism, glaucoma, color blindness, amblyopia, presbyopia, and other known conditions which alter the ability of the eyes to perceive objects. In one embodiment, these conditions may be specified by a user. In one embodiment, some settings, for example diopters, may be manually set by the user. In one embodiment, the user may have a separate setting based on whether he or she is wearing prescription glasses or contacts. In one embodiment, the system is designed so it can be utilized with and without corrective lenses. In one embodiment, some conditions, such as amblyopia, may be automatically detected by the eye tracking system, which can determine the focus location of each eye.

[0084] At block 740, the modulation path and other aspects of the image element are adjusted for each eye, based on the condition of the eye, and external conditions, if appropriate.

[0085] At block 750, the binocular display system is used to display a plurality of image elements, for each eye, providing the correct parallax. [0086] Figure 8 is a flowchart of one embodiment of using the binocular display to create the correct disparity. The process starts at block 810.

[0087] At block 820, the vergence and perceived focal distance of the user is identified. In one embodiment, this may be done by observing the user's eyes as test images are displayed. In one embodiment, this may be done via user input. In one embodiment, this may be done by receiving user feedback during display of a plurality of test images or other content. In one embodiment, this may be done by requesting feedback from a user, the feedback reflecting a comparison of real objects in the environment and displayed object in the binocular NED system.

[0088] At block 830, the light path length is adjusted for each eye of the binocular display for the appropriate perceived focal distances and image element locations for correct disparity.

[0089] At block 840, the plurality of image elements are displayed in a plurality of subframes at the selected plurality of perceived focal distances.

[0090] At block 850, the system determines whether the focal distance should be changed. If so, at block 860 the new correct focal distance and parallax are identified, based on the user data and the data in the content being presented. Otherwise, the process returns to block 840 to display the next plurality of

subframes. In this way, the system continuously adjusts the data being displayed to account for both the vergence and perceived focal distance of the user, as the data changes. In one embodiment, the system performs the differential adjustment of block 830 once when initially setting up the system. Thereafter, the data for both eyes are adjusted uniformly, to maintain continuity of content. Thus, as focal distance changes, the system maintains the needed disparity between the eyes. In one embodiment, this process does not utilize eye tracking. Rather, the known focal distance and vergence data is used.

[0091] Figure 9 is a flowchart of one embodiment of using the binocular display to match vergence, perceived focal distance, and parallax. The process starts at block 910. At block 920, the condition of each eye of the user is identified. This may be done based on data entered by the user, or a medical professional. Alternatively, testing of the user's eyes may be used to produce this data.

[0092] At block 925, the gaze vector of the user's eye is identified. The gaze vector is detected, in one embodiment, using eye tracking. At block 930, the process determines whether either eye needs a diopter adjustment. If so, at block 935, diopters are adjusted. In one embodiment, diopters may be adjusted without moving parts, using the modulation stack and image source. In one embodiment, the diopter adjustment may be a manual adjustment. In one embodiment the distance between the two eye pieces of the binocular display may also be adjusted to conform to the IPD (inter-pupillary distance) of the user.

[0093] At block 940, the process determines whether either eye needs a luminosity or color adjustment. If so, at block 945, the luminosity and/or color is adjusted for the appropriate eye(s).

[0094] At block 950, the process determines whether there is a need for an opia adjustment. Opias are visual disorders which change the user's ability to perceive image data, ranging from myopia to amblyopia and presbyopia, and others. If there is such an adjustment needed, the light level, focus, disparity, color, or other aspect of the display are adjusted, as needed at block 955.

[0095] At block 960, the defined display difference between the user's eyes is stored, in one embodiment. This display difference is maintained as a fixed difference while adjusting through different display categories, in one embodiment. In one embodiment, the difference may change based on the perceived focal distance, for example, utilizing a different adjustment at different perceived focal distances. The display difference, defines the differential between the user's two eyes.

[0096] The process then ends, at block 970. In one embodiment, these settings are maintained and utilized for the user, adjusting the display data in realtime. In one embodiment, some of these adjustments may be precalculated. The adjustments may be made by image source (either altering the light level from light source, or adjusting other aspects via digital correction system), by the modulation stack, or by intermediate optics within the display system. In one embodiment, a combination of these elements may be used to provide the controls contemplated.

[0097] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.