Young Harvill

Priority Claim/Related Applications

This application claims the benefit of and priority under 35 USC 119(e) and 120 to U.S. Provisional Patent Application Serial No. 61/581,389, filed on December 29, 2011 and entitled "System and Method for the Efficient Recording of Large Aperture Wave Fronts of Visible and Near Visible Light", the entirety of which is incorporated herein by reference.

Field

The disclosure relates generally to an imaging system and method and in particular to a system and method for recording wave fronts of light using holography and integral imaging.

Background

There are many approaches to recording wave fronts of light. These approaches can be arbitrarily divided into three classes of recording:

1. Approaches that record the amplitude of a real image on a recording medium, the real image being produced by an optical system. Typically, these processes are called photography.

2. Approaches that record the phase of the wave front of light by recording the amplitude of an interference pattern on a recording medium produced by a signal wave front and a reference wave front. Typically, these processes are called holography.

3. Approaches that record the phase of a wave front of light through the simultaneous recording of light amplitude on a recording medium of a large number of points of view. These processes are often known as integral imaging.

Each of the above approaches has advantages and limitations.

For example, photography produces an image that is easily processed and understood, but is limited to a single point of view and a single focal plane. Technologies for recording photographs are well understood, are widely used, and its recording and computational components have benefited from cost efficiencies from mass production. Unfortunately, photography has poor efficiency in the use of the energy of the input wave front of light. This is due to the fact that the photography process must filter the light spatially using a small aperture for optimal depth of field, and by frequency, to record color. The use of a small aperture also limits the amount of information a photograph may capture due to the diffraction limits of an optical system using a small aperture.

Holography produces an image that is a full recreation of the wave front of light over the aperture of the recording media, with no limitations on viewpoint or focal plane.

Holography is very efficient in its use of the energy of the input wave front of light. The entire incident light from the two wave fronts is recorded. Holograms are recorded using interference, which is matched to the diffraction limits of the wave fronts recorded. Since the recording aperture is the entire recording area, holograms approach the theoretical information and resolution limits of recording visual information. Unfortunately, there are severe constraints limiting when a hologram may be recorded that makes them impractical as a general approach to recording a wide variety of imagery. For example, holograms may only be recorded using light that has spatial and frequency coherence.

Integral imaging can record multiple points of view, and through use of additional computation, is free of focal plane constraints. The use of classic optical systems in integral imaging imposes the same efficiency and information limits as photography, with the additional resource burden of imaging many points of view, often on a single recording device or surface.

Thus, it is desirable to provide an imaging system and method that overcomes the above limitations with the known approaches and it is to this end that the disclosure is directed.

Brief Description of the Drawings

Figures 1 and 2 are each an example of an implementation of a hybrid integral imaging system;

Figures 3A- 3D illustrate more details of a hybrid integral imaging system;

Figure 4A-4B illustrate a method for hybrid integral imaging;

Figure 5 illustrates an embodiment of the holographic multiplexer element for red/green/blue mapping; Figure 6 illustrates an embodiment of the holographic multiplexer element for three dimensional mapping; and

Figure 7 A, 7B, 8 A and 8B illustrate examples of the user interface used to adjust the depth of view or field of view of the system. Detailed Description of One or More Embodiments

The disclosure is particularly applicable to the hybrid integral imaging system and method described below that efficiently records large aperture wave fronts of visible and near visible light and it is in this context that the disclosure will be described. It will be appreciated, however, that the system and method has greater utility. For example, the embodiment described below may be used for visible light, but the hybrid integral imaging system and method may also be used with non- visible light. More generally, the hybrid integral imaging system and method may be used with various types of electromagnetic radiation.

In one implementation, the system and method may be used with electromagnetic radiation that has line spectra that fall within a Visible region and a Near infrared (IR) region of the electromagnetic spectrum. For example, the line spectra (with a range of wavelengths) may include one or more of violet at 380-450 nm, blue at 450-495 nm, green at 495-570 nm, yellow at 570-590 nm, orange at 590-620 nm, red at 620-750 nm and near ir at 2,500 to 750 nm. Figures 1 and 2 are each an example of an implementation of a hybrid integral imaging system 100. In Figure 1, the system may comprise a processing module 102 that is coupled to a hybrid integral imaging unit 104. The processing module 102 may be any typical microprocessor, microcontroller, state machine and the like that is able to process the data that is generated by the hybrid integral imaging unit 104 as described below in more detail. In some embodiments, the processing module 102 and the hybrid integral imaging unit 104 may be both housed in a single enclosure. In the example in Figure 1, the processing module 102 may include storage that allows the data from the hybrid integral imaging unit 104 to be stored and processed. In Figure 2, the processing module 102 may be a computer system 106 or part of the computer system. The computer system may be a personal computer, a server computer, a cloud computing resource, a laptop computer, a smartphone device and the like. In the example in Figure 2, the computer system is a desktop computer system that may have a display 106a, a chassis 106b (that may house the processing unit 102 and a memory or persistent storage unit 106c) and one or more input/output devices, such as a keyboard 106d and a mouse 106e, that allow a user to interact with the computer system. The computer system may also have an interface and circuits 106f that allows the processing unit 102 to communicate, wirelessly or in a wired manner, to a remote device. As shown in Figure 2, the hybrid integral imaging unit 104 may be coupled to the computer system.

In addition to the implementation of the processing module 102 shown in Figures 1-2, the processing module may contain a set of individual microprocessor elements with each microprocessor element being mapped to a region of the image recording element as an input. Each microprocessor element also may be mapped to many portions of memory representing a portion of the processed image as an output in a manner that allows parallel computation of the multiplexed input into a portion of the requested view. Alternatively, the processing module 102 may contain a set of individual microprocessor elements with each

microprocessor element being mapped to several regions of the image recording element or an associated processor, as an input, and to a single portion of the requested view as an output, in a manner that allows parallel computation of the multiplexed input into a portion of the requested view. The details of the processing unit 102 and the hybrid integral imaging unit 104 are provided below. Figures 3A and 3B illustrates more details of a hybrid integral imaging unit 104. As shown in Figure 3A, the hybrid integral imaging unit 104 may include an enclosure 104A that houses the other components on the unit shown in Figure 3B. The hybrid integral imaging has the cost advantages of photography and integral imaging, and removes many of their constraints by replacing the standard optical system of photography, or of integral light field imaging with a holographic multiplexing element 200 that is shown in Figure 3B and described below in more detail.

The holographic multiplexing element 200 provides efficient multiplexing of information in the input light wave front to individual recording devices. More particularly, the holographic multiplexing element 200 may efficiently multiplex the wave front into discrete channels for Color, Viewpoint, and Depth.

The hybrid integral imaging unit 104 may be composed of five components including the holographic multiplexing element 200, one or more image recording elements 202, a support component 204, one or more light ways 206 that are shown in Figure 3. The hybrid integral imaging unit 104 may further comprise the processing module 102 shown in Figures 1-2, a user interface module, such as the display 106a in Figure 2 and other portions of the processing module, and the enclosure 104A shown in Figure 3A.

Holographic Multiplexing Element 200

The holographic multiplexing element 200 may be a volume reflection hologram recorded in the Lipmann-Bragg regime. This type of hologram contains recorded interference fringes slicing through a volume of the recording medium at 1/2 wavelength layers. This geometry reflects and filters light of a specific wavelength. Recording materials are available that are very efficient in reconstruction, so it is possible to record in this hologram many virtual off-axis concave parabolic (or spheroid) focusing sub-elements, such as mirrors, that efficiently reflect and focus a narrow wavelength of light. Each virtual mirror sub-element may be designed to focus an image of a particular frequency band from a specific point of view, with a specific virtual aperture, and a specific depth of field to an individual recording element. The sub-elements may also be mapped to portions of an image recording element. Alternatively, each focusing sub-element may be paired with an image recording element.

In one implementation, the holographic multiplexing element 200 may be implemented as a set of discrete volume reflection hologram elements. Each holographic element may filter the incoming wave front of light by color spectra using constructive interference in the Lippman/Bragg domain. Each holographic element may filter based on view point and/or depth by reconstructing a real image of a portion of the incoming wave front of light using constructive interference in the Fresnel domain.

To manufacture the holographic multiplexing element 200, each discrete volume hologram may be written to a holographic recording substrate (such as, for example, dichromated gelatin on optical glass) sequentially using an automated process than uses specific spectra of coherent light as an object wave and a reference wave to describe the mapping the aperture of each discrete holographic element to an input point in the incoming wave front and an output point on the image recording element.

The holographic multiplexing element 200 may additionally be implemented as a set of discrete relief transmission holograms that have a specific blazed coating applied. Each discrete holographic element may filter based on view point and depth by reconstructing a real image of a portion of the incoming wave front of light using constructive interference in the Fresnel domain. Each holographic element may filter the color spectra of the incoming wave front of light by constructive interference of the blazed coating. In the manufacture of this embodiment of the holographic multiplexing element 200, each discrete relief hologram may be written to a holographic recording substrate (such as, for example, a photo sensitive plastic applied to optical glass) sequentially using an automated process than uses specific spectra of coherent light as an object wave and a reference wave to describe the inverse mapping the aperture of each discrete holographic element to an input point in the incoming wave front and an output point on the image recording element. Each recording element is coated with a series of films through a mask to provide spectral filtering. The resulting holographic multiplexing element may be then mounted in a manner to use its conjugate (or psuedoscopic) reconstruction geometry. The above methods may also be used to manufacture a master holographic multiplexing element that is then replicated using methods known to those that are skilled in the art.

The Holographic Multiplexing Element may have data encoded in it that may be read by a processing module 102. For example, it is possible to encode, on an unused portion of the holographic multiplexing element, binary data using a known gray code to produce a pattern of lines or dots. In this manner, digital data may be associated with a specific holographic multiplexing element. The digital data may contain information of how the holographic multiplexing element maps the incoming wave front of light to portions of the one or more image recording elements. For example, the digital data may indicate that a first element of the holographic multiplexing element maps to a first image recording element at a particular location.

The characteristics of the holographic multiplexing element 200 may be changed to optimize the Imaging Device for a specific use. For instance, a holographic multiplexing element 200 may be manufactured that provides discrete elements that provide a large variety of spectral filters, and only a few spatial points and this would be useful for spectroscopy. Alternatively, the holographic multiplexing element 200 may be manufactured that encodes only red, green and blue spectral filters and a variety of spatial viewpoints arranged on the horizontal plane which would be useful for three dimensional imaging. In the example in Figure 3B, the holographic multiplexing element 200 may be a single element, but the holographic multiplexing element 200 may also be implemented as one or more separate elements when combined together have the same effect as the single holographic

multiplexing element 200.

Figure 5 illustrates an embodiment of the holographic multiplexer element 200 and image recording element 202 for red/green/blue mapping. As shown in Figure 5, the holographic multiplexer element 200 may have one or more holographic sub-elements 200i and those holographic sub-elements 200i may include a holographic element with red constructive filtering 200 ₂ that reflect red wavelength electromagnetic radiation from an input spatial point 500, a holographic element with blue constructive filtering 2ΟΟ ₃ that reflect blue wavelength electromagnetic radiation from an input spatial point 500and a holographic element with green constructive filtering 200 ₄ that reflect green wavelength electromagnetic radiation from an input spatial point 500. As shown in Figure 5, the red, green and blue wavelength electromagnetic radiation are reflected to the one or more image recording elements 202 (a single image recording element is shown in the example in Figure 5, but the system is not limited to only the single image recording element.) In the embodiment in Figure 5, the image recording element 202 may have one or more output sub-elements to which each of the holographic elements direct the different wavelengths of electromagnetic radiation. For example, the image recording element may have a recording element blue output 202], a recording element green output 202 ₂ and a recording element red output 202 ₃ . Thus, using this embodiment the red, green and blue wavelength electromagnetic radiation are mapped to different portions on the image recording element.

Figure 6 illustrates an embodiment of the holographic multiplexer element 200 and image recording element 202 for three dimensional mapping. As shown in Figure 6, the holographic multiplexer element 200 may have one or more holographic sub-elements 200i and those holographic sub-elements 200i may include one or more holographic elements with a particular wavelength of constructive filtering 200 ₄ , such as green in this example, that reflect electromagnetic radiation, such as green wavelength electromagnetic radiation in this example, from one or more input spatial points 500, 502, 504 so that three dimension mapping occurs due to the one or more input spatial points. As shown in Figure 6, the single wavelength electromagnetic radiation, such as green in the example in Figure 6, is reflected to the one or more image recording elements 202 (a single image recording element is shown in the example in Figure 5, but the system is not limited to only the single image recording element.) In the embodiment in Figure 6, the image recording element 202 may have one or more output sub-elements to which each of the holographic elements directs the

electromagnetic radiation from the different input spatial points. For example, the image recording element may have a recording element spatial point one output 202 ₄ , a recording element spatial point two output 202s and a recording element spatial point three output 202 ₆ . Thus, using this embodiment one wavelength of electromagnetic radiation is mapped, at three different input spatial points, to different portions on the image recording element to generate a three dimensional mapping.

Image Recording Elements 202

The holographic multiplexing element 200 multiplexes the input wave front of light onto the one or more image recording elements 202 that are positioned at predetermined locations on the support component 204, such as the positioning shown in Figure 3B. Each image recording element may be an image recording device that can receive the

electromagnetic radiation and store/convert the received electromagnetic radiation into electrical signals. In one implementation, each image recording device may be, but is not limited to, a CCD imaging device that has a moderate number of pixel elements that match an inexpensive high- yield production process. Since overall system resolution may be based on an aggregate of all of the image recording elements, the image recording devices may be chosen that have the best cost-per-pixel, rather than choosing high resolution devices. In addition, since the holographic multiplexing element is focusing light on each of the image recording elements from a very large aperture, low cost devices that have a smaller sensing area may be chosen. For example, a theoretical limit of the sensor size is 2x the wavelength that the sensor must record.

Each image recording element has a mechanism, such as a wire or cable, to transfer the electrical signals to the processing module 102.

Support Component 204

The support component 204 supports the holographic multiplexing element 200 and the one or more image recording elements 202 and matches the off- axis multiplexing relationship between the holographic multiplexing element 200 and the one or more image recording elements 202. Specifically, as shown in Figure 3B, if there is an axis perpendicular to the holographic multiplexing element 200, the one or more image recording elements 202 are off-axis. The support component may be made of a material, such as plastic or metal, so that it maintains the geometric relationship within design tolerances over the range of temperature and humidity required. Alterantively, the support component 204 may be a carbon fiber composite instead of aluminum so that a camera with a large aperture may remain calibrated (provide a sharp and true image) over a wide range of temperatures. An example of the geometric relationship between the holographic multiplexing element 200 and the one or more image recording elements 202 is shown in Figure 3B, but other geometric relationships may also be used.

Light Ways 206

The light ways 206 may each be a piece of material, such as plastic or metal with certain properties to suppress ambient or reflected electromagnetic energy, that is connected to the support component 204, such as in the configuration shown in Figure 3B. Other configurations of the light ways 206 may also be used. A first set of light ways 206i may be positioned at an input of the hybrid integral imaging unit 104 to allow a signal wave front of light 208 (examples of which are shown in Figure 3C with dotted lines) to strike the holographic multiplexing element 200. A second set of light ways 206 ₂ , such as two for each image recording element 202 in one implementation, allow a set of multiplexed light components 210 (examples of which are shown in Figure 3D with dotted lines) to strike the image recording elements 202. Both of the sets of light ways 206 suppress all other ambient and reflected light.

For simple mapping geometries, a light way may be constructed using a honeycomb shape, where the proportion of the width to depth of a honeycomb element is chosen to match the camera angle of view, and the width of the honeycomb is chosen as a proportion to the thickness of the honeycomb wall to optimize throughput of the electromagnetic radiation wave front and the reduction of shadows or artifacts from the light ways. For more complex light ways, the path of each input and output ray for a given discrete holographic sub-element may be laser cut into a lightweight, rigid foam block.

Processing Module 102

The processing module 102 may be a processing unit or a processing unit based device (such as shown in Figures 1 and 2) that may execute a piece of software (or a plurality of lines of computer code) to perform various processes associated with the hybrid integral imaging unit. A method for hybrid integral imaging 400 using those processes are disclosed in more detail in Figures 4 A and 4B. The processes receive the data from each of the image recording elements that are reflected off of the holographic multiplexing element and maps that data to the input wave front of light. The processing module may thus control the image recording elements (402.) During the control of the one or more image recording element, the processing module may read the data encoded within the holographic multiplexing element that, among other data, contains data about how the particular holographic multiplexing element has been configured so that the processing module is able to process the data from the one or more image recording element, map the data to and reconstruct an input wave front of light. During the control, the processing module may also start and/or stop a recording for a specific time so that the one or more image recording elements record the wave front data (404.)

The processing module may then acquire the data from the image recording elements

(406) using specific methods and attributes. For example, the image data may be acquired from several optical elements containing similar viewpoints, and using multiple processing elements mapped to portions of the image recording elements, combine these viewpoints to comprise a sharper, higher resolution image data. In addition, input image data from a variety of view may be analyzed, and converted into a three dimensional representation such as a

Binary Space Partition or Octree, representing component watersheds of the wave front. The image data may also be compressed, using known methods such as entropy encoding based on differences in one or more spatial or color spaces.

The processing module may then store the acquired data from the image recording elements and/or load already stored data from the image recording elements (408.) The processing module (such as the computer system in Figure 2 described above) may have the interface and circuits 106f so that the processing module can communicate the acquired the image recording element data to another device over a network. Thus, the processing module determines whether or not to send the image recording data to another device (410) and then is able to send the data (412.) Then, the processing module (or the other device that has similar resources) may reconstruct the acquired data into images with specific methods and attributes (414.)

During the reconstruction process, for example, the processing module may use multiple processors to decode information stored in a 3d form, and transform the input data to represent a given point of view, synthetic aperture, or color space. The processing module may also construct an image by directly assembling images from a specific point of view encoded in the holographic multiplexing element, and mapped to the image recording element in a manner that is easily tiled and reconstructed. The processing module may then refine the image such a reconstructed image using edges or high frequencies found in similar or adjacent views.

The processing module (or the other device) may also adjust its reconstruction process based on configuration options from the user interface component or from another device over a network.

User Interface Component/Module

The User Interface component has device/mechanism/process for a user to set attributes and methods and communicate them to the processing module 102. For instance, one attribute may be a specific method of how the processing Module will acquire data from the Recording Elements. Additionally, one of these attributes may be when the processing module starts and/or stops recording of the wave front of light from the image recording elements. The attributes may also, for instance, include a method to display a reconstruction of the acquired data as an image. Additionally, one of the attributes that are set by the user and communicated to the processing module may be what point of view to reconstruct, at what depth. Figures 7A and 7B show a change of view left and change of view right selection by the user from the user interface while Figures 8A and 8B illustrates a depth of field far and a depth of field near selection by the user from the user interface.

Enclosure

The enclosure holds all of the other system components in a manner that protects them from light, dust and/or physical damage. It does this in a way that is pleasing for a user to hold, to touch and to see. The enclosure may be made of a carbon composite material which may be rigid and light and may be cast with a texture on the surface that may be decorative and may provide an easy to hold surface.

Implementation of the Hybrid Integral Imaging Geometry

The hybrid integral imaging geometry may be designed to maximize the efficiency of the holographic multiplexing element in the Lippman-Bragg regime while reducing internal reflections from the various system elements. One example of a configuration is to place the Holographic Multiplexing Element plane to the incoming wave front of light at the back of the device, and to place the Recording Elements off axis under the device aperture as shown in Figures 3A-3D. The hybrid integral imaging system may use other geometries and the disclosure is not limited to the geometry shown in Figures 3A-3D.

The Holographic Multiplexing Element may be removed and replaced from the device as needed. Once the hybrid integral imaging geometry is determined, the geometry between the Holographic Multiplexing Element and the image recording elements remains fixed. However, the form of multiplexing may be changed, with a replacement of the holographic multiplexing element. For instance, if the Hybrid Integral Imaging System had 64 Image Recording Elements of 3200x2400, possible multiplexing configurations (requiring a holographic multiplexing element change) could be:

1. 530 (640x480) points of view at 3 colors with an aggregate 491 Mega Pixels.

2. 2130 (320x240) points of view at 3 colors.

3. 16 (3200x2400) points of view, 12 depths, Monochrome

4. 1 (3200x2400) view. 8 depths, 8 color samples for full spectrum imaging.

In the above examples, a same number of image recording element pixels (or sensing areas) are used for each configuration and the change needed to change from a large aperture 3d light field camera represented by example (2) and a high resolution spectral depth imaging camera represented by example (4) is the holographic multiplexing element 200. In the case of example (2) the holographic multiplexing element 200 may be manufactured to map many input wave front spatial points into small sections of the image recording element. In the case of example [4], more frequency filtering, and fewer input spatial points are used to map the input wave front of light to large areas of image recording element. An example of the geometry of an implementation of the system is shown in Figs 3A-3D.

The hybrid integral imaging system receives a wave front of light and uses the constructive interference of light, distributes the wave front of light into a multiplicity of amplitude images (by the holographic multiplexing element) focused onto the one or more image recording elements. In the system, each focused image recording element is distributed by wavelength, position of view, and/or depth of focus.

While the foregoing has been with reference to a particular embodiment of the invention, it will be appreciated by those skilled in the art that changes in this embodiment may be made without departing from the principles and spirit of the disclosure, the scope of which is defined by the appended claims.