CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No.

62/057,830 filed on September 30, 2014, which is hereby incorporated by reference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH &

DEVELOPMENT

[0002] This invention was made with government support under grants DPI EB016986 and R01CA186567, both awarded by the U.S. National Institutes of Health. The U.S.

government may have certain rights in this invention.

FIELD OF THE INVENTION

[0003] The present invention relates to systems and methods of compressed-sensing ultrafast photography (CUP). In particular, the present invention relates to about 100 billion frames per second dynamic imaging of non-repetitive events.

BACKGROUND

[0004] Capturing transient scenes at a high imaging speed has been pursued by photographers for centuries, tracing back to Muybridge's 1878 recording of a horse in motion and Mach's 1887 photography of a supersonic bullet. However, not until the late 20 ^th century were breakthroughs achieved in demonstrating ultra-high speed imaging (>100 thousand, or 10 ⁵ , frames per second). In particular, the introduction of electronic imaging sensors, such as the charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS), revolutionized high-speed photography, enabling acquisition rates up to ten million (10 ⁷ ) frames per second. Despite the widespread impact of these sensors, further increasing frame rates of imaging systems using CCD or CMOS is fundamentally limited by their on-chip storage and electronic readout speed.

[0005] 3D imaging techniques have been used in many applications, including remote sensing, biology, and entertainment, as well as in safety and national security applications such as biometrics, under- vehicle inspection, and battlefield evaluation. The suitability of 3D imaging for these diverse applications is enhanced if the 3D images may be captured and transmitted to users in a secure and fast manner. Photons scattered from the object to be imaged carry a variety of tags, such as emittance angle and time-of-flight (ToF), which convey 3D surface information used in various 3D imaging methods, including structured-illumination, holography, streak imaging, integral imaging, multiple camera or multiple single-pixel detector photogrammetry, and time of flight (ToF) detection. Holography is one 3D imaging method that enables an intrinsic holography-based encryption of the 3D images that makes use of a pseudo-random phase or amplitude mask used to obtain the 3D image as a decryption key for reconstructing images of the 3D object. However, the holographic imaging method is sensitive to motion of the object due to relatively long exposure times, which may degrade image quality.

[0006] ToF is another 3D imaging method that makes use of the ToF of a light signal from the object to a detector to quantify the distances of various regions of the object for use in reconstructing a 3D image of the object. Some ToF imaging systems acquire 3D images using multiple ToF measurements, which limits suitability of these systems for imaging fast-moving 3D objects. In other ToF imaging systems, single-shot ToF detection has been incorporated to mitigate motion distortion in 3D images. However, existing single-shot ToF 3D imaging systems are characterized by relatively low imaging speeds of up to 30 Hz and relatively low image resolution on the order of about 10 cm. In addition, existing ToF 3D imaging systems lack the intrinsic encryption capability associated with holography.

SUMMARY

[0007] Provided herein is a compressed-sensing ultrafast photography system to obtain a series of final recorded images of an object. The system may include a spatial encoding module to receive a first series of object images and to produce a second series of spatially encoded images, each spatially encoded image of the second series comprising one object image of the first series superimposed with a pseudo-random binary spatial pattern and a temporal encoding module operatively coupled to the spatial encoding module, the temporal encoding module configured to receive an entire field of view of each spatially encoded image of the second series, to deflect each spatially encoded image by a temporal deflection distance proportional to time-of-arrival, and to record each deflected image as a third series of spatially/temporally encoded images, each spatially/temporally encoded image of the third series comprising an object image superimposed with a pseudo-random binary spatial pattern and deflected by the temporal deflection distance.

[0008] Further provided herein is a method of obtaining a series of final recorded images of an object using a compressed-sensing ultrafast photography system at a rate of up to one billion frames per second. The method may include collecting a first series of object images, superimposing a pseudo-random binary spatial pattern onto each object image of the first series to produce a second series of spatially encoded images, deflecting each spatially encoded image of the second series by a temporal deflection distance proportional to a time-of-arrival of each spatially encoded image, recording each deflected spatially encoded image as a third series of spatially/temporally encoded images, and reconstructing a fourth series of final object images by processing each spatially/temporally encoded image of the third series according to an image reconstruction algorithm.

[0009] Additionally provided herein is a compressed-sensing ultrafast photography system to obtain a series of final recorded images of an object. The system may include an optical module including a camera lens operatively coupled to a beam splitter, a beam splitter operatively coupled to a temporal encoding module and operatively coupled to a tube lens, the tube lens operatively coupled to an objective, the objective operatively coupled to a spatial encoding module, the spatial encoding module configured to receive the first series of object images from the objective and to transfer a second series of spatially encoded images to the objective, each spatially encoded image of the second series comprising one object image of the first series superimposed with a pseudo-random binary spatial pattern, and a temporal encoding module operatively coupled to the beam splitter. The temporal encoding module may be configured to receive an entire field of view of each spatially encoded image of the second series via the objective, the tube lens, and the beam splitter, to deflect each spatially encoded image by a temporal deflection distance proportional to time-of-arrival, and to record each deflected image as a third series of spatially/temporally encoded images, each

spatially/temporally encoded image of the third series comprising an object image superimposed with a pseudo-random binary spatial pattern and deflected by the temporal deflection distance.

[0010] Also provided is a time of flight compressed-sensing ultrafast 3D imaging system to obtain a series of 3D images of an outer surface of an object. The system includes: a spatial encoding module to receive a first series of object images and to produce a second series of spatially encoded images, each spatially encoded image of the second series including one object image of the first series superimposed with a pseudo-random binary spatial pattern; a temporal encoding module operatively coupled to the spatial encoding module, the temporal encoding module configured to receive an entire field of view of each spatially encoded image of the second series, to deflect each spatially encoded image by a temporal deflection distance proportional to time-of-arrival, and to record each deflected image as a third series of spatially/temporally encoded images, each spatially/temporally encoded image of the third series including an object image superimposed with a pseudo-random binary spatial pattern and deflected by the temporal deflection distance; an illumination source including a pulsed laser operatively coupled to the temporal encoding module. The illumination source delivers a laser pulse to illuminate the object and records a pulse delivery time, and an elapsed time between the pulse delivery time and the time of arrival is the round-trip time of flight. The system further includes a reference camera to record a 2D reference image of the object, in which the reference image is used as an intensity mask to enhance 3D image quality.

BRIEF DESCRIPTION OF THE DRAWINGS

[001 1] The following figures illustrate various aspects of the disclosure.

[0012] FIG. 1 is a schematic diagram illustrating the elements of a compressed ultrafast photography (CUP) system according to one aspect.

[0013] FIG. 2A is a schematic diagram illustrating the imaging of a stripe pattern using a CUP system according to one aspect.

[0014] FIG. 2B is an image of a reconstructed datacube of the striped pattern and a representative frame from the reconstructed datacube obtained using the CUP system illustrated in FIG. 2A.

[0015] FIG. 2C is a reference image obtained using a CUP system according to one aspect without introducing temporal dispersion.

[0016] FIG. 2D is a projected vertical stripe image obtained using a CUP system according to one aspect and calculated by summing over x, y, and t datacube voxels along a temporal axis.

[0017] FIG. 2E is a projected horizontal stripe image obtained using a CUP system according to one aspect and calculated by summing over x, y, and t datacube voxels along a temporal axis.

[0018] FIG. 2F is a graph comparing the average light fluence distributions along the x axis from FIG. 2C (Reference), along the x axis from FIG. 2D (CUP (x axis)), and along the y axis from FIG. 2E (CUP (y axis)).

[0019] FIG. 2G is a graph summarizing the spatial frequency responses of a CUP system according to one aspect for five different orientations of a stripe pattern.

[0020] FIG. 3A is a series of images of laser pulse reflection obtained using obtained using a CUP system according to one aspect. [0021] FIG. 3B is a series of images of laser pulse refraction obtained using a CUP system according to one aspect.

[0022] FIG. 3C is a series of images of two laser pulses propagating in air and in resin obtained using a CUP system according to one aspect.

[0023] FIG. 3D is a graph comparing the change in position with time of a laser pulse in air and in resin measured from the images of FIG. 3C.

[0024] FIG. 4A is a photographic image of a stripe pattern with a constant period of 12mm.

[0025] FIGS. 4B is a series of images of an optical wavefront sweeping across the stripe pattern depicted in FIG. 4A that was obtained using a CUP system according to one aspect.

[0026] FIGS. 4C is a schematic diagram illustrating the intersection of optical wavefronts with the pattern depicted in FIG. 4A.

[0027] FIG. 5A is a schematic diagram illustrating the elements of a multicolor compressed-sensing ultrafast photography (Multicolor-CUP) system according to one aspect.

[0028] FIG. 5B is a series of images of a pulsed-laser-pumped fluorescence emission process obtained using the multicolor compressed-sensing ultrafast photography (Multicolor- CUP) system illustrated in FIG. 5A.

[0029] FIG. 5C is a graph summarizing the time-lapse pump laser and fluorescence emission intensities within the dashed box shown in FIG. 5B.

[0030] FIG. 6A is a graph of an event function describing the pulsed laser fluorescence excitation from a simulated temporal response of a pulsed-laser-pumped fluorescence emission.

[0031] FIG. 6B is a graph of an event function describing the fluorescence emission from a simulated temporal responses of a pulsed-laser-pumped fluorescence emission.

[0032] FIG. 6C is a graph of a measured temporal point-spread- function (PSF).

[0033] FIG. 6D is a graph illustrating the simulated temporal responses of the two event functions shown in FIG. 6A and FIG. 6B after being convolved with the temporal PSF shown in FIG. 6C.

[0034] FIG. 7 is a schematic diagram illustrating a CUP image formation model according to one aspect.

[0035] FIG. 8 is an image of a temporally undispersed CCD image of a mask used to encode the uniformly illuminated field with a pseudo-random binary pattern according to the CUP imaging method according to one aspect. [0036] FIG. 9 is a schematic diagram illustrating a time of flight- compressed ultrafast photography (ToF-CUP) 3D imaging system according to one aspect.

[0037] FIG. 10A is a schematic diagram of a target body positioned beneath a camera lens of a ToF-CUP system.

[0038] FIG. 10B is a graph of the reconstructed x, y, t _XoF datacube representing the backscattered laser pulse intensity from the fins with different depths of the target body illustrated in FIG. 10A.

[0039] FIG. IOC is a series of representative x-y frames obtained by a ToF-CUP system at t _ToF = 120, 200, and 280 ps.

[0040] FIG. 11A is a depth-encoded ToF-CUP image of the stationary letters "W" and "U" with a depth separation of 40 mm.

[0041] FIG. 1 IB is a depth-encoded ToF-CUP image of a wooden mannequin.

[0042] FIG. 11C is a depth-encoded ToF-CUP image of a human hand.

[0043] FIG. 12A is a graph summarizing the cross-correlation coefficients between an image decrypted using the correct decryption key and images decrypted using 50 brute force attacks with incorrect random binary masks.

[0044] FIG. 12B is a graph illustrating a 3D datacube of the letters "W" and "U" (see FIG. 1 1A) decrypted using the correct decryption key.

[0045] FIG. 12C is a graph illustrating a 3D datacube of the letters "W" and "U" (see FIG. 11A) decrypted using an invalid correct decryption key from one of the brute force attacks presented in FIG. 12A.

[0046] FIG. 12D is a graph summarizing the cross-correlation coefficients between a reconstructed image decrypted using a correct decryption key and a series of images reconstructed using a subset of the correct decryption key with a different horizontal shifts to the left (negative pixel shift values) and to the right (positive pixel shifts).

[0047] FIG. 12E is a graph illustrating a 3D datacube of the letters "W" and "U" (see FIG. 1 1A) decrypted using the correct decryption key shifted horizontally by a single encoded pixel.

[0048] FIG. 13 A is a schematic illustration of a target body that includes two rotating balls.

[0049] FIG. 13B is a series of representative depth-encoded 3D images obtained at different time points in the motion of the two balls showing the relative depth positions of the two balls. [0050] FIG. 14A is a series of representative depth-encoded 3D images of a live comet goldfish swimming in a tank obtained at different time points in the motion of the goldfish.

[0051] FIG. 14B is graph summarizing changes in 3D position of a goldfish swimming in a tank obtained by the analysis of 3D images obtained using the ToF-CUP system according to one aspect.

[0052] FIG. 15A is a schematic diagram illustrating a moving target in a scattering medium.

[0053] FIG. 15B is a series of images of a moving object in a scattering medium obtained using a ToF CUP 3D imaging system according to one aspect.

[0054] FIG. 15C is a graph summarizing the normalized intensity profiles for a cross section of a target airplane wing at different scattering conditions.

[0055] FIG. 15D is a series of images of a moving object in a scattering medium obtained using a ToF CUP 3D imaging system according to one aspect.

[0056] FIG. 15E is a series of images of a moving object in a scattering medium obtained using a ToF CUP 3D imaging system according to one aspect.

[0057] While multiple embodiments are disclosed, still other embodiments of the present disclosure will become apparent to those skilled in the art from the following detailed description, which shows and describes illustrative embodiments of the disclosure. As will be realized, the invention is capable of modifications in various aspects, all without departing from the spirit and scope of the present disclosure. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive.

DETAILED DESCRIPTION

[0058] Provided herein are systems and methods for compressed-sensing ultrafast photography (CUP) for capturing images at up to 100 billion (10 ¹¹ ) frames per second. CUP overcomes the shortcomings of previous existing ultrafast imaging techniques by measuring two spatial coordinates (x, y) as well as time (t) with a single camera snapshot, thereby allowing observation of transient events occurring on a time scale down to tens of picoseconds. In an aspect, CUP may be used to visualize at least four fundamental physical phenomena using single laser pulses: laser pulse reflection, laser pulse refraction, photon racing in two media, and faster- than-light travel of non-information. Moreover, CUP's functionality may be expanded in reproducing colors of different wavelengths λ, thereby enabling single-shot four-dimensional (4D) (x, y, λ, t) measurements of a pulsed-laser-pumped fluorescence emission process with unprecedented temporal resolution. In addition, another aspect of the CUP method, time of flight CUP (ToF-CUP), may obtain the time-of-flight of pulsed light scattered by an object in order to reconstruct a volumetric image of the object from a single snapshot.

Compressed-Sensing Ultrafast Photography System

a. Configuration

[0059] Provided herein is a compressed-sensing ultrafast photography system to obtain a series of recorded images of an object. FIG. 1 is a schematic diagram of a CUP system 1000 in one aspect. Referring to FIG. 1, the CUP system 1000 may include a spatial encoding module 100 and a temporal encoding module 200 operatively coupled to the spatial encoding module 100. The system 1000 may further include a spectral separation module 300 (not illustrated) operatively coupled to the spatial encoding module 100 and the temporal encoding module 200.

[0060] In an aspect, the spatial encoding module 100 receives a first series of object images and produces a second series of spatially encoded images. Each of the spatially encoded images of the second series includes an object image of the first series superimposed with a pseudo-random binary spatial pattern. The temporal encoding module 200 may receive an entire field of view of each spatially encoded image of the second series, deflect each spatially encoded image of the second series by a temporal deflection distance proportional to the time- of-arrival of each portion of each spatially encoded image of the second series. The temporal encoding module 200 also records each deflected spatially encoded image as a third series of spatially and temporally encoded images. Each spatially and temporally encoded image of the third series may include an object image superimposed with a pseudo-random binary spatial pattern and deflected by the temporal deflection distance.

[0061] In an aspect, the spectral separation module 300 deflects each spatially encoded image of the second series by a spectral deflection distance. In one aspect, the spectral deflection distance of the spectral encoding module 300 may be oriented perpendicular to the temporal deflection distance of the temporal encoding module 200. In an aspect, the spectral separation module 300 may receive the second series of spatially encoded images from the spatial encoding module. In another aspect, the spectral separation module 300 deflects a first spectral portion of each spatially encoded image including a first wavelength and a second spectral portion of each spatially encoded image including a second wavelength by a first and second spectral deflection distance proportional to the first and second wavelengths, respectively. In yet another aspect, the spectral separation module may produce a fourth series of spatially/spectrally encoded images, each spatially/spectrally encoded image comprising an object image superimposed with a pseudo-random binary spatial pattern and with the first and second spectral portions deflected by spectral deflection distances. In another aspect, the spectral separation module 300 may deflect more than 2 spectral portions corresponding to more than 2 different wavelengths. In various other aspects, the spectral separation module 300 may deflect up to 3 spectral portions corresponding to 3 different wavelengths, up to 4 spectral portions corresponding to 4 different wavelengths, up to 5 spectral portions corresponding to 5 different wavelengths, up to 6 spectral portions corresponding to 6 different wavelengths, up to 7 spectral portions corresponding to 7 different wavelengths, up to 8 spectral portions corresponding to 8 different wavelengths, up to 9 spectral portions corresponding to 9 different wavelengths, and up to 10 spectral portions corresponding to 10 different wavelengths.

[0062] FIG. 5A is a schematic diagram of a spectral separation module 300 in one aspect. Referring to FIG. 5 A, the spectral separation module 300 may include a dichroic filter 302 mounted on a mirror 304 at a tilt angle 314. In this aspect, the first spectral portion 306 of each spatially encoded image including the first wavelength reflects off of the dichroic filter 302 at a first angle 310 and the second spectral portion 308 of each spatially encoded image including the second wavelength passes through the dichroic filter 302 and reflects off of the mirror at a second angle 312 comprising the combined first angle 310 and tilt angle 314.

[0063] Referring again to FIG. 1, the spatial encoding module 100 may include a digital micromirror device (DMD) 102. The DMD 102 may include an array of micromirrors, where each micromirror may be configured to reflect or absorb a portion of the object image according to the pseudo-random binary pattern.

[0064] In various aspects, the temporal encoding module 200 enables temporal shearing of the spatially encoded images and spatiotemporal integration to produce the spatially and temporally encoded images of the third series of images to be analyzed according to the CUP image reconstruction methods described herein below. In various aspects, the temporal encoding module 200 includes any camera capable of performing the temporal shearing and

spatiotemporal integration used to form a single spatially and temporally encoded image to be reconstructed according to the CUP reconstruction method described herein. In one aspect, the camera's exposure time spans the entire data acquisition process. During the exposure, images recorded from the previous time points are shifted in one spatial dimension and mixed with images recorded at following time points. All these temporally-sheared images are recorded in a single snapshot as the camera output. Non-limiting examples of camera types suitable for use as a temporal encoding module 200 includes a streak cameras, time-delay-and-integration (TDI) cameras, frame transfer CCD cameras including various types of sCMOS, ICCD, and EMCCD cameras that employ frame transfer CCD sensors.

[0065] Referring again to FIG. 1, the temporal encoding module 200 may include a streak camera 202, a 2D detector array 204, and combinations thereof in one aspect. The 2D detector array 204 may include, but is not limited to a CCD, CMOS, or any other detector array capable of capturing the encoded 3D scene. In an aspect, the entrance slit 206 of the streak camera 202 may be fully open. The temporal deflection distance may be proportional to the time-of-arrival and a sweep voltage 208 triggered within the streak camera 202. In one aspect, a CCD may be coupled to a streak camera 202 to form the temporal encoding module 200, such that the streak camera 202 performs a shearing operation in the temporal domain and the encoded 3D scene is measured by the CCD.

[0066] As used herein, the term "streak camera" refers to an ultrafast photo-detection system that transforms the temporal profile of a light signal into a spatial profile by shearing photoelectrons perpendicular to their direction of travel with a time-varying voltage. When used in conjunction with a narrow entrance slit, a typical streak camera is a one-dimensional (ID) imaging device. The narrow entrance slit, which ranges from about 10 - 50 μιη in width, limits the imaging field of view (FOV) to a line. To achieve two-dimensional (2D) imaging with the narrow slit, additional mechanical or optical scanning may be incorporated along the other spatial axis. Although this paradigm is capable of providing a frame rate fast enough to catch photons in motion, the event itself must be repetitive, following exactly the same spatial- temporal pattern while the entrance slit of a streak camera scans across the entire FOV. In cases where the physical observations are either difficult or impossible to repeat, such as optical rogue waves, a nuclear explosion, or gravitational collapse in a supernova, this 2D streak imaging method is inapplicable.

[0067] Referring again to FIG. 1, 2D dynamic imaging is enabled using the streak camera 202 without employing any mechanical or optical scanning mechanism with a single exposure by fully opening the entrance slit 206 to receive a 2D image. In various aspects, the exposure time of the streak camera 202 outfitted with a fully-opened entrance slit 206 spans the time course of entire events, thereby obviating the need to observe multiple events as described previously in connection with the streak camera 202 with narrow entrance slit 206. In various aspect, the spatial encoding of the images performed by the spatial encoding module 100 enables the streak camera 202 to receive 2D images with minimal loss of spatial information. [0068] Referring again to FIG. 1, the system 1000 may further include an optical module 400 to direct the first series of object images to the spatial encoding module 100 and to direct the second series of spatially encoded images to the temporal encoding module 200. The optical module 400 may include, but is not limited to a camera lens 402, a beam splitter 404, a tube lens 406, an objective 408, and combinations thereof. In an aspect, the optical module 400 includes the camera lens 402 operatively coupled to the beam splitter 404, the tube lens 406 coupled to the beam splitter 404, and an objective 408 operatively coupled to the tube lens 406. In this aspect, the camera lens 402 receives the first series of object images, the objective 408 is operatively coupled to the spatial encoding module 100 to deliver the first series of object images, and the beam splitter 404 is operatively coupled to the temporal encoding module 200 to deliver the second series of spatially encoded images via the objective 408 and tube lens 406.

[0069] In an aspect, the system 1000 may further include a microscope not illustrated) operatively coupled to the spatial encoding module 102. The first series of object images may include images of microscopic objects obtained by the microscope. In another aspect, the system 1000 may further include a telescope (not illustrated) operatively coupled to the spatial encoding module 100. In this other aspect, the first series of object images comprise images of distant objects obtained by the telescope.

[0070] Referring back to FIG. 1, the object 500 may first be imaged by a camera lens 402. In an aspect, the camera lens 402 may have a focal length (F.L.) of about 75 mm. The intermediate image may then be passed to a DMD 102 by a 4-f imaging system including a tube lens 406 (F.L. = about 150 mm) and a microscope objective 408 (F.L. = about 45 mm, NA = about 0.16). To encode the input image, a pseudo-random binary pattern may be generated and displayed on the DMD 102, with a single pixel size of about 21.6 μιη x 21.6 μιη (3 x3 binning). Since the DMD's resolution pixel size (7.2 μιη ^χ 7.2 μιη) may be much larger than the light wavelength, the diffraction angle may be small (~ 4°). With a collecting objective 408 of an N.A. = 0.16, the throughput loss caused by DMD's diffraction may be negligible.

[0071] The light reflected from the DMD 102 may be collected by the same microscope objective 408 and tube lens 406, reflected by a beam splitter 404, and imaged onto the entrance slit 206 of a streak camera 202. To allow 2D imaging, this entrance slit 206 may be opened to its maximal width (about 5 mm). Inside the streak camera 202, a sweeping voltage 208 may be applied along the y" axis, deflecting the encoded images towards different y" locations according to their times of arrival. The final temporally dispersed image may be captured by a CCD 204 within a single exposure. In an aspect, the CCD 204 may have 512x672 pixels;

12.9x 12.9 μιη ² pixel size; and 2x2 binning.

b. Effective fleld-of-view measurement

[0072] In CUP, a streak camera temporally disperses the light. The streak camera's entrance slit may be fully opened to a 17 mm x 5 mm rectangle (horizontal x vertical axes). Without temporal dispersion, the image of this entrance slit on the CCD may have an approximate size of 51 Ox 150 pixels. However, because of a small angle between each individual micro-mirror's on-state normal and the DMD's surface normal, the DMD as a whole may need to be tilted horizontally so that the incident light can be exactly retroreflected. With an A of 0.16, the collecting objective's depth of focus thereby may limit the horizontal encoding field of view (FOV) to approximately 150 pixels at the CCD. FIG. 8 shows a temporally undispersed CCD image of the DMD mask, which encodes the uniformly illuminated field with a pseudo-random binary pattern. The effective encoded FOV is approximately 150 x 150 pixels. Note that with temporal dispersion, the image of this entrance slit on the CCD may be stretched along the " axis to approximately 150x500 pixels.

c. Calibration

[0073] To calibrate for operator matrix C, defined herein below, a uniform scene may be used as the input image and a zero sweeping voltage may be applied in the streak camera. The coded pattern on the DMD may therefore be directly imaged onto the CCD without introducing temporal dispersion. A background image may also be captured with all DMD pixels turned on. The illumination intensity non-uniformity may be corrected for by dividing the coded pattern image by the background image pixel by pixel, yielding operator matrix C. Note that because CUP's image reconstruction may be sensitive to mask misalignment, a DMD may be used for better stability rather than premade masks that would require mechanical swapping between system alignment and calibration or data acquisition.

d. Time of Flight CUP 3D imaging System

[0074] In various aspects, the CUP imaging system 1000 may be modified by the addition of an illumination source conduct time of flight CUP (ToF-CUP) 3D imaging. In these various aspects, the CUP system is synchronized with short-pulsed laser illumination to enable dynamic three-dimensional (3D) imaging. By leveraging the time-of-flight (ToF) information of pulsed light backscattered by the object, ToF-CUP can reconstruct a volumetric image from a single camera snapshot. In addition, the approach unites the encryption of depth data with the compressed acquisition of 3D data in a single snapshot measurement, thereby allowing efficient and secure data storage and transmission.

[0075] FIG. 9 is a schematic diagram of a ToF-CUP 3D imaging system 2000 in one aspect. A solid-state pulsed laser (532 nm wavelength, 7 ps pulse duration) is used as the light source 602. The laser beam passes through an engineered diffuser 604 and illuminates a 3D object 606. The object 606 is first imaged by a camera zoom lens 608 (focal length 18-55 mm). Following the intermediate image plane, a beam splitter 610 reflects half of the light to an external CCD camera 612, hereinafter called the reference camera, which records a reference 2D image of the 3D object 606. The other half of the light is transmitted through the beam splitter 610 and passed to a digital micromirror device (DMD) 614 by a 4-f imaging system consisting of a tube lens 616 and a microscope objective 618 (focal length 45 mm, numerical aperture 0.16). The total demagnification of the imaging system 2000 from the object 606 to the DMD 614 is about 46-fold.

[0076] To encrypt the input image, a pseudo-random binary pattern 632 is generated by the host 630 as the key and displayed on the DMD 614. Each encoded pixel in the binary pattern 632 contains 3 x3 DMD pixels (21.6 μιη x 21.6 μιη). The encrypted image is retro-reflected through the same 4-f system, reflected by the beam splitter 610, and imaged onto the fully opened entrance slit 620 (~5 mm wide) of a streak camera 622. Deflected by a time-varying sweeping voltage 624, V, the light signal lands at various spatial locations on the y' axis according to its ToF. This temporally sheared image is recorded by an internal CCD sensor 624 in a single snapshot. This CCD sensor 626 has 672x512 binned pixels (2x2 binning), and each encoded pixel is imaged by 3 x3 binned CCD pixels. Finally, the encrypted data is transmitted to the user 628 who decrypts the image with the key provided by the host 630.

[0077] The external CCD camera 612 is synchronized with the streak camera 622 for each snapshot. An USAF resolution target is used to co-register images acquired by these two devices. Used as an intensity mask, the reference image is overlaid with the reconstructed 3D image to enhance the image quality. For each snapshot, the reconstructed 3D datacube contains N _x x N _y x N _z = 150 x 150 x 350 voxels along the x, y, and z axes, respectively. In the x-y plane, this size gives a maximum imaging field-of-view (FOV) of L _x x L _y = 150 mm x 150 mm. Given the collocated illumination and detection, the depth, z, can be calculated by where n _z is the pixel index along the z axis, d is the CCD's binned pixel size along the y' axis, and v is the shearing velocity of the streak camera 622. In our experiments, N _z = 350, d = 12.9 μηι, and v is set to 0.66 mm/ns. Therefore, the maximum depth range is L _z = 1050 mm.

CUP Method

[0078] Presented herein is an ultrafast 2D imaging technique, compressed-sensing ultrafast photography (CUP), which can capture non-repetitive events at up to 100 billion frames per second. CUP takes advantage of the compressibility of an event datacube and realizes an acquisition of petahertz data flux (105 frame pixels x 101 1 frames per second) using a CCD with only 0.3 megapixels. CUP has been demonstrated by imaging transient events involving fundamental physical phenomena such as light reflection, refraction, laser pulses racing in different media, and FTL travel of non-information. Furthermore, by utilizing a custom-built spectral separation unit, multicolor CUP may be accomplished, expanding its functionality into the realm of 4D x, y, λ, t ultrafast imaging.

[0079] In an aspect, the method may include obtaining a series of final recorded images of an object using a compressed-sensing ultrafast photography system at a rate of up to one billion frames per second. The method may include collecting a first series of object images, superimposing a pseudo-random binary spatial pattern onto each object image of the first series to produce a second series of spatially encoded images, deflecting each spatially encoded image of the second series by a temporal deflection distance proportional to a time-of-arrival of each spatially encoded image, recording each deflected spatially encoded image as a third series of spatially/temporally encoded images, and reconstructing a fourth series of final object images by processing each spatially/temporally encoded image of the third series according to an image reconstruction algorithm.

[0080] The CUP system's frame rate and temporal resolution may be determined by the shearing velocity of the streak camera: a faster shearing velocity results in a higher frame rate and temporal resolution. Unless the illumination is intensified, however, the shortened observation time window may reduce the signal-to-noise ratio, which may reduce image reconstruction quality. The shearing velocity thus may be balanced to accommodate a specific imaging application at a given illumination intensity.

[0081] In an aspect, the size of the reconstructed event datacube, N _x x N _y x

N _t (N _x , N _y , and N _t ■ the numbers of voxels along x, y, and t), may be influenced by the acceptance NA of the collecting objective, photon shot noise, and sensitivity of the

photocathode tube as well as by the number of binned CCD pixels (N _R x N _c ; N _R , the number of rows; N _c , the number of columns). Provided that the image formation closely follows the ideal forward model, the number of binned CCD pixels may become an additional influencing factor on the size of the reconstructed event datacube. Along the horizontal direction, the number of reconstructed voxels may be less than the number of detector columns, i.e., N _x < N _c . In multicolor CUP, this becomes N _x < N _C /N _L , where N _L is the number of spectral channels (i.e., wavelengths). Along the vertical direction, to avoid field clipping, the sampling obeys N _y + N _t — 1 < N _R because the spatial information and temporal information overlap and occupy the same axis.

[0082] Secure communication using CUP may be possible because the operator O is built upon a pseudo-randomly generated code matrix sheared at a preset velocity. The encrypted scene therefore may be decoded by only recipients who are granted access to the decryption key. Using a DMD (instead of a premade mask) as the field encoding unit in CUP facilitates pseudorandom key generation and potentially allows the encoding pattern to be varied for each exposure transmission, thereby minimizing the impact of theft with a single key decryption on the overall information security. Furthermore, compared with other compressed-sensing-based secure communication methods for either a ID signal or a 2D image, CUP operates on a 3D dataset, allowing transient events to be captured and communicated at faster speed.

[0083] CUP may be potentially coupled to a variety of imaging modalities, such as microscopes and telescopes, allowing imaging of transient events at scales from cellular organelles to galaxies. For example, in conventional fluorescence lifetime imaging microscopy (FLIM), point scanning or line scanning is typically employed to achieve 2D fluorescence lifetime mapping. However, since these scanning instruments cannot collect light from all elements of a dataset in parallel, there is a loss of throughput by a factor of N _x x N _y (point scanning) or N _y (line scanning) when measuring an image of N _x x N _y pixels. Additionally, scanning-based FLIM suffers from severe motion artifacts when imaging dynamic scenes, limiting its application to fixed or slowly varying samples. By integrating CUP with FLIM, parallel acquisition of a 2D fluorescence lifetime map may be accomplished within a single snapshot, thereby providing a simple solution to these long-standing problems in FLIM.

a. Image formation and reconstruction

[0084] In an aspect, CUP may operate in two steps: image formation and image reconstruction. In a non-limiting example, the image formation may be described by a forward model. During this step, the input image may be encoded with a pseudo-random binary pattern and then temporally dispersed along a spatial axis using a streak camera. Mathematically, this process is equivalent to successively applying a spatial encoding operator, C, and a temporal shearing operator, S, to the intensity distribution from the input dynamic scene, I(x, y, t):

I _s (x", y",t) = SCI(x, y,t) , (1)

[0085] Where I _s (x ' ',y ' ', t) represents the resultant encoded, sheared scene. Next, I _s may be imaged by a CCD, a process that may be mathematically formulated as Eqn. 2:

E(m,n) = TI _s (x",y",t) ₍₂₎

[0086] Here, J is a spatiotemporal integration operator (spatially integrating over each CCD pixel and temporally integrating over the exposure time). E(m,n) is the optical energy measured at pixel m, n on the CCD. Substituting Eqn. 1 into Eqn. 2 yields

E(x y') = OS(x,y, t)

(3)

[0087] where O represents a combined linear operator, i.e., O = TSC.

[0088] The image reconstruction is solving the inverse problem of Eq. 3. Given the operator O and spatiotemporal sparsity of the event, the input scene, I(x, y, t), can reasonably be estimated from measurement, E(m,n), by adopting a compressed-sensing algorithm, such as Two-Step Iterative Shrinkage/Thresholding (TwIST). The reconstructed frame rate, r, is determined by r =— . (4)

Ay"

[0089] Here, v is the temporal shearing velocity of the operator S, i.e., the shearing

Δν"

velocity of the streak camera, and ^γ is the CCD's binned pixel size along the temporal shearing direction of the operator S.

b. Forward Model

[0090] CUP's image formation process may use a forward model. The intensity distribution of the dynamic scene, I(x,y,t) , is first imaged onto an intermediate plane by an optical imaging system. Under the assumption of unit magnification and ideal optical imaging — i.e., the point-spread-function (PSF) approaches a delta function, the intensity distribution of the resultant intermediate image is identical to that of the original scene. To encode this image, a mask which contains pseudo-randomly distributed, squared, and binary-valued (i.e., either opaque or transparent) elements is placed at this intermediate image plane. The image immediately after this encoding mask has the following intensity distribution:

[0091] Here, C is an element of the matrix representing the coded mask, i, j are matrix element indices, and d' is the mask pixel size. For each dimension, the rectangular function is defined as

1, if |x| < - rect(x) : ^{1 1} 2

0, else

[0092] In this section, a mask or camera pixel is equivalent to a binned DMD or CCD pixel defined in the experiment.

[0093] This encoded image is then passed to the entrance port of a streak camera. By applying a voltage ramp, the streak camera acts as a shearing operator along the vertical axis (axis in FIG. 7) on the input image. FIG. 7 is a CUP image formation model, where x, y, are spatial coordinates; t is time; m, n, k are matrix indices; / _mjTljfc is input dynamic scene element; C _{m n} is coded mask matrix element; C _{m n} _ _k l _{m n} _ _{k k} is encoded and sheared scene element; E _{m n} is image element energy measured by a 2D detector array; and t _max is maximum recording time.

[0094] If ideal optics are assumed with unit magnification, the sheared image may be expressed as

I _s (x",y",t) = I _c (x",y" -vt,t) , (6) [0095] where v is the shearing velocity of the streak camera.

[0096] I _s {x", y",t) is then spatially integrated over each camera pixel and temporally integrated over the ex osure time. The optical energy, E(m, n) , measured at pixel m, n, is

[0097] Here, d" is the camera pixel size. Accordingly, the input scene, I(x,y,t) , can be voxelized into / . . , as follows:

I(x, y, t) ~∑ /, rect[ - (/ + ±), - (j + ), -J- - (* + ±)] , (8)

i ,k " I d 1 Δ, 1

[0098] where A _t = d" I v . If the mask elements are mapped 1 : 1 to the camera pixels (i.e., d' = d" ) and perfectly registered, combining Eq. 5-8 yields d"^ " ^-1

E(m, n) = ∑ _k I _{k k} . (9)

[0099] Here C _{m n} _ _k I _{m n} _ _{k k} represents a coded, sheared scene, and the inverse problem of

Eq. S5 can be solved using existing compressed-sensing algorithms.

[0100] It is worth noting that only those indices where n-k>0 should be included in Eqn. 9. Thus, to convert Eqn. 9 into a matrix equation, the matrices C and I need to be augmented with an array of zeros. For example, to estimate a dynamic scene with dimensions N _x x N _y x N _t , where the coded mask itself has dimensions N _x x N _y , the actual matrices I and C used in Eq. 9 will have dimensions N _x x (N _y +N _t-1 ) ^x N _t and N _x x (N _y +N _t-1 ), respectively, with zeros padded to the ends. After reconstruction, these extra voxels, containing nonzero values due to noise, are simply discarded.

c. CUP image reconstruction algorithm

[0101] Given prior knowledge of the coded mask matrix, to estimate the original scene from the CUP measurement, the inverse problem of Eqn. 9 needs to be solved. This process can be formulated in a more general form as arg min{^||£ - 0/|| ² + ¾>(/)} , (10)

[0102] where O is the linear operator, Φ(Ι) is the regularization function, and β is the regularization parameter. In CUP image reconstruction, an algorithm called Two-Step Iterative Shrinkage/Thresholding (TwIST) may be used, with (Ι) in the form of total variation (TV):

+∑∑V(^) ² +(A; J ² .

[0103] Here the discretized form of / is assumed to have dimensions N _x x N _y x N _t , and m, n, k are the three indices. I _m , I„, denote the 2D lattices along the dimensions m, n, k, respectively. Δ* and Δ! are horizontal and vertical first-order local difference operators on a

2D lattice. In TwIST, the minimization of the first term, ||ii - 0i | ² , occurs when the actual measurement E closely matches the estimated measurement OI , while the minimization of the second term, , encourages / to be piecewise constant (i.e., sparse in the gradient domain). The weighting of these two terms is empirically adjusted by the regularization parameter, β , to lead to the results that are most consistent with the physical reality. To reconstruct a datacube of size 150x150x350 (x, y, t), approximately ten minutes is required on a computer with Intel Ϊ5-2500 CPU (3.3 GHz) and 8 GB RAM. The reconstruction process may be further accelerated by using GPUs.

[0104] Traditionally, the TwIST algorithm is initialized with a pseudo-random matrix as the discretized form of and then converged to a solution by minimizing the objective function in Eqn. 10. Thus no spatiotemporal information about the event is typically employed in the basic TwIST algorithm. However, it is important to remember that the solution of TwIST might not converge to a global minimum, and hence might not provide a physically reasonable estimate of the event. Therefore, the TwIST algorithm may include a supervision step that models the initial estimate of the event. For example, if the spatial or temporal range within which an event occurs is known a priori, one can assign non-zero values to only the corresponding voxels in the initial estimate of the discretized form of / and start optimization thereafter. Compared with the basic TwIST algorithm, the supervised-TwIST approach can significantly reduce reconstruction artefacts and therefore provide a more reliable solution.

d. ToF-CUP image reconstruction algorithm

[0105] In various aspect, the CUP system is provided with active illumination to enable ToF-CUP 3D imaging that uses the time of flight of photons backscattered from a 3D object to reconstruct a 3D image of an object. For collocated illumination and detection, the round-trip ToF signal carries information about the depth, z, relative to the point of light incidence on the object's surface, which can be recovered by

z = ct _ToF 1 2 _^ where t _XoF is the ToF of received photons, and c is the speed of light. The factor of two in the denominator on the right side of Eq. 12 accounts for the round-trip flight of photons.

[0106] A collimated laser beam illuminates the 3D object having intensity reflectivity R(x, y, z). The backscattered light signal from this 3D object, I(x, y, t _XoF ), enters the ToF-CUP system 2000 described herein. The depth information of the 3D object is conveyed as the ToF of the backscattered light signal. Mathematically, this process can be described by

I(x, y, t _ToF ) = PR(x, y, z) ^ where P is a linear operator for light illumination and backscattering. Considering that the scattering is a linear process, I(x, y, t _XoF ) is linearly proportional to R(x, y, z). The ToF-CUP system then images this 3D object in three steps. First, the collected photons are spatially encrypted with a pseudo-random binary pattern, in which each pixel is set to either on or off. This pattern also acts as the decryption key to unlock and retrieve the image of the 3D object. Second, a streak camera temporally shears the ToF signal along the vertical direction. Third, the encrypted and sheared image is recorded on a CCD sensor in the streak camera via pixel-wise spatiotemporal integration. The optical energy measured at pixel (m, n) on the CCD, E(m, n), is related to the original 3D light intensity reflectivity, R(x, y, z), by

E(m, n) = TSCPR(x,y,z) (14)

Here, T, S, and C are linear operators that represent spatiotemporal integration, temporal shearing, and spatial encryption, respectively. Equation 14 shows that the encryption process is inherently embedded in the ToF-CUP method.

[0107] Image decryption can be computationally performed by users who are granted the decryption key. If the 3D object is spatiotemporally sparse, I(x, y, t _XoF ) can be reasonably estimated by solving the inverse problem of Eq. (14) using compressed-sensing algorithms. In one aspect, a two-step iterative shrinkage/thresholding (TwIST) algorithm may be used, which minimizes a convex objective function given by argmin -\E - TSCPRf + λ _τν {ΡΚ) . (15) where Φ _τν denotes the total-variation (TV) regularizer that encourages sparsity in the gradient domain during reconstruction.

[0108] The TwIST algorithm is initialized with a pseudo-random matrix of the discretized form of PR and then converged to a solution by minimizing the objective function in Eq. 15. The regularization parameter λ, which controls the weight of the TV regularizer, is adjusted empirically to provide the best for a given physical reality. Finally, R(x, y, z) can be recovered given the linear relation between the backscattered light signal and the intensity reflectivity of the object. Further, in continuous shooting mode, the evolution of the 3D images over the "slow time", t _s , R(x, y, z, t _s ), can be recovered by decrypting sequential snapshots. Here, the "slow time", t _s , relative to t _XoF , is defined as the time of capture of the imaged volume.

[0109] Besides security, the ToF-CUP method offers the advantage of more efficient information storage and transmission because data is compressed during acquisition. ToF-CUP method compresses a 3D datacube with N _x x N _y x N _z voxels to a 2D encrypted image with N _x x (N _y + N _z — 1) pixels. The data compression ratio can therefore be calculated as =

_(ΐ Ίι - _ΐ) ^{■ With the current setu} P ( ^N y ^{= 150 and N} z ^{= 350} )' = 105. Therefore, ToF-CUP can potentially improve the data transmission rate by over two orders of magnitude. However, compared with optical bandwidth-limited images, the implementation of ToF-CUP degrades the spatial resolutions by factors of 1.8 and 2.2 along the x and y axes. In addition, the depth resolution is degraded by 3.3 along the z axis, compared to the streak camera's native resolution in resolving a ToF signal. Thus, regarding actual information content, the data compression ratio may be estimated by rij = — . For the current system, rij = 8.0.

^{J 1} 1.8 X2.2 X 3.3 ^{J 11}

EXAMPLES

Example 1 : 2D ultrafast imaging of the impingement of a laser pulse upon a stripe pattern and characterization of the system 's spatial frequency responses

[0110] To characterize the system's spatial frequency responses, a dynamic scene was imaged. A laser pulse 22 impinging upon a stripe pattern 24 with varying periods is shown in FIG. 2A. The stripe periods frequency (in line pairs/mm) descends stepwise along the x axis from one edge to the other. A pulsed laser 26 delivered a collimated laser pulse (532 nm wavelength, 7 ps pulse duration, Attodyne APL-4000) to the stripe pattern 24 at an oblique angle of incidence a of about 30 degrees. The imaging system 1000 faced the pattern surface 24 and collected the scattered photons from the scene. The impingement of the light wavefront upon the pattern surface 24 was imaged by CUP at 100 billion frames per second with the streak camera's shearing velocity set to 1.32 mm/ns. The reconstructed 3D x, y, t image of the scene in intensity (W/m ² ) is shown in FIG. 2B, and the corresponding time-lapse 2D x, y images (50 mmx50 mm FOV; 150x 150 pixels frame size) were created.

[011 1] FIG. 2B also shows a representative temporal frame at t = 60 ps. The dashed line indicates the light wavefront on the pattern surface, and the arrow denotes the in-plane light propagation direction (k _xy ). Within a 10 ps frame exposure, the wavefront propagates about 3 mm in space. Including the thickness of the wavefront itself, which is about 2 mm, the wavefront image is approximately 5 mm thick along the wavefront propagation direction. The corresponding intersection with the x-y plane is 5 mm/ sin a « 10 mm thick, which agrees with the actual measurement (about 10 mm).

[0112] To provide a reference, the scene was directly imaged in fluence (J/m ² ) without introducing temporal dispersion (FIG. 2C). Next, the stripe pattern was rotated in 22.5° steps to four additional angles (22.5°, 45°, 67.5°, and 90° with respect to the x axis) and repeated the light sweeping experiment. The x, y, t scenes were projected onto the x, y plane by summing over the voxels along the temporal axis. The resultant images at two representative angles (0° and 90°) are shown in FIGS. 2D and 2E, respectively. The average light fluence distributions were compared along the x axis from FIG. 2C and FIG. 2D as well as that along the y axis from FIG. 2E. The comparison in FIG. 2F indicates that the CUP system can recover spatial frequencies up to 0.3 line pairs/mm (groups Gi, G2, and G3) along both x and y axes; however, the stripes in group G ₄ — having a fundamental spatial frequency of 0.6 line pairs/mm— are beyond the CUP system's resolution. This bandwidth limitation was further analysed by computing the spatial frequency spectra of the average light fluence distributions for all five orientations (FIG. 2G). Each angular branch appears continuous rather than discrete because the object has multiple stripe groups of varied frequencies and each has a limited number of periods. As a result, the spectra from the individual groups are broadened and overlapped. The CUP's spatial frequency response band is delimited by the inner white dashed circle, whereas the band purely limited by the optical modulation transfer function of the system without temporal shearing— derived from the reference image (FIG. 2C)— is enclosed by the outer yellow dash-dotted circle. Thus, the CUP system achieved temporal resolution at the expense of some spatial resolution.

Example 2: 2D ultrafast imaging of laser pulse reflection, refraction, and racing of two pulses in different media, and characterization of the system 's temporal resolution

[0113] To demonstrate CUP's 2D ultrafast imaging capability, three fundamental physical phenomena were imaged with single laser shots: laser pulse reflection, refraction, and racing of two pulses in different media (air and resin). It is important to mention that one-time events were truly recorded: only a single laser pulse was fired during image acquisition. In these experiments, to encompass the events within a preset time window (10 ns) on the streak camera, the pulsed laser (Attodyne APL-4000) was synchronized with the streak camera through a digital delay generator (Stanford Research Systems DG645). Moreover, to scatter light from the media to the CUP system, dry ice was evaporated into the light path in the air and added zinc oxide powder into the resin, respectively.

[0114] FIGS. 3 A and 3B show representative time-lapse frames of a single laser pulse reflected from a mirror in the scattering air and refracted at an air-resin interface, respectively. With a shearing velocity of 0.66 mm/ns in the streak camera, the reconstructed frame rate is 50 billion frames per second. Such a measurement allows the visualisation of a single laser pulse's compliance to the laws of light reflection and refraction, the underlying foundations of optical science. It is worth noting that the heterogeneities in the images are likely attributable to turbulence in the vapour and non-uniform scattering in the resin.

[0115] To validate CUP's ability to quantitatively measure the speed of light, photon racing was imaged in real time. The original laser pulse was split into two beams: one beam propagated in the air and the other in the resin. The representative time-lapse frames of this photon racing experiment are shown in FIG. 3C. As expected, due to the different refractive indices (1.0 in air and 1.5 in resin), photons ran faster in the air than in the resin. By tracing the centroid from the time-lapse laser pulse images (FIG. 3D), the CUP-recovered light speeds in the air and in the resin were (3.1 ± 0.5) x 10 ^s m/s and (2.0 ± 0.2) x 10 ^s m/s, respectively, consistent with the theoretical values (3.0 x 10 ⁸ m/s and 2.0 x 10 ⁸ m/s). Here the standard errors are mainly attributed to the resolution limits.

[0116] By monitoring the time-lapse signals along the laser propagation path in the air, CUP's temporal resolution was quantified. Because the 7 ps pulse duration is shorter than the frame exposure time (20 ps), the laser pulse was considered as an approximate impulse source in the time domain. The temporal point-spread-functions (PSF) were measured at different spatial locations along the light path imaged at 50 billion frames per second (20 ps frame exposure time), and their full widths at half maxima averaged 74 ps. Additionally, to study the dependence of CUP's temporal resolution on the frame rate, this experiment was repeated at 100 billion frames per second (10 ps frame exposure time) and re-measured the temporal PSFs. The mean temporal resolution was improved from 74 ps to 31 ps at the expense of signal-to-noise ratio. At a higher frame rate (i.e., higher shearing velocity in the streak camera), the light signals are spread over more pixels on the CCD camera, reducing the signal level per pixel and thereby causing more potential reconstruction artefacts.

Example 3 : 2D ultrafast imaging of faster-than-light (FTL) travel of non-information

[0117] To explore CUP's potential application in modern physics, apparent faster-than- light phenomena were imaged in 2D movies. According to Einstein's theory of relativity, the propagation speed of matter cannot surpass the speed of light in vacuum because it would need infinite energy to do so. Nonetheless, if the motion itself does not transmit information, its speed can be faster than light. This phenomenon is referred to as faster-than-light propagation of non-information. To visualise this phenomenon with CUP, an experiment was designed using a setup similar to that shown in FIG. 2A. The pulsed laser illuminates the scene at an oblique angle of incidence of about 30 degrees, and CUP images the scene normally (0 degree angle). To facilitate the calculation of speed, a stripe pattern was imaged with a constant period of 12 mm (FIG. 4A).

[0118] The movement of a light wavefront intersecting with this stripe pattern is captured at 100 billion frames per second with the streak camera's shearing velocity set to 1.32 mm/ns. Representative temporal frames are provided in FIG. 4B. As shown in FIG. 4B, the white stripes shown in FIG. 4A are illuminated sequentially by the sweeping wavefront. The speed of this motion, calculated by dividing the stripe period by their lit-up time interval, is v _FTL = 12 mm / 20 ps = 6x l0 ⁸ m/s, two times of the speed of light in the air due to the oblique incidence of the laser beam. As shown in FIG. 4C, although the intersected wavefront— the only feature visible to the CUP system— travels from location A to B faster than the light wavefront, the actual information is carried by the wavefront itself, and thereby its transmission velocity is still limited by the speed of light in the air.

Example 4: Multicolor compressed-sensing ultrafast photography (Multicolor-CUP)

[0119] To extend CUP's functionality to reproducing colors, a spectral separation module was added in front of the streak camera. As shown in FIG. 5A, a dichroic filter 302 (562 nm cut-on wavelength) is mounted on a mirror 304 at a small tilt angle 314 (~5°). The light reflected from this module is divided into two beams according to the wavelength: green light (wavelength < 562 nm) is directly reflected from the dichroic filter 302, while red light (wavelength > 562 nm) passes through the dichroic filter 302 and bounces from the mirror 304. Compared with the depth of field of the imaging system, the introduced optical path difference between these two spectral channels is negligible, therefore maintaining the images in focus for both colors.

[0120] Using the multicolor CUP system, a pulsed-laser-pumped fluorescence emission process was imaged. A fluorophore, Rhodamine 6G, in water solution was excited by a single 7 ps laser pulse at 532 nm. To capture the entire fluorescence decay, 50 billion frames per second was used by setting a shearing velocity of 0.66 mm/ns on the streak camera. Some

representative temporal frames are shown in FIG. 5B. In addition, the time-lapse mean signal intensities were calculated within the dashed box in FIG. 5B for both the green and red channels (FIG. 5C). Based on the measured fluorescence decay, the fluorescence lifetime was found to be 3.8 ns, closely matching a previously reported value.

[0121] In theory, the time delay from the pump laser excitation to the fluorescence emission due to the molecular vibrational relaxation is ~6 ps for Rhodamine 6G. However, results show that the fluorescence starts to decay -180 ps after the pump laser signal reaches its maximum. In the time domain, with 50 billion frames per second sampling, the laser pulse functions as an approximate impulse source while the onset of fluorescence acts as a decaying edge source. Blurring due to the temporal PSF stretches these two signals' maxima apart. This process was theoretically simulated by using the experimentally measured temporal PSF and the fitted fluorescence decay as the input. The time lag between these two events was found to be 200 ps, as shown in FIG. 6D, which is in good agreement with experimental observation.

[0122] A simulation of temporal responses of pulsed-laser-pumped fluorescence emissions was conducted. FIG. 6A shows an event function Bdescribing the pulsed laser fluorescence excitation. FIG. 6b shows an event function describing the fluorescence emission. FIG. 6c is a measured temporal point-spread- function (PSF), with a full width at half maximum of -80 ps. Due to reconstruction artefacts, the PSF has a side lobe and a shoulder extending over a range of 280 ps. FIG. 6d are simulated temporal responses of these two event functions after being convolved with the temporal PSF. The maxima of these two time-lapse signals are stretched by 200 ps.

Example 5: Simulation of temporal responses of pulsed-laser-pumped fluorescence emission

[0123] The temporal response of pulsed-laser-pumped fluorescence emission was simulated in Matlab. The arrival of the pump laser pulse and the subsequent fluorescence emission are described by a Kronecker delta function (FIG. 6a) and an exponentially decaying edge function (FIG. 6b), respectively. For the Rhodamine 6G fluorophore, the molecular vibrational relaxation time was neglected and the arrival of the pump laser pulse and the onset of fluorescence emission were considered to be simultaneous. After pump laser excitation, the decay of the normalized fluorescence intensity, I(t) , is modelled as l(t) = exp(— t/τ), where τ = 3.8 ns. To simulate the temporal-PSF-induced blurring, an experimentally measured temporal PSF (Extended Data Fig. lc) was convolved with these two event functions shown in FIGS. 6A and 6B. The results in FIG. 6D indicate that this process introduces an approximate 200 ps time delay between the signal maxima of these two events. Although the full width at half maximum of the main peak in the temporal PSF is only ~80 ps, the reconstruction-induced side lobe and shoulder extend over a range of 280 ps, which temporally stretches the signal maxima of these two events apart.

Example 6: Depth resolution of ToF-CUP 3D imaging

[0124] To quantify the ToF-CUP system's depth resolution, a 3D target with fins of varying heights (FIG. 10A) was imaged . This target (100 mm x 50 mm along the x and y axes) was fabricated using a 3D printer (Form 1+, Formlabs). Along the x axis, each fin had a width of 5 mm, and the height of the fins ascended from 2.5 mm to 25 mm, in steps of 2.5 mm. The imaging system was placed perpendicular to the target and collected the backscattered photons from the surface. Image reconstruction retrieved the ToF 2D images (FIG. 10B). Three representative temporal frames at t _XOF = 120, 200, and 280 ps are shown in FIG. IOC. In each frame, five fins are observed, indicating that the system's depth resolution is approximately 10 mm.

Example 7: ToF-CUP 3D imaging of static objects

[0125] To demonstrate ToF-CUP's 3D imaging capability, static objects were imaged. Specifically, two letters, "W" and "U", were placed with a depth separation of 40 mm. The streak camera acquired a spatially-encrypted, temporally-sheared image of this 3D target in a single snapshot. The reference camera also directly imaged the same 3D target without temporal shearing to acquire a reference. The ToF signal was converted into depth information as described herein above, and ToF-CUP reconstructed a 3D x, y, z image of the target. For each pixel in the x-y plane, we found the maximum intensity in the z axis and recorded that coordinate to build a depth map. We color-encoded this depth map and overlaid it with the reference image to produce a depth-encoded image (FIG. 11 A). For this object, the depth distance between the two letters was measured to be -40 mm, which agreed with the true value. In addition, we imaged two additional static objects, a wooden mannequin and a human hand (FIGS. 1 IB and 1 1C). In both cases, the depth information of the object was obtained using ToF-CUP. It is worth noting that the lateral resolution of the reconstructed datacube was ~0.1 line pairs per mm, and the reference images taken by the external CCD camera had a higher lateral resolution (-0.8 line pairs per mm). Because the depth-encoded image was produced by overlaying the depth map with the reference image, it had a lateral resolution limited by the reconstructed datacube.

Example 8: Encryption of ToF-CUP 3D images

[0126] To verify the system's encryption capability, the image quality of the 3D datacubes reconstructed under two types of decryption attacks was compared to a reference image. The static 3D object "WU" was used in these tests. First, a brute force attack was simulated, which attempted to guess the decryption key without any prior information. Pseudorandom binary masks were generated as invalid decryption keys. For each invalid key, the percentage of resemblance to the correct key was calculated. After the reconstruction, the cross correlations between the 3D datacubes based on these invalid keys and the one based on the correct key were calculated to quantify the reconstructed image quality (FIG. . 12A). Without the valid decryption key, the reconstructed image quality was largely degraded, as reflected in the decreased correlation coefficients. For direct comparison, the reconstructed 3D datacube of the "WU" target produced by the valid and invalid keys are shown (FIGS. 12B and 12C, respectively). With the correct decryption key, the reconstructed image resembled the object. The image reconstructed using the invalid decryption key, on the contrary, yielded no useful information. In each attack, the reconstruction using the invalid key failed to retrieve the depth information, which demonstrated that the ToF-CUP encryption method is resistant to brute force attacks.

[0127] In addition, the strength of encryption was assessed when part of the encryption key was known, but its position with respect to the encrypted image was unknown. To simulate this situation, a subarea (40x40 encoded pixels in the x and y axes) was selected from the full encryption key (50x50 encoded pixels in the x and y axes) as the decryption key (FIG. 12D). This decryption key was horizontally shifted by various numbers of the encoded pixels. For each shift, the reconstructed 3D datacube was compared with the correct reconstruction result to calculate the cross-correlation coefficient (FIG. 12D). The comparison showed that the reconstruction quality was sensitive to the relative position between the decryption key and the encrypted data (FIG. 12E), demonstrating that ToF-CUP encrypting can protect the information in the 3D datacube even when part of the encryption key is leaked. The reconstructed datacubes from invalid decryption keys contained randomly distributed artifacts, some of which may have high intensity. These artifacts may affect the cross-correlation calculation. However, as shown in FIGS. 12C and 12E, even with seemingly high cross-correlation coefficients, the

reconstruction using invalid encryption keys did not resemble the original 3D object.

Example 9: ToF-CUP 3D images of moving objects

[0128] To demonstrate ToF-CUP's dynamic 3D imaging capability, a rotating object was imaged in real time (FIG. 13 A). In the experiment, a foam ball with a diameter of 50.8 mm was rotated by a motorized stage at -150 revolutions per minute. Two "mountains" and a "crater" were added as features on this object. Another foam ball, 25.4 mm in diameter, was placed 63.5 mm from the larger foam ball and rotated concentrically at the same angular speed. The ToF-CUP camera captured the rotation of this two-ball system by sequentially acquiring images at 75 volumes per second. Once each image was reconstructed to a 3D x, y, z datacube, these datacubes formed a time-lapse 4D x, y, z, ts datacube. FIG. 13B shows representative depth-encoded images at six different slow-time points, which revealed the relative depth positions of these two balls. Example 10: ToF-CUP 3D images of live organisms

[0129] To apply ToF-CUP's dynamic 3D imaging capability to biological applications, a swimming comet goldfish (Carassius auratus) was imaged. The ToF-CUP camera acquired 3D images at two volumes per second to capture the fish's relatively slow movement over a sufficiently long time. FIG. 14A shows six representative depth-encoded images of the fish. By tracing the centroid of each reconstructed datacube, we demonstrated 3D spatial position tracking of the fish (FIG. 14B). In this representative example, the ToF-CUP camera revealed that the fish first stayed at the rear lower left corner and then moved toward the right, after which it started to move toward the front wall of the fish tank.

[0130] In these dynamic 3D imaging experiments, the external CCD camera was operated at a relatively long exposure time to tolerate relatively weak backscattered light. As a result, the movement of objects blurred the reference image. In contrast, because the exposure time of the streak camera is on the nanosecond level, the movement of the object did not noticeably affect the reconstructed datacube. Hence, the lateral and depth resolutions in the reconstructed images were not degraded.

Example 1 1 : ToF-CUP 3D images of objects in scattering media

[0131] To explore ToF-CUP's imaging capability in a real-world environment, the ToF- CUP system was used to image an object moving behind a scattering medium that was composed by adding various concentrations of milk to water in a tank. The experimental setup is illustrated in FIG. 15 A. Specifically, the incident laser beam was first de-expanded to ~2 mm in diameter. A beam sampler reflected a small fraction of the energy of the beam toward the tank. After propagating through the scattering medium, the transmitted beam passed through an iris (~2 mm in diameter). Then, the transmitted beam was measured by a photodiode detector to quantify the scattering level in the medium, which is presented as the equivalent scattering thickness in units of the mean free path (l _t ). The rest of the incident laser beam was sent through the beam sampler and reflected by a mirror to an engineered diffuser (see FIG. 9), which generated wide-field illumination of a moving airplane-model target behind the scattering medium. This manually operated airplane-model target moved in a curved trajectory illustrated in FIG. 15A.

[0132] The ToF-CUP camera imaged this moving object through the scattering medium with various scattering thicknesses. To quantitatively compare the image quality, we selected a representative reconstructed 3D x, y, z, image at a single slow-time point for each scattering thickness, and summed over the 3D image voxels along the z axis. The resultant projected images are shown in FIG. 15B. In addition, the intensity profile of a cross section of the airplane wing is plotted under these conditions in FIG. 15C. The image contrast decreased with increased scattering in the medium and finally vanishes when the scattering thickness reaches 2.2l _t . FIGS 15D and 15E show representative images of this moving airplane target at five different slow- time points with two scattering thicknesses (1.0/ _/ in FIG. 15D and 2Al _t in FIG. 15E), which record that the airplane-model target moved from the lower left to the upper right, as well as toward the ToF-CUP camera in the depth direction. Although scattering causes loss of contrast and features in the image, the depth can still be perceived. Due to the manual operation, the speed of the airplane- model target was slightly different in each experiment. As a result, the recorded movies with two scattering thicknesses (1.0/ _/ and 2.1 l _t ) have different lengths, and so have the selected representative images in FIG. 15D and 15E.

[0133] The foregoing merely illustrates the principles of the invention. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements and methods which, although not explicitly shown or described herein, embody the principles of the invention and are thus within the spirit and scope of the present invention. From the above description and drawings, it will be understood by those of ordinary skill in the art that the particular embodiments shown and described are for purposes of illustrations only and are not intended to limit the scope of the present invention. References to details of particular embodiments are not intended to limit the scope of the invention.